Thanks for your answer,
I didn't think that using: Document doc irHourly.doc(<id in hit>);
would be much faster than using the searcher. I will try that.
I have one question though: what is the <id in hit> you reffer to. Since I
have searched in the daily index, what is the corresponding hit on the
hourly index if i haven't searched in the hourly index yet?
Thanks for your help,
Francois
PS: Concerning your idea to merge the hourly and daily index, I did think
about that but the quantity of info I have is already way too large in both
indexes (about 100 fields in each) to repeat that information so many times.
My Index is already of 10GB.
Erick Erickson wrote:
>
> Well, making a large OR clause is definitely more efficient than making
> N different requests, but you would have to search the results. It doesn't
> sound very performant.
>
> Could you go to 50,000 ids? yes, but you have to fiddle with
> setMaxClauseCount
> because Lucene defaults to a max of 1,024.
>
> There's no way I know of to emulate a DB join. Usually, whenever I try
> the answer is to flatten my data but I don't see a good way to do that
> in this case.
>
> Hmmm, what do you think would happen if you opened an IndexReader
> into your Hourly index straight from your Daily query? Something like:
>
> IndexReader irHourly = <open index reader to Hourly>.
> for (each hit in Daily) {
> for (each id in this Hourly doc) {
> Document doc irHourly.doc(<id in hit>);
> addToResponse(doc); // your method here.....
> }
> }
>
>
> You *probably* want to keep the Hourly reader open between requests, but
> since the above isn't searching, you *might* be able to open it each time.
> I'd
> go for keeping it open between requests if at all possible.
>
> And here's a wild and crazy idea. Remember that Lucene documents don't
> require that *any* fields be in common. It might make your management
> easier if you *combined* the indexes. Crudely, prefix each Daily field
> with
> "D_" and each hourly field with "H_" and put 'em all in the same index.
> I'm
> not claiming that's a good solution in your case, but I thought I'd
> mention
> it
> as a possibility. You still can't do joins on them though.....
>
> Erick
>
>
>
> On Fri, Dec 18, 2009 at 9:08 AM, François Eric
> <francois.e...@jarca.ca>wrote:
>
>> Hello,
>>
>> I have a performance problem and would need expert advice on how to go
>> about fixing it:
>>
>> I currently have 2 indexes: Daily and Hourly. The Daily index contains
>> about 1,000,000 documents and my Hourly index approximately: 24,000,000
>> documents. My Daily index contains many fields and some of them are IDs
>> to
>> my Hourly Index.
>> What I want to do is fetch data in one request (if possible).
>> Right now I do it in many requests:
>> 1- Get the matching Daily documents (say it returns 500 documents)
>> 2- For each of these documents, locate the Hourly Index Id and fetch it.
>>
>> Therefore I make 501 requests to lucene. This causes some performance
>> issues I guess because of the overhead to making a request to Lucene.
>>
>> Is it possible to do this in 1 request? I'm thinking no because I'm not
>> sure what the result set would be but maybe I'm missing something.
>>
>> If not I guess it would be possible to build a query with my 500 hourly
>> ids
>> and make a OR between them to make it in 2 requests....but then I have to
>> find the matching documents. Will this overflow if I have 50000 ids in
>> my
>> query?
>>
>> Anyway, I just want advice on how one would address this situation.
>>
>> Thank you very much,
>>
>> François
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: java-user-h...@lucene.apache.org
>>
>>
>
>
--
View this message in context:
http://old.nabble.com/Query-joining-2-indexes-tp26843980p26846129.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
