Re: Using Lucene partly as DB and 'joining' search results.

2008-04-14 Thread Antony Bowesman
Chris Hostetter wrote: you can't ... that's why i said you'd need to rebuild the smaller index completley on a periodic basis (going in the same order as the docs in the Mmm, the annotations would only be stored in the index. It would be possible to store them elsewhere, so I can investigate

Re: Using Lucene partly as DB and 'joining' search results.

2008-04-14 Thread Chris Hostetter
: would then have to make a join using mailId against the core. However, if I : want to use PR, I could have a single Document with multiple field, and using : stored fields can 'modify' that Document. However, what happens to the DocId : when the delete+add occurs and how do I ensure it stays t

Re: Using Lucene partly as DB and 'joining' search results.

2008-04-14 Thread Antony Bowesman
Thanks all for the suggestions - there was also another thread "Lucene index on relational data" which had crossover here. That's an interesting idea about using ParallelReader for the changable index. I had thought to just have a triplet indexed 'owner:mailId:label' in each Doc and have multi

Re: Using Lucene partly as DB and 'joining' search results.

2008-04-14 Thread Chris Hostetter
: The archive is read only apart from bulk deletes, but one of the requirements : is for users to be able to label their own mail. Given that a Lucene Document : cannot be updated, I have thought about having a separate Lucene index that : has just the 3 terms (or some combination of) userId + ma

Re: Using Lucene partly as DB and 'joining' search results.

2008-04-12 Thread Paul Elschot
Op Saturday 12 April 2008 00:03:13 schreef Antony Bowesman: > Paul Elschot wrote: > > Op Friday 11 April 2008 13:49:59 schreef Mathieu Lecarme: > >> Use Filter and BitSet. > >> From the personnal data, you build a Filter > >> (http://lucene.apache.org/java/2_3_1/api/org/apache/lucene/search/ > >>F

Re: Using Lucene partly as DB and 'joining' search results.

2008-04-11 Thread Antony Bowesman
Paul Elschot wrote: Op Friday 11 April 2008 13:49:59 schreef Mathieu Lecarme: Use Filter and BitSet. From the personnal data, you build a Filter (http://lucene.apache.org/java/2_3_1/api/org/apache/lucene/search/Fil ter.html) wich is used in the main index. With 1 billion mails, and possibly

Re: Using Lucene partly as DB and 'joining' search results.

2008-04-11 Thread Paul Elschot
Op Friday 11 April 2008 13:49:59 schreef Mathieu Lecarme: > Antony Bowesman a écrit : > > We're planning to archive email over many years and have been > > looking at using DB to store mail meta data and Lucene for the > > indexed mail data, or just Lucene on its own with email data and > > structu

Re: Using Lucene partly as DB and 'joining' search results.

2008-04-11 Thread Mathieu Lecarme
Antony Bowesman a écrit : We're planning to archive email over many years and have been looking at using DB to store mail meta data and Lucene for the indexed mail data, or just Lucene on its own with email data and structure stored as XML and the raw message stored in the file system. For so

Using Lucene partly as DB and 'joining' search results.

2008-04-11 Thread Antony Bowesman
We're planning to archive email over many years and have been looking at using DB to store mail meta data and Lucene for the indexed mail data, or just Lucene on its own with email data and structure stored as XML and the raw message stored in the file system. For some customers, the volumes a