replication question

2008-12-15 Thread Michael Stoppelman
I've got a question from Doug's original email about replication ( http://www.mail-archive.com/lucene-u...@jakarta.apache.org/msg12709.html): "1. On the index master, periodically checkpoint the index. Every minute or so the IndexWriter is closed and a 'cp -lr index index.DATE' command is executed

Need Opinion!!

2008-12-15 Thread Shardul Bhatt
Hi All, I am facing a scenrio where I am considering using Lucene over the existing implementation. The move to Lucene is going to require a lot of re-work so I though I better post this and ask for an expert opinion. Background There are 3 tables. 1. University 2. Course 3. Subjects A un

Re: Lucene Data Structures

2008-12-15 Thread Prafulla Kiran
Well, I have seen this link many times before. It doesn't really explain the data structures part of it. Perhaps I should have asked my question this way: "What data structures are being used by Lucene to read the posting lists from the index ?" . My guess is that a hash table is being used for

Re: Singleton and Lucene: org.apache.lucene.store.AlreadyClosed

2008-12-15 Thread Zender00
Hi Paul, thanks for your reply. I compared your source code and it seems that you are using the same mechanism for the singleton like me... I am using lucene 2.4 and get the exception above explained. I am wondering why because I only create an instance of my lucene object (which itself construct

process dies with OOM after processing 10k docs

2008-12-15 Thread jm
Hi, I am having a memory issue with Lucene2.4. I am strating a process with 128MB of ram, this process handles incoming request from others, and indexes objects in a number of lucene indexes. My lucene docs, all have 6 fields: -one is small: Field.Store.YES, Field.Index.UN_TOKENIZED, Field.TermVe

Re: searching in 2 indexes

2008-12-15 Thread Erick Erickson
This is where things get exciting, when theory runs right up against the particular problem at hand... What if your document consisted of a content field (perhaps broken into as many pieces as necessary) and multiple references? Something like: Document doc = new Document() doc.add("content", ...

Re: searching in 2 indexes

2008-12-15 Thread Erik Hatcher
On Dec 15, 2008, at 12:44 PM, Chris Bamford wrote: One question (thinking ahead now): how to differentiate between content and reference hits if they are retrieved from the same index? Presumably I will need to try to retrieve a field which only exists in one and if it fails, I know it must

Re: searching in 2 indexes

2008-12-15 Thread Chris Bamford
Hi Erick, You raise some excellent points. First let me explain why our structure is the way it is .. There is not actually a strict 1-1 relationship between entries in the two indexes. One represents content, the other, references. There can only ever be 1 content item, but there can be seve

Re: Payload Question

2008-12-15 Thread Todd Benge
Thanks for the help. I've been reading through some of the postings so I'll look that one up. Appreciate it. Todd On Mon, Dec 15, 2008 at 8:43 AM, Greg Shackles wrote: > Hey Todd, > > If you look for a thread I started a month or two ago, there was a pretty > good discussion of payloads (it is

Re: searching in 2 indexes

2008-12-15 Thread Erick Erickson
Stop it right now . You've gotta take off your DB hat and put on your searching hat to get the most out of Lucene. So I'd think about the following: 1> Why do you have two indexes? Why not just put all the data into a single index? The fields are disjoint anyway Note that there is no

searching in 2 indexes

2008-12-15 Thread Chris Bamford
Hi I have a situation where I have two related indexes which are logically linked by a common field called INDEXID. All other fields differ between the two indexes. For any given INDEXID I would like to be able to retrieve the matching pair of documents, one from each index. (Logically this i

Re: Payload Question

2008-12-15 Thread Greg Shackles
Hey Todd, If you look for a thread I started a month or two ago, there was a pretty good discussion of payloads (it is where I initially learned about them). In that thread should also be an explanation of the solution I ended up using for implementing payloads, so maybe that would be helpful for

Re: Open IndexReader read-only

2008-12-15 Thread Chris Bamford
Thanks Mark, worked a treat. Mark Miller wrote: Chris Bamford wrote: Mark > Look for the static factory methods on IndexReader. I take it you mean IndexReader.open (dir, true) ? Yeah. If so, how do I then pass that into DelayCloseIndexSearcher() so that I can continue to rely on all the e

Re: Lucene SpellChecker returns no suggetions after changing Server

2008-12-15 Thread Matthias W.
I fixed this problem by writing an external Java class and changing completly to Lucene-2.4.0. The SpellChecker now returns results but it is very slow. My index contains about 2 words, I don't think this is too much. How can I speed it up? My temporary method: public static Vector suggest

Re: Lucene Data Structures

2008-12-15 Thread Grant Ingersoll
http://lucene.apache.org/java/2_4_0/fileformats.html On Dec 15, 2008, at 12:15 AM, Prafulla Kiran wrote: Hi Everybody, Could someone please explain the actual data structures being used by Lucene for storing the postings list in the index. I see a file called MultileveSkipListReader and Mu

Re: Singleton and Lucene: org.apache.lucene.store.AlreadyClosed

2008-12-15 Thread Paul Libbrecht
Le 15-déc.-08 à 13:09, Zender00 a écrit : in my opinion it is possible to use a reader and writer simultanesously on the same index. In my already working lucene class, I use they are both open all the time. mmmh... But this isnt the reason for my exception. It seems that lucene cannot

Re: Singleton and Lucene: org.apache.lucene.store.AlreadyClosed

2008-12-15 Thread Zender00
Hi paul, in my opinion it is possible to use a reader and writer simultanesously on the same index. In my already working lucene class, I use they are both open all the time. But this isnt the reason for my exception. It seems that lucene cannot handle my singleton. How did you implement your si

Re: Singleton and Lucene: org.apache.lucene.store.AlreadyClosed

2008-12-15 Thread Paul Libbrecht
We use singletons all the time, Just be careful that are writer and a reader cannot be opened simultaneously on the same index. Either you write or your read (and delete). paul Le 15-déc.-08 à 12:06, Zender00 a écrit : I extend my original post: How is it possible to use lucene in 2 di

Re: Singleton and Lucene: org.apache.lucene.store.AlreadyClosed

2008-12-15 Thread Zender00
I extend my original post: How is it possible to use lucene in 2 diffent classes / objects, if I dont want to use static methods/variables? (one object read/writer, the other is read only) Kind regards, D. Penning Zender00 wrote: > > Hi, > I tried to implement a singleton pattern for my already

Re: Payload Question

2008-12-15 Thread David Causse
Hi, After adding fields, those fields are analyzed and this is the step you are looking for. The payloads are stored on each Token, so you need your own Analyzer to do so. just use reusableToken.setPayload(myPayLoad) somewhere, look at already existing analyzers. In our case we use TokenStream