Re: Syncing lucene index with a database

2009-03-27 Thread Matt Schraeder
Thanks a million. You really helped me get on the right direction. I'm going to start building some test cases this afternoon so I can really begin to get my hands dirty and see how everything works. It's good to know that there isn't one "right" way for my purposes, which is mostly what I wanted

Re: Syncing lucene index with a database

2009-03-27 Thread Erick Erickson
Yes, updating a document in Lucene is "expensive" for two reasons: 1> deleting and adding a document does mean there's internal work being done. But it's not all *that* expensive. So this really comes down to how many records you expect to update every 15 minutes. You've gotta try it. 2

Re: Syncing lucene index with a database

2009-03-27 Thread Matt Schraeder
I'm going to try and cover all replies so far, but for the most part this first one since it had the most help so far. Thanks to everyone who replied so far, you've given me a lot of great ideas to think about and look into. I'm going to begin some small test indexes with our data so we have somet

Re: Syncing lucene index with a database

2009-03-27 Thread Amin Mohammed-Coleman
Hi I was going to suggest looking at hibernate search. It comes with event listeners that modify your indexes when the persistent entity changes. It use lucene under the hood so if you need to access lucene the you can. Indexing can be done sync or async and the documentation shows how to

Re: Syncing lucene index with a database

2009-03-26 Thread Tim Williams
On Thu, Mar 26, 2009 at 6:28 PM, Matt Schraeder wrote: > I'm new to Lucene and just beginning my project of adding it to our web > app.  We are indexing data from a MS SQL 2000 database and building > full-text search from it. > > Everything I have read says that building the index is a resource h

Re: Syncing lucene index with a database

2009-03-26 Thread Chris Lu
There are many things you need to synchronize with database. Besides just changed fields, you may need to deal with deleted database records, etc. In general, it's not efficient to pull over data that's changing often.and may not have much effect on search. It'll overload Lucene unnecessarily

Re: Syncing lucene index with a database

2009-03-26 Thread Erick Erickson
You've got a great grasp of the issues, comments below. But before you do, a lot of this kind if thing is incorporated in SOLR, which is build on Lucene. Particularly updating an index then using it. So you might take a look over there. It even has a DataImportHandler... WARNING: I've only been mo

Syncing lucene index with a database

2009-03-26 Thread Matt Schraeder
I'm new to Lucene and just beginning my project of adding it to our web app. We are indexing data from a MS SQL 2000 database and building full-text search from it. Everything I have read says that building the index is a resource heavy operation so we should use it sparingly. For the most part