At the risk of sounding overly critical here, I would say you need to scrap your entire approach of building one small index per request, and just build your entire searchable data store in Lucene/Solr. This is the simplest and probably most maintainable and scalable solution. Even if your index contains 10M+ documents, returning at most 500 search results should be lightning fast compared to the latencies you're seeing right now. To facilitate data export from the DB, take a look at this: http://wiki.apache.org/solr/DataImportHandler
On Tue, May 20, 2014 at 7:36 AM, Shruthi <sse...@imedx.com> wrote: > > > > > -----Original Message----- > From: Toke Eskildsen [mailto:t...@statsbiblioteket.dk] > Sent: Tuesday, May 20, 2014 3:48 PM > To: java-user@lucene.apache.org > Subject: Re: NewBie To Lucene || Perfect configuration on a 64 bit server > > On Tue, 2014-05-20 at 11:56 +0200, Shruthi wrote: > > Toke: > > Is 20 second an acceptable response time for your users? > > > > Shruthi: Its definitely not acceptable. PFA the piece of code that we > > are using..Its taking 20seconds. That’s why I drafted this ticket to > > see where I was going wrong. > > Indexing 1000 documents/sec in Lucene is quite common, so even taking > into account large documents, 20 seconds sounds like quite a bit. > Shruthi: I had attached the code snippet in previous mail. Do you suspect > a foul play there? > > > Shruthi: Well, its two stage process: Client is looking at > > historical data based on a parameters like names, dates,MRN, fields > > etc.. SO the query actually gets the data set fulfilling the > > requirements > > > > If client is interested in doing a text search then he would pass the > > search phrase on the result set. > > So it is not possible for a client to perform a broad phrase search to > start with. And it sounds like your DB-queries are all simple matching? > No complex joins and such? If so, this calls even more for a full > Lucene-index solution, which handles all aspect of the search process. > Shruthi: We call a DB stored procedure to get us the result set for > working with.. > We will be using highlighter API and I don’t think Memory index can be > used with highlighter. > > > > - Toke Eskildsen, State and University Library, Denmark > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
