@ganesh: For increasing memory in tomcat, you wanna increase it in CATALINA_OPTS in catalina.sh file add this : -Xmx1500m which means it shud not use more than 1500 megs or -Xms500m should have atleast 500 megs
On Mon, Sep 22, 2008 at 5:15 PM, Ganesh - yahoo <[EMAIL PROTECTED]>wrote: > My index crossed 5 GB and 5 million documents are indexed. > My query includes searching and sorting returns 40000 hits. > > If i do search from a standalone application, the results are returned in > 12 seconds. If i perform the same from web application running inside > Tomcat, out of memory exception is occured. > > Could any one clarify it? > > Regards > Ganesh > > ----- Original Message ----- From: "Ganesh - yahoo" <[EMAIL PROTECTED] > > > To: <java-user@lucene.apache.org> > Sent: Friday, September 19, 2008 10:56 AM > > Subject: Re: Exception while doing sorting > > > Ok. If i distribure the indexes, whether sorting would be faster? >> >> In Lucene user group mailing list, most emails suggests to use single >> indicies. Searching across the indexes may not be slower? >> >> Lucene uses FieldCache for sorting on non-tokenized field and tries to >>>> maintain fields from all your 4 millions documents, even if you need >>>> to sort only 4000 docs. >>>> >>> Don't know why Lucene keeps all terms in FieldCache for sorting. It >> supposed to sort only the hits. Please clarify? >> >> Regards >> Ganesh >> >> ----- Original Message ----- From: "Otis Gospodnetic" < >> [EMAIL PROTECTED]> >> To: <java-user@lucene.apache.org> >> Sent: Thursday, September 18, 2008 12:17 PM >> Subject: Re: Exception while doing sorting >> >> >> If your index is increasing in size so fast, you should start thinking >>> about sharding your index (breaking it into multiple smaller indices that >>> each fits on its server) and searching across them (aka distributed search). >>> >>> Yes, Lucene can handle millions of records if run on adequate hardware >>> and if used correctly. >>> >>> Otis >>> -- >>> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch >>> >>> >>> >>> ----- Original Message ---- >>> >>>> From: Ganesh - yahoo <[EMAIL PROTECTED]> >>>> To: java-user@lucene.apache.org >>>> Sent: Thursday, September 18, 2008 12:53:19 AM >>>> Subject: Re: Exception while doing sorting >>>> >>>> My index is growing by 1 million records per day. How much memory do i >>>> need >>>> to increase. >>>> >>>> What kind of sorting algorithm being used in Lucene. Is this efficient >>>> enough to handle millions of records. >>>> >>>> Whether we could do sorting using our own algorithm? >>>> >>>> Regards >>>> Ganesh >>>> >>>> ----- Original Message ----- From: "Fuad Efendi" >>>> To: >>>> Sent: Wednesday, September 17, 2008 7:28 PM >>>> Subject: Re: Exception while doing sorting >>>> >>>> >>>> Increase memory. >>>> >>>> Lucene uses FieldCache for sorting on non-tokenized field and tries to >>>> maintain fields from all your 4 millions documents, even if you need >>>> to sort only 4000 docs. >>>> ============== >>>> http://www.tokenizer.org/bot.html >>>> >>>> >>>> Quoting Ganesh - yahoo : >>>> >>>> > Hello all, >>>> > >>>> > I am have indexed more than 4 million documents. My query fetches >>>> > 300,000 hits. If i perform sorting on any field, then tomcat reports >>>> > out of memory exception. >>>> > Sometimes the query results may be around 1000, but sorting on any >>>> > field might take more than 30 - 50 secs. >>>> > >>>> > I don't know what's going wrong. >>>> > >>>> > My index searcher is static object and it is getting refreshed every >>>> > minute. JSP pages directly calls the index searcher object and > >>>> performs >>>> > search. >>>> > >>>> > Regards >>>> > Ganesh >>>> > >>>> > >>>> > Send instant messages to your online friends >>>> > http://in.messenger.yahoo.com >>>> > --------------------------------------------------------------------- >>>> > To unsubscribe, e-mail: [EMAIL PROTECTED] >>>> > For additional commands, e-mail: [EMAIL PROTECTED] >>>> >>>> >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>>> For additional commands, e-mail: [EMAIL PROTECTED] >>>> >>>> Send instant messages to your online friends >>>> http://in.messenger.yahoo.com >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>>> For additional commands, e-mail: [EMAIL PROTECTED] >>>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>> For additional commands, e-mail: [EMAIL PROTECTED] >>> >>> >> Send instant messages to your online friends >> http://in.messenger.yahoo.com >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [EMAIL PROTECTED] >> For additional commands, e-mail: [EMAIL PROTECTED] >> >> > Send instant messages to your online friends http://in.messenger.yahoo.com > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > >
