Thanks Mike... As i explained before, I created a small app, which loads all the db, does term search (using term query) and calculates the memory consumption. I tried this with divisor value but after 100 there seems to be no difference.
Just load the database with different divisor value and then perform warming using TermQuery. I feel there should be some difference when opening with different value. But it didn't. I will try to perform the same using profiler. Regards Ganesh ----- Original Message ----- From: "Michael McCandless" <luc...@mikemccandless.com> To: <java-user@lucene.apache.org> Sent: Sunday, December 06, 2009 3:14 PM Subject: Re: IndexDivisor You really have to use a profiler, to trust the RAM measurements. Certain queries, eg function queries, can load the field cache. What kind of queries are you running? Mike On Fri, Dec 4, 2009 at 9:11 AM, Ganesh <emailg...@yahoo.co.in> wrote: > I didn't run with profiler. I created a test app and run that.. I am opening > multiple database. > > IndexReader opened with IndexDivisor: 100 //Open the reader with the divisor > value > TermCount: 7046764 //Available unique terms in the db > Warmup done: //warming up > Memory: UsedMemory: 4 :FreeMemory:788 //Memory consumed after GC. > > I couldn't see the memory difference for the value btw 100 to 10000. I will > try to run with profiler and get back to you. > > Other than terms what all the other things which consumes memory. There is no > deletions, no norms and i am not doing sorting > > Regards > Ganesh > > > ----- Original Message ----- > From: "Michael McCandless" <luc...@mikemccandless.com> > To: <java-user@lucene.apache.org> > Sent: Friday, December 04, 2009 7:01 PM > Subject: Re: IndexDivisor > > > I'm confused -- what are these attachments? Output from a memory profiler? > > Can you post the app you created? > > Mike > > On Fri, Dec 4, 2009 at 12:24 AM, Ganesh <emailg...@yahoo.co.in> wrote: >> Thanks mike.. >> >> Please find the attached file. I ran the testing for 1,100,1000,10000 >> divisor value. There is difference from 1 to 100 but there is no difference >> between 100 to 10000. >> >> I created a new application, in which i opened all reader and searcher and >> warmed up. Sleep for a minute and close all the searcher and reader. >> >> There is no deletions, no norms and i am not doing sorting. >> >> Regards >> Ganesh >> >> >> >> ----- Original Message ----- >> From: "Michael McCandless" <luc...@mikemccandless.com> >> To: <java-user@lucene.apache.org> >> Sent: Thursday, December 03, 2009 9:35 PM >> Subject: Re: IndexDivisor >> >> >> Can you run w/ a memory profiler? I don't trust that gc is truly running. >> >> Mike >> >> On Thu, Dec 3, 2009 at 10:47 AM, Ganesh <emailg...@yahoo.co.in> wrote: >>> I am doing GC before calculating the memory. Even i set my indexdivisor to >>> 10000 but there seems to be no change. >>> >>> Below are my stats >>> IndexDivisor Memory >>> -1 7 MB >>> 1 486 MB >>> 100 180 MB >>> 1000 176 MB. >>> 10000 176MB >>> >>> Regards >>> Ganesh >>> >>> ----- Original Message ----- >>> From: "Danil ŢORIN" <torin...@gmail.com> >>> To: <java-user@lucene.apache.org> >>> Sent: Thursday, December 03, 2009 8:35 PM >>> Subject: Re: IndexDivisor >>> >>> >>> Run System.gc() exactly before measuring memory usage. >>> >>> On sun jvm it will FORCE gc (unless DisableExplicitGC is used). >>> >>> >>> On Thu, Dec 3, 2009 at 16:30, Ganesh <emailg...@yahoo.co.in> wrote: >>>> Thanks mike. >>>> >>>> I am opening the reader and warming it up and then calculating the memory >>>> consumed. >>>> long usedMemory = runtime.totalMemory() - runtime.freeMemory(); >>>> >>>> Regards >>>> Ganesh >>>> >>>> ----- Original Message ----- >>>> From: "Michael McCandless" <luc...@mikemccandless.com> >>>> To: <java-user@lucene.apache.org> >>>> Sent: Thursday, December 03, 2009 6:22 PM >>>> Subject: Re: IndexDivisor >>>> >>>> >>>> On Thu, Dec 3, 2009 at 7:15 AM, Ganesh <emailg...@yahoo.co.in> wrote: >>>> >>>>> Below are my stats >>>>> IndexDivisor Memory >>>>> -1 7 MB >>>>> 1 486 MB >>>>> 100 180 MB >>>>> 1000 176 MB. >>>> >>>> Do you simply create the IndexWriter & IndexReader, but do no >>>> searching/indexing? >>>> >>>> How are you measuring RAM? You should use a RAM profiler (eg YourKit) >>>> to get the "real" usage, not counting garbage that's yet to be >>>> collected. >>>> >>>> My guess is the shear process of Lucene scanning your terms index >>>> creates lots of garbage and you're measuring the RAM consumed by that >>>> garbage. The garbage should be harmless (it'll eventually get GCd & >>>> reused). >>>> >>>>> 1. Whether term vector will consume RAM. I think it should only consume >>>>> disk space. >>>> >>>> No RAM is consumed. >>>> >>>>> 2. By setting RAM buffer for IW, Does it will allocate memory Or based on >>>>> the usage the memory will be allocated / increased. >>>> >>>> Only a little RAM is allocated up front, I think. It's only as you >>>> start indexing docs that the big RAM is really allocated. After a >>>> flush the RAM is then reused (ie, not freed) for subsequent segments. >>>> >>>> Mike >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>>> >>>> Send instant messages to your online friends http://in.messenger.yahoo.com >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >>> Send instant messages to your online friends http://in.messenger.yahoo.com >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > Send instant messages to your online friends http://in.messenger.yahoo.com > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org Send instant messages to your online friends http://in.messenger.yahoo.com --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
