Visiting all the index terms, which must be done for any divisor != -1, generates a good amount of garbage. So if you're including that garbage in your memory measurement, that would explain what you're seeing, and switching to a memory profiler should show the true RAM usage.
Mike On Tue, Dec 8, 2009 at 6:29 AM, Ganesh <emailg...@yahoo.co.in> wrote: > Thanks Mike... > > As i explained before, I created a small app, which loads all the db, does > term search (using term query) and calculates the memory consumption. I tried > this with divisor value but after 100 there seems to be no difference. > > Just load the database with different divisor value and then perform warming > using TermQuery. I feel there should be some difference when opening with > different value. But it didn't. I will try to perform the same using profiler. > > Regards > Ganesh > > > ----- Original Message ----- > From: "Michael McCandless" <luc...@mikemccandless.com> > To: <java-user@lucene.apache.org> > Sent: Sunday, December 06, 2009 3:14 PM > Subject: Re: IndexDivisor > > > You really have to use a profiler, to trust the RAM measurements. > > Certain queries, eg function queries, can load the field cache. What > kind of queries are you running? > > Mike > > On Fri, Dec 4, 2009 at 9:11 AM, Ganesh <emailg...@yahoo.co.in> wrote: >> I didn't run with profiler. I created a test app and run that.. I am opening >> multiple database. >> >> IndexReader opened with IndexDivisor: 100 //Open the reader with the divisor >> value >> TermCount: 7046764 //Available unique terms in the db >> Warmup done: //warming up >> Memory: UsedMemory: 4 :FreeMemory:788 //Memory consumed after GC. >> >> I couldn't see the memory difference for the value btw 100 to 10000. I will >> try to run with profiler and get back to you. >> >> Other than terms what all the other things which consumes memory. There is >> no deletions, no norms and i am not doing sorting >> >> Regards >> Ganesh >> >> >> ----- Original Message ----- >> From: "Michael McCandless" <luc...@mikemccandless.com> >> To: <java-user@lucene.apache.org> >> Sent: Friday, December 04, 2009 7:01 PM >> Subject: Re: IndexDivisor >> >> >> I'm confused -- what are these attachments? Output from a memory profiler? >> >> Can you post the app you created? >> >> Mike >> >> On Fri, Dec 4, 2009 at 12:24 AM, Ganesh <emailg...@yahoo.co.in> wrote: >>> Thanks mike.. >>> >>> Please find the attached file. I ran the testing for 1,100,1000,10000 >>> divisor value. There is difference from 1 to 100 but there is no difference >>> between 100 to 10000. >>> >>> I created a new application, in which i opened all reader and searcher and >>> warmed up. Sleep for a minute and close all the searcher and reader. >>> >>> There is no deletions, no norms and i am not doing sorting. >>> >>> Regards >>> Ganesh >>> >>> >>> >>> ----- Original Message ----- >>> From: "Michael McCandless" <luc...@mikemccandless.com> >>> To: <java-user@lucene.apache.org> >>> Sent: Thursday, December 03, 2009 9:35 PM >>> Subject: Re: IndexDivisor >>> >>> >>> Can you run w/ a memory profiler? I don't trust that gc is truly running. >>> >>> Mike >>> >>> On Thu, Dec 3, 2009 at 10:47 AM, Ganesh <emailg...@yahoo.co.in> wrote: >>>> I am doing GC before calculating the memory. Even i set my indexdivisor to >>>> 10000 but there seems to be no change. >>>> >>>> Below are my stats >>>> IndexDivisor Memory >>>> -1 7 MB >>>> 1 486 MB >>>> 100 180 MB >>>> 1000 176 MB. >>>> 10000 176MB >>>> >>>> Regards >>>> Ganesh >>>> >>>> ----- Original Message ----- >>>> From: "Danil ŢORIN" <torin...@gmail.com> >>>> To: <java-user@lucene.apache.org> >>>> Sent: Thursday, December 03, 2009 8:35 PM >>>> Subject: Re: IndexDivisor >>>> >>>> >>>> Run System.gc() exactly before measuring memory usage. >>>> >>>> On sun jvm it will FORCE gc (unless DisableExplicitGC is used). >>>> >>>> >>>> On Thu, Dec 3, 2009 at 16:30, Ganesh <emailg...@yahoo.co.in> wrote: >>>>> Thanks mike. >>>>> >>>>> I am opening the reader and warming it up and then calculating the memory >>>>> consumed. >>>>> long usedMemory = runtime.totalMemory() - runtime.freeMemory(); >>>>> >>>>> Regards >>>>> Ganesh >>>>> >>>>> ----- Original Message ----- >>>>> From: "Michael McCandless" <luc...@mikemccandless.com> >>>>> To: <java-user@lucene.apache.org> >>>>> Sent: Thursday, December 03, 2009 6:22 PM >>>>> Subject: Re: IndexDivisor >>>>> >>>>> >>>>> On Thu, Dec 3, 2009 at 7:15 AM, Ganesh <emailg...@yahoo.co.in> wrote: >>>>> >>>>>> Below are my stats >>>>>> IndexDivisor Memory >>>>>> -1 7 MB >>>>>> 1 486 MB >>>>>> 100 180 MB >>>>>> 1000 176 MB. >>>>> >>>>> Do you simply create the IndexWriter & IndexReader, but do no >>>>> searching/indexing? >>>>> >>>>> How are you measuring RAM? You should use a RAM profiler (eg YourKit) >>>>> to get the "real" usage, not counting garbage that's yet to be >>>>> collected. >>>>> >>>>> My guess is the shear process of Lucene scanning your terms index >>>>> creates lots of garbage and you're measuring the RAM consumed by that >>>>> garbage. The garbage should be harmless (it'll eventually get GCd & >>>>> reused). >>>>> >>>>>> 1. Whether term vector will consume RAM. I think it should only consume >>>>>> disk space. >>>>> >>>>> No RAM is consumed. >>>>> >>>>>> 2. By setting RAM buffer for IW, Does it will allocate memory Or based >>>>>> on the usage the memory will be allocated / increased. >>>>> >>>>> Only a little RAM is allocated up front, I think. It's only as you >>>>> start indexing docs that the big RAM is really allocated. After a >>>>> flush the RAM is then reused (ie, not freed) for subsequent segments. >>>>> >>>>> Mike >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>>>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>>>> >>>>> Send instant messages to your online friends http://in.messenger.yahoo.com >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>>>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>>>> >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>>> >>>> Send instant messages to your online friends http://in.messenger.yahoo.com >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> Send instant messages to your online friends http://in.messenger.yahoo.com >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > Send instant messages to your online friends http://in.messenger.yahoo.com > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
