RE: debugging growing index size

2015-11-14 Thread Rob Audenaerde
ier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > > -Original Message- > > From: Michael McCandless [mailto:luc...@mikemccandless.com] > > Sent: Friday, November 13, 2015 8:30 PM > > To: Lucene Users > > Subject: R

RE: debugging growing index size

2015-11-14 Thread Uwe Schindler
ndless [mailto:luc...@mikemccandless.com] > Sent: Friday, November 13, 2015 8:30 PM > To: Lucene Users > Subject: Re: debugging growing index size > > So with MMapDir at defaults (unmap is enabled) you see old files, with > no open file handles as reported by lsof, still existing i

Re: debugging growing index size

2015-11-13 Thread Michael McCandless
D-28213 Bremen >>> http://www.thetaphi.de >>> eMail: u...@thetaphi.de >>> >>> > -Original Message- >>> > From: will martin [mailto:wmartin...@gmail.com] >>> > Sent: Friday, November 13, 2015 6:04 PM >>> > To: java-user@luc

Re: debugging growing index size

2015-11-13 Thread Michael McCandless
men >> http://www.thetaphi.de >> eMail: u...@thetaphi.de >> >> > -Original Message- >> > From: will martin [mailto:wmartin...@gmail.com] >> > Sent: Friday, November 13, 2015 6:04 PM >> > To: java-user@lucene.apache.org >> > Subject: Re:

RE: debugging growing index size

2015-11-13 Thread Rob Audenaerde
http://www.thetaphi.de > eMail: u...@thetaphi.de > > > -Original Message- > > From: will martin [mailto:wmartin...@gmail.com] > > Sent: Friday, November 13, 2015 6:04 PM > > To: java-user@lucene.apache.org > > Subject: Re: debugging growing index size > &g

RE: debugging growing index size

2015-11-13 Thread Uwe Schindler
: will martin [mailto:wmartin...@gmail.com] > Sent: Friday, November 13, 2015 6:04 PM > To: java-user@lucene.apache.org > Subject: Re: debugging growing index size > > Hi Rob: > > > Doesn’t this look like known SE issue JDK-4724038 and discussed by Peter > Levart and Uwe Sch

RE: debugging growing index size

2015-11-13 Thread Uwe Schindler
m: Rob Audenaerde [mailto:rob.audenae...@gmail.com] > Sent: Friday, November 13, 2015 5:24 PM > To: java-user@lucene.apache.org > Subject: Re: debugging growing index size > > I'm currently running using NIOFS. It seems to prevent the issue from > appearing. > > This is a s

Re: debugging growing index size

2015-11-13 Thread will martin
Hi Rob: Doesn’t this look like known SE issue JDK-4724038 and discussed by Peter Levart and Uwe Schindler on a lucene-dev thread 9/9/2015? MappedByteBuffer …. what OS are you on Rob? What JVM? http://bugs.java.com/view_bug.do?bug_id=4724038 http://mail-archives.apache.org/mod_mbox/lucene-dev/

Re: debugging growing index size

2015-11-13 Thread Rob Audenaerde
I'm currently running using NIOFS. It seems to prevent the issue from appearing. This is a second run (with applied deletes etc) raudenaerd@:/<6>index/index$sudo ls -lSra *.dvd -rw-r--r--. 1 apache apache 7993 Nov 13 16:09 _y_Lucene50_0.dvd -rw-r--r--. 1 apache apache 39048886 Nov 13 17:12

Re: debugging growing index size

2015-11-13 Thread Rob Audenaerde
I got the data (beware, it is about 180MB download, xz-zipped, unpacked it is about 6.6 GB). Unfortunately, I accidentally restarted the application so the index-files and lsof output could not be determined for this run. Hopefully the infoStream log with the extra logging will provide enough inf

Re: debugging growing index size

2015-11-12 Thread Michael McCandless
Hi Rob, A couple more things: Can you print the value of MMapDirectory.UNMAP_SUPPORTED? Also, can you try your test using NIOFSDirectory instead? Curious if that changes things... Mike McCandless http://blog.mikemccandless.com On Thu, Nov 12, 2015 at 7:28 AM, Rob Audenaerde wrote: > Curiou

Re: debugging growing index size

2015-11-12 Thread Rob Audenaerde
Curious indeed! I will turn on the IndexFileDeleter.VERBOSE_REF_COUNTS and recreate the logs. Will get back with them in a day hopefully. Thanks for the extra logging! -Rob On Thu, Nov 12, 2015 at 11:34 AM, Michael McCandless < luc...@mikemccandless.com> wrote: > Hmm, curious. > > I looked at

Re: debugging growing index size

2015-11-12 Thread Michael McCandless
Hmm, curious. I looked at the [large] infoStream output and I see segment _3ou7 present on init of IW, a few getReader calls referencing it, then a forceMerge that indeed merges it away, yet I do NOT see IW attempting deletion of its files. And indeed I see plenty (too many: many times per second

debugging growing index size

2015-11-11 Thread Rob Audenaerde
Hi all, I'm still debugging the growing-index size. I think closing index readers might help (work in progress), but I can't really see them holding on to files (at least, using lsof ). Restarting the application sheds some light, I see logging on files that are no longer referenced. What I see i