Problem not fixed! I contacted infra on IRC already. ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de
> -----Original Message----- > From: Uwe Schindler [mailto:u...@thetaphi.de] > Sent: Wednesday, August 15, 2012 4:26 PM > To: java-user@lucene.apache.org > Subject: RE: howto run CheckIndex on huge index size > > I hope the problem is fixed now; this mail is just to check! It was hard to > unsubscribe because of the strange eMail. Have no idea at all... > > Uwe > > ----- > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > > > -----Original Message----- > > From: Uwe Schindler [mailto:u...@thetaphi.de] > > Sent: Wednesday, August 15, 2012 3:13 PM > > To: java-user@lucene.apache.org > > Subject: RE: howto run CheckIndex on huge index size > > > > I got is, too. As a moderator of this list, I will look into finding > > the > root cause > > and forcefully unsubscribe the failing address! > > > > ----- > > Uwe Schindler > > H.-H.-Meier-Allee 63, D-28213 Bremen > > http://www.thetaphi.de > > eMail: u...@thetaphi.de > > > > > > > -----Original Message----- > > > From: Bernd Fehling [mailto:bernd.fehl...@uni-bielefeld.de] > > > Sent: Wednesday, August 15, 2012 3:04 PM > > > To: java-user@lucene.apache.org > > > Subject: Re: howto run CheckIndex on huge index size > > > > > > > > > I guess that ulimit could be a default setting of XenServer when it > > > was > > first time > > > setup. > > > We started with about 27G. > > > I already raised ulimit -n when setting up XenServer because this > > > was also limited. > > > > > > By the way, am I the only one getting this nasty DELIVERY FAILURE > > > message from one on this list? > > > > > > Bernd > > > > > > > > > > > > > > > Am 15.08.2012 14:10, schrieb Uwe Schindler: > > > > So my blog post, last section, helped? I think the ulimits came > > > > from > > there. > > > > What distribution do you use that ulimit was actually limited - or > > > > was it some sysadmin doing this? :-) > > > > > > > > We should maybe refer to this blog post from docs or create a copy > > > > of the page inside lucene's distribution! > > > > > > > > Uwe > > > > > > > > ----- > > > > Uwe Schindler > > > > H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de > > > > eMail: u...@thetaphi.de > > > > > > > > > > > >> -----Original Message----- > > > >> From: Bernd Fehling [mailto:bernd.fehl...@uni-bielefeld.de] > > > >> Sent: Wednesday, August 15, 2012 2:07 PM > > > >> To: java-user@lucene.apache.org > > > >> Subject: Re: howto run CheckIndex on huge index size > > > >> > > > >> Hi Uwe, > > > >> > > > >> index size is: > > > >> -rw-r--r-- 1 solr users 82G 15. Aug 07:50 _2rhe.fdt > > > >> -rw-r--r-- 1 solr users 303M 15. Aug 07:50 _2rhe.fdx > > > >> -rw-r--r-- 1 solr users 1,2k 15. Aug 07:36 _2rhe.fnm > > > >> -rw-r--r-- 1 solr users 39G 15. Aug 09:04 _2rhe.frq > > > >> -rw-r--r-- 1 solr users 757M 15. Aug 09:05 _2rhe.nrm > > > >> -rw-r--r-- 1 solr users 47G 15. Aug 09:04 _2rhe.prx > > > >> -rw-r--r-- 1 solr users 257M 15. Aug 09:04 _2rhe.tii > > > >> -rw-r--r-- 1 solr users 25G 15. Aug 09:04 _2rhe.tis > > > >> -rw-r--r-- 1 solr users 300 15. Aug 09:05 segments_5u > > > >> -rw-r--r-- 1 solr users 20 15. Aug 09:05 segments.gen > > > >> > > > >> Setting ulimit -v and ulimit -m to unlimited solved this and I > > > >> can run > > > > CheckIndex > > > >> without any java options. > > > >> > > > >> The setting with ulimit should be pinned on lucene and solr pages! > > > >> > > > >> Thanks for help, > > > >> Bernd > > > >> > > > >> > > > >> Am 15.08.2012 13:34, schrieb Uwe Schindler: > > > >>> You don't get a heap-related OOM in your stack trace, it is "Map > > > >>> failed" - caused by MMapDirectory. You don't have enough virtual > > > >>> memory to map the index into address space. I think your heap is > > > >>> way too mch (-Xmx25g is way too big for any existing index and > > > >>> drives GC > > > > crazy). > > > >> How big is your index? > > > >>> > > > >>> See > > > >>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64b > > > >>> it.h > > > >>> tm > > > >>> l > > > >>> > > > >>> ----- > > > >>> Uwe Schindler > > > >>> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de > > > >>> eMail: u...@thetaphi.de > > > >>> > > > >>> > > > >>>> -----Original Message----- > > > >>>> From: Bernd Fehling [mailto:bernd.fehl...@uni-bielefeld.de] > > > >>>> Sent: Wednesday, August 15, 2012 1:25 PM > > > >>>> To: java-user@lucene.apache.org > > > >>>> Subject: howto run CheckIndex on huge index size > > > >>>> > > > >>>> > > > >>>> I'm trying to run CheckIndex as seperate tool on a large index > > > >>>> to > get > > > >>>> nice > > > >>> infos > > > >>>> about number of terms, number of tokens, ... but always get OOM > > > >> exception. > > > >>>> > > > >>>> Already have JAVA_OPTS -d64 -Xmx25g -Xms25g -Xmn6g > > > >>>> > > > >>>> > > > >>>> Any idea how to use CheckIndex on huge index size? > > > >>>> > > > >>>> > > > >>>> > > > >>>> Opening index @ /srv/www/solr/solr/solrserver/solr/data/index > > > >>>> Segments file=segments_5u numSegments=1 version=3.6.1 > > > >>>> format=FORMAT_3_1 [Lucene 3.1+] > > > >>>> 1 of 1: name=_2rhe docCount=37821132 > > > >>>> compound=false > > > >>>> hasProx=true > > > >>>> numFiles=8 > > > >>>> size (MB)=182.992,35 > > > >>>> diagnostics = {mergeFactor=1, os.version=2.6.32.23-0.3-xen, > > > >>>> os=Linux, lucene.version=3.6.1-SNAPSHOT 1325634 - solr - > > > >>>> 2012-04-13 > > > >>>> 14:08:0 6, source=merge, os.arch=amd64, > mergeMaxNumSegments=1, > > > >>>> java.version=1.6.0_29, java.vendor=Sun Microsystems Inc.} > > > >>>> no deletions > > > >>>> test: open reader.........FAILED > > > >>>> WARNING: fixIndex() would remove reference to this segment; > full > > > >>>> exception: > > > >>>> java.io.IOException: Map failed > > > >>>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748) > > > >>>> at > > > >>>> > > > >> > > > > > > org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirec > > > >>>> t > > > >>>> ory.java:271) > > > >>>> at > > > >>>> > > > >> > > > > > org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221 > > ) > > > >>>> at > > > >>>> > > > > > > org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:10 > > 4) > > > >>>> at > > > >>>> > > > org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders. > > > >>>> jav > > > >>>> a:83) > > > >>>> at > > > >>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116 > > > >>> ) > > > >>>> at > > > >>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94) > > > >>>> at > > > >>> org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:52 > > > >>> 3) > > > >>>> at > > > >>>> org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1064) > > > >>>> Caused by: java.lang.OutOfMemoryError: Map failed > > > >>>> at sun.nio.ch.FileChannelImpl.map0(Native Method) > > > >>>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745) > > > >>>> ... 8 more > > > >>>> WARNING: 1 broken segments (containing 37821132 documents) > > detected > > > >>>> WARNING: would write new segments file, and 37821132 documents > > > would > > > >>>> be lost, if -fix were specified > > > >>>> > > > >>>> > > > >>>> > > > >>>> > --------------------------------------------------------------------- > > > >>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > > >>>> For additional commands, e-mail: > > > >>>> java-user-h...@lucene.apache.org > > > >>> > > > >>> > > > >>> > --------------------------------------------------------------------- > > > >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > > >>> For additional commands, e-mail: > > > >>> java-user-h...@lucene.apache.org > > > >>> > > > >> > > > >> -- > > > >> > > ************************************************************* > > > >> Bernd Fehling Universitätsbibliothek Bielefeld > > > >> Dipl.-Inform. (FH) LibTec - Bibliothekstechnologie > > > >> Universitätsstr. 25 und Wissensmanagement > > > >> 33615 Bielefeld > > > >> Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de > > > >> > > > >> BASE - Bielefeld Academic Search Engine - www.base-search.net > > > >> > > ************************************************************* > > > >> > > > >> ----------------------------------------------------------------- > > > >> ---- To unsubscribe, e-mail: > > > >> java-user-unsubscr...@lucene.apache.org > > > >> For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > > > > > > > > ------------------------------------------------------------------ > > > > --- To unsubscribe, e-mail: > > > > java-user-unsubscr...@lucene.apache.org > > > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > > > > > > -- > > > ************************************************************* > > > Bernd Fehling Universitätsbibliothek Bielefeld > > > Dipl.-Inform. (FH) LibTec - Bibliothekstechnologie > > > Universitätsstr. 25 und Wissensmanagement > > > 33615 Bielefeld > > > Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de > > > > > > BASE - Bielefeld Academic Search Engine - www.base-search.net > > > ************************************************************* > > > > > > -------------------------------------------------------------------- > > > - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
