I guess that ulimit could be a default setting of XenServer when it was first time setup. We started with about 27G. I already raised ulimit -n when setting up XenServer because this was also limited.
By the way, am I the only one getting this nasty DELIVERY FAILURE message from one on this list? Bernd Am 15.08.2012 14:10, schrieb Uwe Schindler: > So my blog post, last section, helped? I think the ulimits came from there. > What distribution do you use that ulimit was actually limited - or was it > some sysadmin doing this? :-) > > We should maybe refer to this blog post from docs or create a copy of the > page inside lucene's distribution! > > Uwe > > ----- > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > >> -----Original Message----- >> From: Bernd Fehling [mailto:bernd.fehl...@uni-bielefeld.de] >> Sent: Wednesday, August 15, 2012 2:07 PM >> To: java-user@lucene.apache.org >> Subject: Re: howto run CheckIndex on huge index size >> >> Hi Uwe, >> >> index size is: >> -rw-r--r-- 1 solr users 82G 15. Aug 07:50 _2rhe.fdt >> -rw-r--r-- 1 solr users 303M 15. Aug 07:50 _2rhe.fdx >> -rw-r--r-- 1 solr users 1,2k 15. Aug 07:36 _2rhe.fnm >> -rw-r--r-- 1 solr users 39G 15. Aug 09:04 _2rhe.frq >> -rw-r--r-- 1 solr users 757M 15. Aug 09:05 _2rhe.nrm >> -rw-r--r-- 1 solr users 47G 15. Aug 09:04 _2rhe.prx >> -rw-r--r-- 1 solr users 257M 15. Aug 09:04 _2rhe.tii >> -rw-r--r-- 1 solr users 25G 15. Aug 09:04 _2rhe.tis >> -rw-r--r-- 1 solr users 300 15. Aug 09:05 segments_5u >> -rw-r--r-- 1 solr users 20 15. Aug 09:05 segments.gen >> >> Setting ulimit -v and ulimit -m to unlimited solved this and I can run > CheckIndex >> without any java options. >> >> The setting with ulimit should be pinned on lucene and solr pages! >> >> Thanks for help, >> Bernd >> >> >> Am 15.08.2012 13:34, schrieb Uwe Schindler: >>> You don't get a heap-related OOM in your stack trace, it is "Map >>> failed" - caused by MMapDirectory. You don't have enough virtual >>> memory to map the index into address space. I think your heap is way >>> too mch (-Xmx25g is way too big for any existing index and drives GC > crazy). >> How big is your index? >>> >>> See >>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.htm >>> l >>> >>> ----- >>> Uwe Schindler >>> H.-H.-Meier-Allee 63, D-28213 Bremen >>> http://www.thetaphi.de >>> eMail: u...@thetaphi.de >>> >>> >>>> -----Original Message----- >>>> From: Bernd Fehling [mailto:bernd.fehl...@uni-bielefeld.de] >>>> Sent: Wednesday, August 15, 2012 1:25 PM >>>> To: java-user@lucene.apache.org >>>> Subject: howto run CheckIndex on huge index size >>>> >>>> >>>> I'm trying to run CheckIndex as seperate tool on a large index to get >>>> nice >>> infos >>>> about number of terms, number of tokens, ... but always get OOM >> exception. >>>> >>>> Already have JAVA_OPTS -d64 -Xmx25g -Xms25g -Xmn6g >>>> >>>> >>>> Any idea how to use CheckIndex on huge index size? >>>> >>>> >>>> >>>> Opening index @ /srv/www/solr/solr/solrserver/solr/data/index >>>> Segments file=segments_5u numSegments=1 version=3.6.1 >>>> format=FORMAT_3_1 [Lucene 3.1+] >>>> 1 of 1: name=_2rhe docCount=37821132 >>>> compound=false >>>> hasProx=true >>>> numFiles=8 >>>> size (MB)=182.992,35 >>>> diagnostics = {mergeFactor=1, os.version=2.6.32.23-0.3-xen, >>>> os=Linux, lucene.version=3.6.1-SNAPSHOT 1325634 - solr - 2012-04-13 >>>> 14:08:0 6, source=merge, os.arch=amd64, mergeMaxNumSegments=1, >>>> java.version=1.6.0_29, java.vendor=Sun Microsystems Inc.} >>>> no deletions >>>> test: open reader.........FAILED >>>> WARNING: fixIndex() would remove reference to this segment; full >>>> exception: >>>> java.io.IOException: Map failed >>>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748) >>>> at >>>> >> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirec >>>> t >>>> ory.java:271) >>>> at >>>> >> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221) >>>> at >>>> > org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:104) >>>> at >>>> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders. >>>> jav >>>> a:83) >>>> at >>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) >>>> at >>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94) >>>> at >>> org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:523) >>>> at >>>> org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1064) >>>> Caused by: java.lang.OutOfMemoryError: Map failed >>>> at sun.nio.ch.FileChannelImpl.map0(Native Method) >>>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745) >>>> ... 8 more >>>> WARNING: 1 broken segments (containing 37821132 documents) detected >>>> WARNING: would write new segments file, and 37821132 documents would >>>> be lost, if -fix were specified >>>> >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >> >> -- >> ************************************************************* >> Bernd Fehling Universitätsbibliothek Bielefeld >> Dipl.-Inform. (FH) LibTec - Bibliothekstechnologie >> Universitätsstr. 25 und Wissensmanagement >> 33615 Bielefeld >> Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de >> >> BASE - Bielefeld Academic Search Engine - www.base-search.net >> ************************************************************* >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > -- ************************************************************* Bernd Fehling Universitätsbibliothek Bielefeld Dipl.-Inform. (FH) LibTec - Bibliothekstechnologie Universitätsstr. 25 und Wissensmanagement 33615 Bielefeld Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de BASE - Bielefeld Academic Search Engine - www.base-search.net ************************************************************* --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
