ulimit -v unlimited
might help, see
http://stackoverflow.com/questions/8892143/error-when-opening-a-lucene-index-map-failed
Harald.
On 18.08.2014 13:10, Shlomit Rosen wrote:
Hi all,
Using lucene 3.6.2, we are trying to search a pretty small collection.
To open the directory we use Mmap since we are running on a 64 bit linux
machine, and we usually get much better results than using SimpleFS or
NIO.
Although the collection is only a few GB in size, we are getting an OOM
exception right from the start:
IQQS0003E The searchable object for collection
/IndexStore/IndexArea/OS2_Document_20140813023606_9970347625CD4483BD700CA08571491C
cannot be loaded.
Causes of the problem:
IQQS0004E The reader for index
/IndexStore/IndexArea/OS2_Document_20140813023606_9970347625CD4483BD700CA08571491C/data/text
cannot be created.
IQQG0020E java.io.IOException: Map failed
IQQG0020E java.lang.OutOfMemoryError: Map failed
sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:784)
org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:270)
org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:220)
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:96)
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:105)
org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:27)
org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:78)
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:709)
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:72)
org.apache.lucene.index.IndexReader.open(IndexReader.java:375)
...
Collection details:
File Name File Size
--------- ---------
_9d.prx 700520240 bytes (668.07 MB)
_6a.prx 688987818 bytes (657.07 MB)
_im.prx 678185165 bytes (646.77 MB)
_37.prx 672422515 bytes (641.27 MB)
_rv.prx 672034961 bytes (640.90 MB)
_fj.prx 669692961 bytes (638.67 MB)
_cg.prx 666485509 bytes (635.61 MB)
_os.prx 661239341 bytes (630.61 MB)
_lp.prx 656960684 bytes (626.53 MB)
_9d.frq 237580704 bytes (226.57 MB)
_6a.frq 233601024 bytes (222.78 MB)
_im.frq 229835093 bytes (219.19 MB)
_37.frq 227920880 bytes (217.36 MB)
_rv.frq 227857650 bytes (217.30 MB)
_fj.frq 227030974 bytes (216.51 MB)
_cg.frq 225955651 bytes (215.49 MB)
_os.frq 224161119 bytes (213.78 MB)
_lp.frq 222748045 bytes (212.43 MB)
_tl.cfs 104218657 bytes (99.39 MB)
_s2.cfs 99241207 bytes (94.64 MB)
_ta.cfs 96972944 bytes (92.48 MB)
_sz.cfs 95444312 bytes (91.02 MB)
_sd.cfs 95334907 bytes (90.92 MB)
_so.cfs 94020302 bytes (89.66 MB)
_9d.fdt 67804161 bytes (64.66 MB)
_6a.fdt 66670600 bytes (63.58 MB)
_im.fdt 65635643 bytes (62.60 MB)
_37.fdt 65082068 bytes (62.07 MB)
_rv.fdt 65027960 bytes (62.02 MB)
_fj.fdt 64820141 bytes (61.82 MB)
_cg.fdt 64519880 bytes (61.53 MB)
_os.fdt 63995985 bytes (61.03 MB)
_lp.fdt 63578365 bytes (60.63 MB)
_ts.cfs 16675689 bytes (15.90 MB)
_tq.cfs 16630852 bytes (15.86 MB)
_tn.cfs 10548412 bytes (10.06 MB)
_to.cfs 9935882 bytes (9.48 MB)
_tp.cfs 9667762 bytes (9.22 MB)
_tm.cfs 9473559 bytes (9.03 MB)
_tk.cfs 8742824 bytes (8.34 MB)
_9d.tis 8027312 bytes (7.66 MB)
_6a.tis 7898747 bytes (7.53 MB)
_im.tis 7775053 bytes (7.41 MB)
_37.tis 7714470 bytes (7.36 MB)
_rv.tis 7702330 bytes (7.35 MB)
_fj.tis 7682979 bytes (7.33 MB)
_cg.tis 7649182 bytes (7.29 MB)
_os.tis 7585214 bytes (7.23 MB)
_lp.tis 7537160 bytes (7.19 MB)
_tt.cfs 1832496 bytes (1.75 MB)
_9d.fdx 858540 bytes (838.42 KB)
_6a.fdx 844204 bytes (824.42 KB)
_im.fdx 831084 bytes (811.61 KB)
_37.fdx 824068 bytes (804.75 KB)
_rv.fdx 823380 bytes (804.08 KB)
_fj.fdx 820756 bytes (801.52 KB)
_cg.fdx 816972 bytes (797.82 KB)
_os.fdx 810332 bytes (791.34 KB)
_lp.fdx 805044 bytes (786.18 KB)
_tr.cfs 640731 bytes (625.71 KB)
_tu.cfs 419728 bytes (409.89 KB)
_9d.nrm 214638 bytes (209.61 KB)
_6a.nrm 211054 bytes (206.11 KB)
_im.nrm 207774 bytes (202.90 KB)
_37.nrm 206020 bytes (201.19 KB)
_rv.nrm 205848 bytes (201.02 KB)
_fj.nrm 205192 bytes (200.38 KB)
_cg.nrm 204246 bytes (199.46 KB)
_os.nrm 202586 bytes (197.84 KB)
_lp.nrm 201264 bytes (196.55 KB)
_9d.tii 84483 bytes (82.50 KB)
_6a.tii 83166 bytes (81.22 KB)
_im.tii 81863 bytes (79.94 KB)
_37.tii 81277 bytes (79.37 KB)
_rv.tii 81113 bytes (79.21 KB)
_fj.tii 80939 bytes (79.04 KB)
_cg.tii 80709 bytes (78.82 KB)
_os.tii 79774 bytes (77.90 KB)
_lp.tii 79369 bytes (77.51 KB)
segments_qj 5977 bytes (5.84 KB)
_37.fnm 86 bytes
_6a.fnm 86 bytes
_9d.fnm 86 bytes
_cg.fnm 86 bytes
_fj.fnm 86 bytes
_im.fnm 86 bytes
_lp.fnm 86 bytes
_os.fnm 86 bytes
_rv.fnm 86 bytes
Does this seem like a reasonable behavior or is this a bug?
It seems to me that with this small segments, memory should not be an
issue...?
We haven't seen this exception in previous testing.
In the lucene documentation I found that tweaking the chunk size might
help,
But this code goes into customer production environment, and this option
is not feasible.
Is there anything else I can check ahead of time (except for if this is a
32/64 bit platform) to help me decide
Whether or not Mmap should be used?
Thanks in advance,
Shlomit
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
