Thanks Uwe for the insight ! Also, is it advisable to set the lower chunk size for smaller indexes, like below or let Lucene/OS manage by itself. I am just guessing that assigning lower value to smaller index will make sure that bigger index are getting higher mmap address space.
*Index Name Total Records Size (in GB) What should be the max. or optimal chunk size ?* Address index 106,192,963.00 65 1 GiB Name index 97,924,594.00 44 1 GiB GovtId index 81,178,958.00 11 512 MB Phone index 169,691,376.00 14 512 MB Email index 46,602,090.00 5 256 MB Date index 77,243,714.00 6.5 256 MB Thanks On Sat, Sep 27, 2014 at 3:40 AM, Uwe Schindler <u...@thetaphi.de> wrote: > Hi, > > 1 GiB is the maximum possible. The chunk size is only applicable for 32 > bit JDKs because of limited address space. > > Uwe > > ----- > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > > > -----Original Message----- > > From: Gaurav gupta [mailto:gupta.gaurav0...@gmail.com] > > Sent: Friday, September 26, 2014 9:12 PM > > To: java-user@lucene.apache.org > > Subject: Optimum Lucene’s MMapDirectory size on 64bit OS > > > > Hi, > > > > As per the post "The Generics Policeman Blog > > <http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on- > > 64bit.html>" > > , I am using the MMapDirectory for faster access(search and update > > operations ,mainly search) of Lucene 4.8.1 index files. I am > contemplating > > what is the optimal maximum MMap value for my indexes. Is default i.e. 1 > > GB > > (1 << 30) or higher? > > > > I have 6 indexes of size varying from 65GB to 6 GB. Currently, I am > using 1 GB > > as maxChunkSize : - *MMapDirectory(file, null, 1<<30) *for all indexes. > > But thinking of specifying the higher value for mmap (1 GB or higher) for > > bigger index having 65GB size and lower value (0.5 GB or less) for > smaller > > index having size of 6 GB. Any suggestion/guidance on it ? > > > > Also, per blog mmap is not a size of physical memory allocation but just > a > > address space to map the index files. How to allocate more RAM to index > > files for better performance? We have enough RAM free out of 64 GB. Per > > blog, one should use the mmap file, like - *MMapDirectory(file, null, > > 1<<30) *and let OS manage the physical memory allocation for the index > > files. Is my understanding correct ? > > > > The Generics Policeman Blog > > <http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on- > > 64bit.html> :- > > > > - *MMapDirectory does not consume additional memory and the size of > > mapped index files is not limited by the physical memory available on > your > > server.* By mmap() files, we only reserve address space not memory! > > Remember, address space on 64bit platforms is for free! > > > > Thanks > > Gaurav > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
