DirectPostingsFormat holds all postings in RAM, uncompressed, as simple java arrays. But it's quite RAM heavy...
The hotspots may also be in the queries you are running ... maybe you can describe more how you're using Lucene? Mike McCandless http://blog.mikemccandless.com On Thu, Oct 17, 2013 at 10:56 AM, Igor Shalyminov <ishalymi...@yandex-team.ru> wrote: > Hello! > > I've tried two approaches: 1) RAMDirectory, 2) MMapDirectory + tmpfs. Both > work the same for me (the same bad:( ). > Thus, I think my problem is not disk access (although I always see > getPayload() in the VisualVM top). > So, maybe the hard part in the postings traversal is decompression? > Are there Lucene codecs which use light postings compression (maybe none at > all)? > > And, getting back to in-memory index topic, is lucene.codecs.memory somewhat > similar to RAMDirectory? > > -- > Best Regards, > Igor > > 10.10.2013, 03:01, "Vitaly Funstein" <vfunst...@gmail.com>: >> I don't think you want to load indexes of this size into a RAMDirectory. >> The reasons have been listed multiple times here... in short, just use >> MMapDirectory. >> >> On Wed, Oct 9, 2013 at 3:17 PM, Igor Shalyminov >> <ishalymi...@yandex-team.ru>wrote: >> >>> Hello! >>> >>> I need to perform an experiment of loading the entire index in RAM and >>> seeing how the search performance changes. >>> My index has TermVectors with payload and position info, StoredFields, and >>> DocValues. It takes ~30GB on disk (the server has 48). >>> >>> _indexDirectoryReader = DirectoryReader.open(RAMDirectory.open(new >>> File(_indexDirectory))); >>> >>> Is the line above the only thing I have to do to complete my goal? >>> >>> And also: >>> - will all the data be loaded in the RAM right after opening, or during >>> the reading stage? >>> - will the index data be stored in RAM as it is on disk, or will it be >>> uncompressed first? >>> >>> -- >>> Best Regards, >>> Igor >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: java-user-h...@lucene.apache.org > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
