use MMapDirectory with tmpfs?

2013-10-21 Thread Reg
Hi there, If I put Lucene segments on tmpfs and use MMapDirectory to access them, would the kernel be so dumb to load the files from tmpfs to another copy of file system cache before map it to the virtual address? Or it just maps tmpfs to the virtual address directly? I tend to believe it's the

Re: problem found with DiskDocValuesFormat

2013-10-21 Thread Michael McCandless
It's perfectly fine, and recommended, to reuse a thread across different queries (ie, use a thread pool in your app, up above Lucene). The ThreadLocals used in SegmentCoreReaders should not interfere or cause problems with that: they can easily be re-used across queries. Maybe you can boil down t

Re: problem found with DiskDocValuesFormat

2013-10-21 Thread Duke DAI
Hi Mike, My scenario, query thread from a ThreadPool will be used to execute query. So thread must have to be reused to handle various queries. Now that SegmentCoreReaders uses ThreadLocal to hold per-thread instance, I think some private variables must belong to the given thread(file offset? I di

Re: problem found with DiskDocValuesFormat

2013-10-21 Thread Michael McCandless
Can you describe what problem you are actually hitting? The purpose of docValuesLocal is to hold the per-Thread instance of each doc values, and re-use it when that thread comes back again asking for the same doc values. Mike McCandless http://blog.mikemccandless.com On Mon, Oct 21, 2013 at 6:

Re: problem found with DiskDocValuesFormat

2013-10-21 Thread Duke DAI
Hi guys, Seems I have the same problem with Lucene45DocValuesFormat, no problem with MemoryDocValuesFormat. The problem I encountered with Lucene4.4 is with DiskDocValuesFormat, no with Lucene42DocValuesFormat. I dig into a little and found the superficial cause. In SegmentCoreReaders, there is a