Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

So, just cat will do this. Thanks From: Robert Muir Sent: Tuesday, February 23, 2021 4:45 PM To: Baris Kazar Cc: java-user Subject: Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory) The preload isn't magical. It only "reads in

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

e url / site that i can look at for preload? > > Thanks for the explanations. This thread will be useful for many folks i > believe. > > Best regards > > > On 2/23/21 4:15 PM, Robert Muir wrote: > > > > On Tue, Feb 23, 2021 at 4:07 PM wrote: > >> What i w

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

ve. Best regards On 2/23/21 4:15 PM, Robert Muir wrote: On Tue, Feb 23, 2021 at 4:07 PM <mailto:baris.ka...@oracle.com>> wrote: What i want to achieve: Problem statement: base case is disk based Lucene index with FSDirectory speedup case was supposed to be in memor

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

On Tue, Feb 23, 2021 at 4:07 PM wrote: > What i want to achieve: Problem statement: > > base case is disk based Lucene index with FSDirectory > > speedup case was supposed to be in memory Lucene index with MMapDirectory > On 64-bit systems, FSDirectory just invokes MMapDirecto

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

better in that, too: ie, cold start. What i want to achieve: Problem statement: base case is disk based Lucene index with FSDirectory speedup case was supposed to be in memory Lucene index with MMapDirectory Uwe mentioned tmpfs will help. i will try that next. I thought preload was not

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

Lucene index with FSDirectory speedup case was supposed to be in memory Lucene index with MMapDirectory Uwe mentioned tmpfs will help. i will try that next. Thanks On 2/23/21 3:54 PM, Robert Muir wrote: speedup over what? You are probably already using MMapDirectory (it is the default). So I

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

speedup over what? You are probably already using MMapDirectory (it is the default). So I don't know what you are trying to achieve, but giving lots of memory to your java process is not going to help. If you just want to prevent the first few queries to a fresh cold machine instance from being sl

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

Thanks but then how will MMapDirectory help gain speedup? i will try tmpfs and see what happens. i was expecting to get on order of magnitude of speedup from already very fast on disk Lucene indexes. So i was expecting really really really fast response with MMapDirectory. Thanks On 2/23/21

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

Don't give gobs of memory to your java process, you will just make things slower. The kernel will cache your index files. On Tue, Feb 23, 2021 at 1:45 PM wrote: > Ok, but how is this MMapDirectory used then? > > Best regards > > > On 2/23/21 7:03 AM, Robert Muir wrote: > > > > > > On Tue, Feb 23

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

As Uwe suggested some time ago, tmpfs file system usage with MMapDirectory is the only way to get high speedup wrt on disk Lucene index, right? Best regards On 2/23/21 1:44 PM, baris.ka...@oracle.com wrote: Ok, but how is this MMapDirectory used then? Best regards On 2/23/21 7:03 AM, Rob

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

Ok, but how is this MMapDirectory used then? Best regards On 2/23/21 7:03 AM, Robert Muir wrote: On Tue, Feb 23, 2021 at 2:30 AM > wrote: Hi,-   I tried MMapDirectory and i allocated as big as index size on my J2EE Container but Don't alloc

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

On Tue, Feb 23, 2021 at 2:30 AM wrote: > Hi,- > > I tried MMapDirectory and i allocated as big as index size on my J2EE > Container but > > Don't allocate java heap memory for the index, MMapDirectory does not use java heap memory!

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

Hi,-  I tried MMapDirectory and i allocated as big as index size on my J2EE Container but it only gives me at most 25% speedup and even sometimes a small amount of slowdown. How can i effectively use Lucene indexes in memory? Best regards On 12/14/20 6:35 PM, baris.ka...@oracle.com wrote

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

Thanks Robert. I think these valuable comments need to be placed on javadocs for future references. i think i am getting enough info for making a decision: i will use MMapDirectory without setPreload and i hope my index will fit into the RAM. i plan to post a blog for findings. Best regar

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

On Mon, Dec 14, 2020 at 1:59 PM Uwe Schindler wrote: > > Hi, > > as writer of the original bog post, here my comments: > > Yes, MMapDirectory.setPreload() is the feature mentioned in my blog post is > to load everything into memory - but that does not guarantee anything! > Still, I would not recom

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

ctory dir = new MMapDirectory(Paths.get(indexDir)); // Uses FSLockFactory.getDefault() and DEFAULT_MAX_CHUNK_SIZE which is 1GB if (dir.getPreload() == false) dir.setPreload(Constants.PRELOAD_YES); // In-Memory Lucene Index enabled-> *commented out* IndexReader reader = DirectoryReader.open(dir);

RE: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

: > > > /*First way: Using constructor (without setPreload) :*/ > > MMapDirectory dir = new MMapDirectory(Paths.get(indexDir)); // Uses > FSLockFactory.getDefault() and DEFAULT_MAX_CHUNK_SIZE which is 1GB > if (dir.getPreload() == false) > dir.setPreload(Constants.PREL

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

l1TtPJMV80mkA-w$ eMail: u...@thetaphi.de -Original Message- From: baris.ka...@oracle.com Sent: Sunday, December 13, 2020 10:18 PM To: java-user@lucene.apache.org Cc: BARIS KAZAR Subject: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory) Hi,- it would be nice to

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

); // Uses FSLockFactory.getDefault() and DEFAULT_MAX_CHUNK_SIZE which is 1GB if (dir.getPreload() == false)   dir.setPreload(Constants.PRELOAD_YES); // In-Memory Lucene Index enabled-> *commented out* IndexReader reader = DirectoryReader.open(dir); ... /*Second way: Or using open (witho

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

...@thetaphi.de > > > -Original Message- > > From: baris.ka...@oracle.com > > Sent: Sunday, December 13, 2020 10:18 PM > > To: java-user@lucene.apache.org > > Cc: BARIS KAZAR > > Subject: MMapDirectory vs In Memory Lucene Index (i.e., > ByteBuffersDirectory)

RE: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

.ka...@oracle.com > Sent: Sunday, December 13, 2020 10:18 PM > To: java-user@lucene.apache.org > Cc: BARIS KAZAR > Subject: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory) > > Hi,- > > it would be nice to create a Lucene index in files and then effect

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

rg/apache/lucene/store/MMapDirectory.html__;!!GqivPVa7Brio!LEQH8Tyb_BBN_Kc3fEH2w-yhpvS-VwMrpuB0gctqchp3j7L7V6x9piciHOJ_Zf_dhQ$ Happy Holidays Best regards Ps. i know there is also BytesBuffersDirectory class for in memory Lucene but this requires creating Lucene Index on the fly. This is great fo

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

t my > >> JVM heap to 16G or a bit less (since my index is > >> around this much). > >> > >> The Lucene 8.5.2 (8.5.0 as well) javadocs say: > >> public void setPreload(boolean preload) > >> Set to true to ask mapped pages to be loaded into physical

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

-VwMrpuB0gctqchp3j7L7V6x9piciHOJ_Zf_dhQ$ Happy Holidays Best regards Ps. i know there is also BytesBuffersDirectory class for in memory Lucene but this requires creating Lucene Index on the fly. This is great for only such kind of Lucene indexes that can be created quickly on the fly. Ekaterina has a nice article on

Re: MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

memory on init. > The behavior is best-effort and operating system dependent. > > For example Lucene 4.0.0 does not have setPreLoad method. > > > https://lucene.apache.org/core/4_0_0/core/org/apache/lucene/store/MMapDirectory.html > > Happy Holidays > Best regards > > > Ps.

MMapDirectory vs In Memory Lucene Index (i.e., ByteBuffersDirectory)

for in memory Lucene but this requires creating Lucene Index on the fly. This is great for only such kind of Lucene indexes that can be created quickly on the fly. Ekaterina has a nice article on this BytesBuffersDirectory class: https://medium.com/@ekaterinamihailova/in-memory-search-and

RE: in memory lucene

cluster. Another interesting search result link snippet was from eClubPrague. -Original Message- From: Jonathan Willis Sent: Tuesday, February 26, 2019 6:19 PM To: java-user@lucene.apache.org Subject: in memory lucene Hi, i'm looking into using Lucene 7.7.0 and noticed tha

RE: in memory lucene

From: Jonathan Willis > Sent: Wednesday, February 27, 2019 12:19 AM > To: java-user@lucene.apache.org > Subject: in memory lucene > > Hi, i'm looking into using Lucene 7.7.0 and noticed that the RAMDirectory > has been deprecated because of inefficient synchronization issues and t

in memory lucene

Hi, i'm looking into using Lucene 7.7.0 and noticed that the RAMDirectory has been deprecated because of inefficient synchronization issues and that we are encouraged to use MMapDirectory instead. I was hoping to use an in memory only directory and was wondering if that would be possible without RA

RE: In memory Lucene configuration

Thanks for the input. I am not using Solr. Also, my index has a fixed size, I am not going to update it. -Original Message- From: googoo [mailto:liu...@gmail.com] Sent: 18 July 2012 15:21 To: java-user@lucene.apache.org Subject: Re: In memory Lucene configuration Doron, To verify

RE: In memory Lucene configuration

I had a threading issue in the client code calling Lucene, really nothing that has anything to do with this list :) -Original Message- From: Simon Willnauer [mailto:simon.willna...@gmail.com] Sent: 18 July 2012 21:48 To: java-user@lucene.apache.org Subject: Re: In memory Lucene

RE: In memory Lucene configuration

Hi, just to clarify: > In additional, i don't think load whole index to memory is good idea. Since the > index size will always increase. > For me, i change lucene code to disable MMapDirectory, since the index size is > bigger and bigger. > And MMapDirectory will call something like c++ share me

Re: In memory Lucene configuration

> From: Doron Yaacoby [mailto:dor...@gingersoftware.com] > Sent: 16 July 2012 09:43 > To: java-user@lucene.apache.org > Subject: RE: In memory Lucene configuration > > I haven't tried that yet, but it's an option. The reason I'm waiting on this > is that I a

Re: In memory Lucene configuration

ze is bigger and bigger. And MMapDirectory will call something like c++ share memory to load whole index to ram. -- View this message in context: http://lucene.472066.n3.nabble.com/In-memory-Lucene-configuration-tp3995075p3995697.html Sent from the Lucene - Java Users mailing list archive at Nabbl

RE: In memory Lucene configuration

@lucene.apache.org Subject: RE: In memory Lucene configuration I haven't tried that yet, but it's an option. The reason I'm waiting on this is that I am expecting many concurrent requests to my application anyway, so having multiple search threads per request might not be the best idea

Re: In memory Lucene configuration

08:26 > To: java-user@lucene.apache.org > Subject: Re: In memory Lucene configuration > > Have you tried sharding your data? Since you have a fast multi-core box, why > not split your indices N-ways, say the smaller one into 4, and the larger > into 8. Then you can have a pool of dedi

RE: In memory Lucene configuration

Vitaly Funstein [mailto:vfunst...@gmail.com] Sent: 16 July 2012 08:26 To: java-user@lucene.apache.org Subject: Re: In memory Lucene configuration Have you tried sharding your data? Since you have a fast multi-core box, why not split your indices N-ways, say the smaller one into 4, and the larger into

RE: In memory Lucene configuration

] Sent: 15 July 2012 13:40 To: java-user@lucene.apache.org; simon.willna...@gmail.com Subject: RE: In memory Lucene configuration Thanks for the quick input! I ran a few more tests with your suggested configuration (-Xmx1G -Xms1G with MMapDirectory). At the third time I ran the same test I fi

Re: In memory Lucene configuration

t; Sent: 15 July 2012 11:56 > To: java-user@lucene.apache.org > Subject: Re: In memory Lucene configuration > > hey there, > > On Sun, Jul 15, 2012 at 10:41 AM, Doron Yaacoby > wrote: >> Hi, I have the following situation: >> >> I have two pretty large indic

RE: In memory Lucene configuration

;t mention before that I'm using Lucene 3.5 and Java 1.7. -Original Message- From: Simon Willnauer [mailto:simon.willna...@gmail.com] Sent: 15 July 2012 11:56 To: java-user@lucene.apache.org Subject: Re: In memory Lucene configuration hey there, On Sun, Jul 15, 2012 at 10:41 AM, Doron

Re: In memory Lucene configuration

hey there, On Sun, Jul 15, 2012 at 10:41 AM, Doron Yaacoby wrote: > Hi, I have the following situation: > > I have two pretty large indices. One consists of about 1 billion documents > (takes ~6GB on disk) and the other has about 2 billion documents (~10GB on > disk). The documents are very sho

In memory Lucene configuration

Hi, I have the following situation: I have two pretty large indices. One consists of about 1 billion documents (takes ~6GB on disk) and the other has about 2 billion documents (~10GB on disk). The documents are very short (4-5 terms each in the text field, and one numeric field with a long valu