Re: Performance Difference between files getting opened with IoContext.RANDOM vs IoContext.READ

2024-10-16 Thread Navneet Verma
eloading. Please open an issue instead for discussion. > > Uwe > > Am 15.10.2024 um 09:06 schrieb Navneet Verma: > > Hi Uwe, > > > > *>> thinking about it a bit more: In 10.x we already have some ways to > > **preload > > data with WILL_NEED (or

Re: Performance Difference between files getting opened with IoContext.RANDOM vs IoContext.READ

2024-10-15 Thread Navneet Verma
> requires files to be opened multiple times its a better compromise. > > > > Please note: we are focusing on 10.x, so please supply PRs/changes for > > Lucene main branch only, backports will be done automatically. We > > won't change the IOContexts in 9.x anymo

Re: Learning resources for Lucene Development

2024-10-08 Thread Navneet Verma
+1 on the question. On Tue, Oct 8, 2024 at 6:35 PM Marc Davenport wrote: > Hello, > I had this question buried in a previous email. I feel like I have a very > loose grasp on the Lucene API and how to properly implement with it. I'm > working on code that I didn't write myself from the ground

Re: Performance Difference between files getting opened with IoContext.RANDOM vs IoContext.READ

2024-10-01 Thread Navneet Verma
he > checksumming is slower it should not be a big issue, because it won't > affect searches, only merging of segments. And there the throughput > should be high, but not top priority. > > Uwe > > Am 01.10.2024 um 04:52 schrieb Navneet Verma: > > Hi

Re: Performance Difference between files getting opened with IoContext.RANDOM vs IoContext.READ

2024-10-01 Thread Navneet Verma
le times its a better compromise. > > Please note: we are focusing on 10.x, so please supply PRs/changes for > Lucene main branch only, backports will be done automatically. We won't > change the IOContexts in 9.x anymore. > > Uwe > > Am 01.10.2024 um 10:04 schrieb N

Re: Performance Difference between files getting opened with IoContext.RANDOM vs IoContext.READ

2024-09-30 Thread Navneet Verma
all bytes/pages in .vec/.veq files -- this asks the OS to cache > > all of those bytes into page cache (if there is enough free RAM). We do > > this at Amazon (product search) for our production searching processes. > > Otherwise paging in all .vec/.veq pages via random access pr

Performance Difference between files getting opened with IoContext.RANDOM vs IoContext.READ

2024-09-29 Thread Navneet Verma
Hi Lucene Experts, I wanted to understand the performance difference between opening and reading the whole file using an IndexInput with IoContext as RANDOM vs READ. I can see .vec files(storing the flat vectors) are opened with RANDOM and whereas dvd files are opened as READ. As per my testing wi