Re: Exception from the codec layer during indexing

2023-09-28 Thread Rahul Goswami
n before (which doesn't mean it's > not a real bug). > > What JVM are you running? Does this exception frequently occur or was > it a one-off? > > On Thu, Sep 28, 2023 at 4:49 PM Rahul Goswami > wrote: > > > > Hi, > > Following up on my issue...anyone

Re: Exception from the codec layer during indexing

2023-09-28 Thread Rahul Goswami
Hi, Following up on my issue...anyone who's seen similar exceptions ? Or any insights on what might be going on? Thanks, Rahul On Wed, Sep 27, 2023 at 1:00 AM Rahul Goswami wrote: > Hello, > On one of the servers running Solr 7.7.2, during indexing I observe 2 > different kinds

Exception from the codec layer during indexing

2023-09-26 Thread Rahul Goswami
Hello, On one of the servers running Solr 7.7.2, during indexing I observe 2 different kinds of exceptions coming from the Lucene codec layer. I can't think of an application/data issue that could be causing this. In particular, Exception-2 seems like a potential bug since it complains about "term

Re: Reindexing leaving behind 0 live doc segments

2023-09-09 Thread Rahul Goswami
mmits-transaction-logs.html > > Uwe > > Am 31.08.2023 um 21:35 schrieb Rahul Goswami: > > Stefan, Mike, > > Appreciate your responses! I spent some time analyzing your inputs and > > going further down the rabbit hole. > > > > Stefan, > > I looked

Re: Reindexing leaving behind 0 live doc segments

2023-08-31 Thread Rahul Goswami
ers' lifecycle? > > Mike McCandless > > http://blog.mikemccandless.com > > > On Mon, Aug 28, 2023 at 10:20 PM Rahul Goswami > wrote: > >> Hello, >> I am trying to execute a program to read documents segment-by-segment and >> reindex to the same in

Re: Reindexing leaving behind 0 live doc segments

2023-08-30 Thread Rahul Goswami
ug 29, 2023 at 2:46 AM Mikhail Khludnev wrote: > Hi Rahul. > Are you looking for > > https://lucene.apache.org/core/9_0_0/core/org/apache/lucene/index/IndexWriter.html#forceMergeDeletes() > ? > > On Tue, Aug 29, 2023 at 5:20 AM Rahul Goswami > wrote: > > > Hello

Reindexing leaving behind 0 live doc segments

2023-08-28 Thread Rahul Goswami
Hello, I am trying to execute a program to read documents segment-by-segment and reindex to the same index. I am reading using Lucene apis and indexing using solr api (in a core that is currently loaded). What I am observing is that even after a segment has been fully processed and an autoCommit (

Re: Performance regression in getting doc by id in Lucene 8 vs Lucene 7

2023-06-06 Thread Rahul Goswami
ByteBuffer bb = > ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN).asReadOnlyBuffer(); > return new ByteBuffersIndexInput(new > ByteBuffersDataInput(Collections.singletonList(bb)), > "ByteBuffersIndexInput(" + name + ")"); > } > } >

Re: Performance regression in getting doc by id in Lucene 8 vs Lucene 7

2023-06-06 Thread Rahul Goswami
doesn't like the fact that the terms index now reads > > data directly from the directory instead of loading the terms index in > > heap. Would you be able to run the same benchmark with MMapDirectory > > to check if it addresses the regression? > > > > > &g

Performance regression in getting doc by id in Lucene 8 vs Lucene 7

2023-06-05 Thread Rahul Goswami
Hello, We started experiencing slowness with atomic updates in Solr after upgrading from 7.7.2 to 8.11.1. Running several tests revealed the slowness to be in RealTimeGet's SolrIndexSearcher.getFirstMatch() call which eventually calls Lucene's SegmentTermsEnum.seekExact().. In the benchmarks I ran

Re: Questions about Lucene source

2022-12-14 Thread Rahul Goswami
David and Adrien, thanks for your responses. Bringing up an old thread here. Revisiting this question ... > (so deleted docs == max docs) and call commit. Will/Can this segment still > exist after commit? SInce I am using Solr (8.11.1), the default deletion policy is SolrDeletionPolicy which retai

Learning Lucene from ground up

2022-11-04 Thread Rahul Goswami
Hello, I have been working with Lucene and Solr for quite some time and have a good understanding of a lot of moving parts at the code level. However I wish to learn Lucene internals from the ground up and want to familiarize myself with all the dirty details. I would like to know what would be th

Re: Questions about Lucene source

2022-09-16 Thread Rahul Goswami
Following up on my questions since they didn't get much love the first time. Any inputs are greatly appreciated! Thanks, Rahul On Wed, Sep 14, 2022 at 3:58 PM Rahul Goswami wrote: > Hello, > > I was going through some parts of the Lucene source and had some questions: > 1)

Questions about Lucene source

2022-09-14 Thread Rahul Goswami
Hello, I was going through some parts of the Lucene source and had some questions: 1) Can lucene have 0 document segments? Or will they always be purged (either by TMP or otherwise) on a commit? Eg: A segment has 4 docs, and I make a /update call to overwrite all 4 docs (so deleted docs == max doc

Re: Lucene 9.2.0 build fails on Windows

2022-09-14 Thread Rahul Goswami
Uwe, Dawid, and Robert, Thank you for the helpful pointers! I do have Visual Studio 2017 on my machine which I don't use much lately. https://github.com/microsoft/vswhere *"vswhere* is included with the installer as of Visual Studio 2017 version 15.2 and later, and can be found at the following lo

Re: Lucene 9.2.0 build fails on Windows

2022-09-13 Thread Rahul Goswami
Hi Dawid, I believe you. Just that for some reason I have never been able to get it to work on Windows. Also, being a complete newbie to gradle doesn't help much. So would appreciate some help on this while I find my footing. Here is the link to the diagnostics that you requested (since attachments

Re: Lucene 9.2.0 build fails on Windows

2022-09-13 Thread Rahul Goswami
; If you need additional stacktrace or other diagnostics I am happy to provide the same. Thanks, Rahul On Tue, Sep 13, 2022 at 11:37 AM Dawid Weiss wrote: > It does work just fine. Use cmd or powershell though. I don't think > things are even tested with cygwin/msys. > > Da

Lucene 9.2.0 build fails on Windows

2022-09-12 Thread Rahul Goswami
Hello, I am using gitbash to build lucene 9.2.0 on Windows. I checked out the release/lucene/9.2.0 tag and tried running "./gradlew help". But it fails. Running Java 11.0.4. Somehow building lucene 9x on Windows has never worked for me. Had the same issue with 9.0.0 as well. mypc@mypc MINGW64 /c/w

Re: SI File Missing

2022-08-11 Thread Rahul Goswami
Hi Brian, This is a case of index corruption and unless you have a backup of your working index, there is no way to recover the data unfortunately. There is, however, a way for you to recover the index with partial data loss if that is something that you can work with. You can use lucene's CheckIn

Error building lucene 9.0

2022-07-07 Thread Rahul Goswami
Hi, I cloned the lucene repo from github and checked out branch 9.0. I have JDK 11 installed on my Windows machine and am using GitBash to run the build as below: === $ ./gradlew assemble To honour the JVM settings for this build a single-use Daemon process will be fork

Re: Moving from lucene 6.x to 8.x

2022-01-26 Thread Rahul Goswami
Uwe, This is beautiful! Especially for conversion from Trie to Point fields is going to be extremely handy. I am going to have to check this out further. Thank you for the tip! Rahul On Mon, Jan 17, 2022 at 10:23 AM Uwe Schindler wrote: > By the way > > Hi, one thing that always works to "force

Re: Moving from lucene 6.x to 8.x

2022-01-15 Thread Rahul Goswami
orget) offset. If you use term vectors, and have these > broken offsets, which should not but do often occur with custom > analysis chains, this could be a problem. > > On Wed, Jan 12, 2022 at 12:36 AM Rahul Goswami > wrote: > > > > Thanks Vinay for the link to Erick&#x

Re: Moving from lucene 6.x to 8.x

2022-01-11 Thread Rahul Goswami
ch the lucene upgrader is > > unable to upgrade any index EVER touched by <= 6.x ? Or is the > restriction > > more of a safety net at this point for possible future incompatibilities > ? > > > > Thanks, > > Rahul > > > > On Thu, Jan 6, 2022 at 11:46

Re: Moving from lucene 6.x to 8.x

2022-01-08 Thread Rahul Goswami
ies ? Thanks, Rahul On Thu, Jan 6, 2022 at 11:46 PM Rahul Goswami wrote: > Hello, > I am using Apache Solr 7.7.2 with indexes which were originally created on > 4.8 and upgraded ever since. I recently tried upgrading to 8.x using the > lucene IndexUpgrader tool and the upgrade fails. I k

Moving from lucene 6.x to 8.x

2022-01-06 Thread Rahul Goswami
Hello, I am using Apache Solr 7.7.2 with indexes which were originally created on 4.8 and upgraded ever since. I recently tried upgrading to 8.x using the lucene IndexUpgrader tool and the upgrade fails. I know that lucene 8.x prevents opening any segment which was touched by <= 6.x at any point in

Any downsides to using RAFDirectory instead of SimpleFSDirectory ?

2021-10-21 Thread Rahul Goswami
Hello, I know RAFDirectory was marked legacy, but can anyone please share any downsides to using RAFDirectory over SimpleFSDirectory. I am running Solr on a Windows server and mmap doesn't quite work well there, so I have been using SimpleFS. It was working well for the most part, but we recently