Re: Lucene Index Writer in a distributed system

Zookeeper, right? Look how Zookeeper is used in Solr, but Zookeeper does exactly what you want, I believe. Sent from my iPhone > On Oct 19, 2023, at 3:49 AM, Gopal Sharma wrote: > > Hello Team, > > I am new to Lucene and want to use Lucene in a distributed system to write > in a Amazon EFS i

Re: Lucene Index Writer in a distributed system

Hi Gopal, Indeed, for a single Lucene index, only one writer may be open at a time. Lucene tries to catch you if you mess this up, using file-based locking. If you really need concurrent indexing, you could have N IndexWriters each writing into a private Directory, and then periodically use addIn

Lucene Index Writer in a distributed system

Hello Team, I am new to Lucene and want to use Lucene in a distributed system to write in a Amazon EFS index. As per my understanding, the index writer for a particular index needs to be opened by 1 server only. Is there a way we can achieved this in distributed system to write parallelly in Luce

Re: Field[vector]vector's dimensions must be <= [1024]; got 1536

Hi Uwe Thank you very much for your quick feedback, really appreciated! Will change it as you describe below. Thanks Michael Am 19.10.23 um 11:23 schrieb Uwe Schindler: Hi Michael, The max vector dimension limit is no longer checked in the field type as it is responsibility of the codec

Re: Field[vector]vector's dimensions must be <= [1024]; got 1536

Hi Michael, The max vector dimension limit is no longer checked in the field type as it is responsibility of the codec to enforce it. You need to build your own codec that returns a different setting so it can be enforced by IndexWriter. See Apache Solr's code how to wrap the existing KnnVec

Re: Field[vector]vector's dimensions must be <= [1024]; got 1536

I forgot to mention, that when using the custom FieldType and 1536 vector dimension does work with Lucene 9.7.0 Thanks Michael Am 19.10.23 um 10:39 schrieb Michael Wechner: Hi I recently upgraded Lucene to 9.8.0 and was running tests with OpenAI's embedding model, which has the vector dim

Field[vector]vector's dimensions must be <= [1024]; got 1536

Hi I recently upgraded Lucene to 9.8.0 and was running tests with OpenAI's embedding model, which has the vector dimension 1536 and received the following error Field[vector]vector's dimensions must be <= [1024]; got 1536 wheres this worked previously with the hack to override the vector di