Re: Lucene Index Writer in a distributed system

2023-10-19 Thread Cody Amen
Zookeeper, right? Look how Zookeeper is used in Solr, but Zookeeper does exactly what you want, I believe. Sent from my iPhone > On Oct 19, 2023, at 3:49 AM, Gopal Sharma wrote: > > Hello Team, > > I am new to Lucene and want to use Lucene in a distributed system to write > in a Amazon EFS i

Re: Lucene Index Writer in a distributed system

2023-10-19 Thread Michael McCandless
Hi Gopal, Indeed, for a single Lucene index, only one writer may be open at a time. Lucene tries to catch you if you mess this up, using file-based locking. If you really need concurrent indexing, you could have N IndexWriters each writing into a private Directory, and then periodically use addIn

Re: Lucene: Index Writer to write in multiple file instead make one heavy file

2014-05-15 Thread Michael McCandless
Just don't call optimize... In theory, you could make a custom Directory impl that would split a single large file (from Lucene's standpoint) into multiple OS files, but this ... would be a lot of work. It's simpler to just not optimize. Mike McCandless http://blog.mikemccandless.com On Wed,

Re: Lucene: Index Writer to write in multiple file instead make one heavy file

2014-05-15 Thread Yogesh patel
In my case, it creates CFS(Compound File) As 10 GB. Can we split that file while optimize or writing index. Thanks On Wed, May 14, 2014 at 7:38 PM, Yogesh patel wrote: > Thanks for reply!!! > > Can you please provide me sample code for it? I got the idea but i dont > know how to impleme

Re: Lucene: Index Writer to write in multiple file instead make one heavy file

2014-05-14 Thread Yogesh patel
Thanks for reply!!! Can you please provide me sample code for it? I got the idea but i dont know how to implement it. Thanks On Tue, May 13, 2014 at 7:02 PM, Michael McCandless < luc...@mikemccandless.com> wrote: > You can tell the MergePolicy to limit the maximum size of segments it >

RE: Lucene: Index Writer to write in multiple file instead make one heavy file

2014-05-13 Thread Toke Eskildsen
Yogesh patel [yogeshpateldai...@gmail.com] wrote: > I am using lucene 3.0.1. I am writing many documents with lucene > Indexwriter. But Indexwriter add all documents into file which becomes more > than 4GB in my case. so can i distribute files or partitioned ? Normally Lucene does not produce a si

Re: Lucene: Index Writer to write in multiple file instead make one heavy file

2014-05-13 Thread Michael McCandless
You can tell the MergePolicy to limit the maximum size of segments it should merge. Also, you should try to upgrade: 3.0.1 is REALLY old. Mike McCandless http://blog.mikemccandless.com On Tue, May 13, 2014 at 1:58 AM, Yogesh patel wrote: > HI > > I am using lucene 3.0.1. I am writing many doc

RE: Lucene - index writer

2006-05-10 Thread Satuluri, Venu_Madhav
Hi, I don't think it should cause any conflicts in the index itself (the indexing process proper is decoupled from the analyzing), and if you can decide as to which analyzer to use when you're searching based on the field/kind of search, then it should be fine. Regards, Venu -Original Messa