Re: indexing 15 million documents to lucene

2016-07-06 Thread Michael McCandless
Use threads, only commit at the end (and use a near-real-time reader if you want to search at points-in-time), increase IW's indexing buffer. Mike McCandless http://blog.mikemccandless.com On Wed, Jul 6, 2016 at 4:37 PM, Nomar Morado wrote: > Hi > > I am trying to write 15 million documents (a

indexing 15 million documents to lucene

2016-07-06 Thread Nomar Morado
Hi I am trying to write 15 million documents (and maybe more) to lucene for indexing. I would try to call writer.commit at some #/byte size of documents. The entire generated lucene files are about 1 GB total. My timing is about ~ 15-20 mins. I don't know if there are other configurations i ca