Re: Making lucene indexing multi threaded

2014-10-28 Thread Erick Erickson
IndexWriter.close(); > > Thank you, > Jason > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Making-lucene-indexing-multi-threaded-tp4087830p4166123.html > Sent from the Lucene - Java Users mailing list archive at Nabble.com. > > --

Re: Making lucene indexing multi threaded

2014-10-27 Thread Jason Wu
: http://lucene.472066.n3.nabble.com/Making-lucene-indexing-multi-threaded-tp4087830p4166123.html Sent from the Lucene - Java Users mailing list archive at Nabble.com. - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org

Re: Making lucene indexing multi threaded

2014-10-27 Thread G.Long
ext: http://lucene.472066.n3.nabble.com/Making-lucene-indexing-multi-threaded-tp4087830p4166116.html Sent from the Lucene - Java Users mailing list archive at Nabble.com. - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apac

RE: Making lucene indexing multi threaded

2014-10-27 Thread Jason Wu
n and take 22 mins. Did you have any similar experience like the above before? Thank you, Jason -- View this message in context: http://lucene.472066.n3.nabble.com/Making-lucene-indexing-multi-threaded-tp4087830p4166116.html Sent from the Lucene - Java Users mailing list archive at Nabbl

RE: Making lucene indexing multi threaded

2014-10-27 Thread Fuad Efendi
@lucene.apache.org Subject: Re: Making lucene indexing multi threaded Hi Nischal, I had similar indexing issue. My lucene indexing took 22 mins for 70 MB docs. When i debugged the problem, i found out the indexWriter.addDocument(doc) taking a really long time. Have you already found the solution about it

Re: Making lucene indexing multi threaded

2014-10-27 Thread Jason Wu
://lucene.472066.n3.nabble.com/Making-lucene-indexing-multi-threaded-tp4087830p4166094.html Sent from the Lucene - Java Users mailing list archive at Nabble.com. - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For

Re: Making lucene indexing multi threaded

2013-09-02 Thread Danil Ε’ORIN
Don't commit after adding each and every document. On Tue, Sep 3, 2013 at 7:20 AM, nischal reddy wrote: > Hi, > > Some more update on my progress, > > i have multithreaded indexing in my application, i have used thread pool > executor and used a pool size of 4 but had a very slight increase in

Re: Making lucene indexing multi threaded

2013-09-02 Thread nischal reddy
Hi, Some more update on my progress, i have multithreaded indexing in my application, i have used thread pool executor and used a pool size of 4 but had a very slight increase in the performace very negligible, still it is taking around 20 minutes of time to index around 30k files, Some more inf

Re: Making lucene indexing multi threaded

2013-09-02 Thread nischal reddy
Hi Eric, I have commented out the indexing part (indexwriter.addDocument()) part in my application and it is taking around 90 seconds, but when i uncomment the indexing part it is taking lot of time. My machine specs are windows 7, intel i7 processor, 4gb ram and doest have an ssd harddisk. can

Re: Making lucene indexing multi threaded

2013-09-02 Thread Adrien Grand
Hi, Lucene's IndexWriter can safely accept updates coming from several threads, just make sure to share the same IndexWriter instance across all threads, no extrenal locking is necessary. 30 minutes sound slike a lot for 3 files unless they are large. You can have a look at http://wiki.apache

Re: Making lucene indexing multi threaded

2013-09-02 Thread Erick Erickson
Stop. Back up. Test. The very _first_ thing I'd do is just comment out the bit that actually indexes the content. I'm guessing you have some loop like: while (more files) { read the file transform the data create a Lucene document index the document } Just comment out the "index

Making lucene indexing multi threaded

2013-09-02 Thread nischal reddy
Hi, I am thinking to make my lucene indexing multi threaded, can someone throw some light on the best approach to be followed for achieving this. I will give short gist about what i am trying to do, please suggest me the best way to tackle this. What am i trying to do? I am building an index fo