Re: Storing external transaction log-ids in lucene...

2017-08-15 Thread Michael McCandless
You're welcome; I'm glad it worked well! Mike McCandless http://blog.mikemccandless.com On Fri, Aug 11, 2017 at 2:31 AM, Ravikumar Govindarajan < ravikumar.govindara...@gmail.com> wrote: > Many thanks. This is a real cool feature & saves us a lot of time !!! > > -- > Ravi > > On Thu, Aug 10, 20

Re: Storing external transaction log-ids in lucene...

2017-08-10 Thread Ravikumar Govindarajan
Many thanks. This is a real cool feature & saves us a lot of time !!! -- Ravi On Thu, Aug 10, 2017 at 9:40 PM, Michael McCandless < luc...@mikemccandless.com> wrote: > That's exactly right! That is the purpose of the sequence numbers returned > by IndexWriter mutations. You can know exactly wh

Re: Storing external transaction log-ids in lucene...

2017-08-10 Thread Khurram Shehzad
kumar Govindarajan Sent: Thursday, August 10, 2017 6:37 PM To: java-user@lucene.apache.org Subject: Re: Storing external transaction log-ids in lucene... Many thanks for the help... Just one more clarification.. I also see that a sequence number is returned from IW on all mutations in latest

Re: Storing external transaction log-ids in lucene...

2017-08-10 Thread Michael McCandless
That's exactly right! That is the purpose of the sequence numbers returned by IndexWriter mutations. You can know exactly which ops made it into your commit and which didn't. TrackingIndexWriter is replaced by the sequence numbers. Mike McCandless http://blog.mikemccandless.com On Thu, Aug 10

Re: Storing external transaction log-ids in lucene...

2017-08-10 Thread Ravikumar Govindarajan
Many thanks for the help... Just one more clarification.. I also see that a sequence number is returned from IW on all mutations in latest code including prepareCommit() [Apologize, as we are still on an old version of lucene 4.6] So the approach will be to co-relate the seq.no returned from ind

Re: Storing external transaction log-ids in lucene...

2017-08-10 Thread Michael McCandless
IW.setCommitData (now .setLiveCommitData in 7.0) is the right way to store this. Mike McCandless http://blog.mikemccandless.com On Thu, Aug 10, 2017 at 6:57 AM, Ravikumar Govindarajan < ravikumar.govindara...@gmail.com> wrote: > Every mutation (Add/Update/Delete) has a transaction-id (increment

Storing external transaction log-ids in lucene...

2017-08-10 Thread Ravikumar Govindarajan
Every mutation (Add/Update/Delete) has a transaction-id (incremental long) assigned by our Messaging Queue (Kafka) To index these mutations, an indexer thread pulls data from the queue, adds & commits to IndexWriter, then updates the latest transaction-id in an external system (ZooKeeper). During