Hi guys!
My apology for irrelevant post there. I don't wanna receive emails from @lucene.apache.org on this account as I have another one for it. I sent email to unsub@. But still receiving emails. Please suggest!! Regards, Khurram ________________________________ From: Ravikumar Govindarajan <ravikumar.govindara...@gmail.com> Sent: Thursday, August 10, 2017 6:37 PM To: java-user@lucene.apache.org Subject: Re: Storing external transaction log-ids in lucene... Many thanks for the help... Just one more clarification.. I also see that a sequence number is returned from IW on all mutations in latest code including prepareCommit() [Apologize, as we are still on an old version of lucene 4.6] So the approach will be to co-relate the seq.no returned from individual operations of IW with the external queue offset. Background commit thread can then call prepareCommit() & it will exactly know what got through. Have I understood it correctly? Btw, we have a TrackingIndexWriter that returns a sequence number too, in 4.6. Is the new implementation something different from this? -- Ravi On Thu, Aug 10, 2017 at 6:09 PM, Michael McCandless < luc...@mikemccandless.com> wrote: > IW.setCommitData (now .setLiveCommitData in 7.0) is the right way to store > this. > > Mike McCandless > > http://blog.mikemccandless.com Changing Bits<http://blog.mikemccandless.com/> blog.mikemccandless.com The Apache Lucene project will likely release its next major release, 7.0, in a few months! Remember that Lucene developers generally try hard to backport new ... > > On Thu, Aug 10, 2017 at 6:57 AM, Ravikumar Govindarajan < > ravikumar.govindara...@gmail.com> wrote: > > > Every mutation (Add/Update/Delete) has a transaction-id (incremental > long) > > assigned by our Messaging Queue (Kafka) > > > > To index these mutations, an indexer thread pulls data from the queue, > adds > > & commits to IndexWriter, then updates the latest transaction-id in an > > external system (ZooKeeper). During rollback/server-restarts, the threads > > read the previous value from ZK & resume... > > > > Have now moved to NRT implementation, where commit thread runs in the > > background & am unable to find the latest transaction-id that got > committed > > > > Initially thought of adding transaction-id as a first-class field of the > > index itself & then writing a custom-codec to harvest it, but this fails > > when a mutation-set consists only of deletes [as there is no way to pass > > these ids via IW delete APIs] > > > > Is there a way to store this as part of segment-info?. Can > setCommitData() > > of IW help me here in any way? > > > > -- > > Ravi > > >
