Here is a scenario where this could be useful: Add the kafka offset as a field on the record in both Cassandra and Elasticsearch
Now when you get search results from Elastic search and look up details in Cassandra you can know if they come from the same kafka record. If you can use the offset as part of the Cassandra Partition key, or as a clustering key, then you could specifically retrieve a version of the record from Cassandra that matches Elasticsearch (assuming it made it). If your real goal is to guarantee that the two datasets always have the same set of messages from Kafka ... I don't think this is possible. -Dave -----Original Message----- From: kant kodali [mailto:kanth...@gmail.com] Sent: Monday, November 7, 2016 10:48 AM To: users@kafka.apache.org Subject: Re: is there a way to make sure two consumers receive the same message from the broker? Hi AmitHossein, I still don't see how that guarantees consistency at any given time. other words how do I know at time X the data in Cassandra and ES are the same. Thanks On Mon, Nov 7, 2016 at 3:26 AM, AmirHossein Roozbahany <diver...@outlook.com > wrote: > Hi > > Can you use elasticsearch _version field as cassandra's > writetime?(_version is strictly increasing, cassandra uses writetime > for applying LWW, so last write in elasticsearch will always win) > > It needs no transaction and makes databases convergent. > > > ________________________________ > From: kant kodali <kanth...@gmail.com> > Sent: Monday, November 7, 2016 3:08 AM > To: users@kafka.apache.org > Subject: Re: is there a way to make sure two consumers receive the > same message from the broker? > > Hi Hans, > > The two storages we use are Cassandra and Elastic search and they are > on the same datacenter for now. > The Programming Language we use is Java and OS would be Ubuntu or CentOS. > We get messages in JSON format so we insert into Elastic Search > directly and for Cassandra we transform JSON message into appropriate > model so we could insert into a Cassandra table. > The rate we currently get is about 100K/sec which is awesome but I am > pretty sure this will go down once when we implement 2PC or > transactional writes. > > Thanks, > kant > This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
