yes this problem can definetly be approached in many ways but given the hard constraints by our clients we don't seem to have many options. so the problem is we have to keep two storages systems in sync all the time. so whatever the data that is storage 1 should also be in storage 2 at any given time. so we explored the following options
1) we thought about ETL from storage 1 to storage 2 but that approach has bunch of drawbacks given the time constraints. 2) Add a common service on top of two storages and do some sort of 2PC but that would degrade the write performance. Morever we dont really have a control over how fast each write/store can happen at each storage layer (because these two storages are completely different). so I started exploring if there is any tricks I could do with Kafka? On Sat, Nov 5, 2016 at 5:01 PM, Hans Jespersen <h...@confluent.io> wrote: > Yes exactly. If consumer 1 gets message with offset 17 then it can write > that offset into an external storage that consumer 2 can also check to > ensure that it keeps in sync with consumer 1. > > Just curious though why you would need to do this? What is the use case > because there may be a better way to get you the functionality you want? > > -hans > > > > > > On Nov 5, 2016, at 4:31 PM, kant kodali <kanth...@gmail.com> wrote: > > > > I am new to Kafka and reading this statement "write consumer 1 and > consumer > > 2 to share a common external offset storage" I can interpret it many ways > > but my best guess is as follows. > > > > Are you saying write the current offset of each consumer to a common > > external storage? > > > > > > On Sat, Nov 5, 2016 at 4:15 PM, kant kodali <kanth...@gmail.com> wrote: > > > >> Hi Hans, > >> > >> What do you mean by "write consumer 1 and consumer 2 to share a common > >> external offset storage" ? can you please elaborate a bit more. > >> > >> Thanks! > >> > >> On Sat, Nov 5, 2016 at 4:00 PM, Hans Jespersen <h...@confluent.io> > wrote: > >> > >>> There is no built in mechanism to do this in Apache Kafka but if you > can > >>> write consumer 1 and consumer 2 to share a common external offset > storage > >>> then you may be able to build the functionality you seek. > >>> > >>> -hans > >>> > >>> > >>> > >>>> On Nov 5, 2016, at 3:55 PM, kant kodali <kanth...@gmail.com> wrote: > >>>> > >>>> Sorry there is a typo. here is a restatement. > >>>> > >>>> Is there a way to make sure two consumers receive the same message > from > >>> the > >>>> kafka broker in a atomic way? such that if consumer 1 gets a message > >>>> consumer 2 should also get that message and if consumer 1 fails for > >>>> whatever reason consumer 2 should also rollback to previous offset (to > >>> the > >>>> same offset as consumer 1) or invalidate or something like that. is > that > >>>> possible? > >>> > >>> > >> > >
