On Sun, Sep 9, 2018 at 6:09 AM Jonathan Haddad <j...@jonhaddad.com> wrote:
> I'll be honest, I'm having a hard time wrapping my head around an > architecture where you use CDC to push data into Kafka. I've worked on > plenty of systems that use Kafka as a means of communication, and one of > the consumers is a process that stores data in Cassandra. That's pretty > normal. Sending Cassandra mutations to Kafka, on the other hand, feels > backwards and for 99% of teams, more work than it's worth. > > There may be some use cases for it.. but I'm not sure what they are. It > might help if you shared the use cases where the extra complexity is > required? When does writing to Cassandra which then dedupes and writes to > Kafka a preferred design then using Kafka and simply writing to Cassandra? > > Somewhat contrived, but: 1) Sending all mutations to an audit service to look for suspicious activity (e.g. looking for someone doing something mailicous in an app with direct db access), 2) General purpose composable pipelines (job A writes spark -> cassandra, job B takes cassandra -> mysql / hadoop / whatever via kafka CDC) Agree that it seems less common, but I'm sure there's a real use case for it somewhere.
