Re: Seeking advice on Kafka Streams and Kafka Connect

2017-12-21 Thread Manoj Khangaonkar
Hi I am not a big fan of kafka connect. I had use case for kafka messages that needed to be written to MongoDb. The available third party connectors were less than ideal. To me a well written Kafka consumer is simpler and better longer term solution instead of an additional moving part and addi

Re: Seeking advice on Kafka Streams and Kafka Connect

2017-12-21 Thread Hans Jespersen
It might be possible to do all the transformations in #2 inside Kafka Connect. Connect has a simple one message at a time transformation capability called Single Message Transforms (SMT). There are built in Transformation functions that you can declaratively add to any existing connector via con

Re: Seeking advice on Kafka Streams and Kafka Connect

2017-12-21 Thread Bill Bejeck
Hi Mads, Great question and yes your use case here is an excellent fit for Kafka Streams and Kafka Connect. For step 2 you could use a KStram#flatMap operation to split it up into multiple rows. Regarding a Cassandra connector, there is an existing one: 1. For some background try https://