Re: Forwarding consumer with kafka streams

2017-08-14 Thread Ricardo Costa
Thanks you your help Eno and Guozhang. Indeed I missed the obvious, I made a bad assumption about defaults, should have checked the source code. I thought Kafka Streams was setting AUTO_OFFSET_RESET_CONFIG to "earliest", and it is, but not for the version I'm using! I'm using version 0.10.0.1 whic

Re: Forwarding consumer with kafka streams

2017-08-13 Thread Guozhang Wang
Hi Richardo, What you described seems very similar to the demo example code as stated here: https://github.com/apache/kafka/blob/trunk/streams/quickstart/java/src/main/resources/archetype-resources/src/main/java/Pipe.java If you started the program it should just pipe all data starting from the e

Re: Forwarding consumer with kafka streams

2017-08-12 Thread Eno Thereska
Hi Ricardo, Kafka Streams should handle that case as well. What streams config are you using, could you share it? There is one parameter that is called “ConsumerConfig.AUTO_OFFSET_RESET_CONFIG” and by default it’s set to “earliest”. Any chance your app has changed it to “latest”? Thanks Eno >

Forwarding consumer with kafka streams

2017-08-12 Thread Ricardo Costa
Hi, I've implemented a forwarding consumer which literally just consumes the messages from a source topic, logs them and then publishes them to a target topic. I wanted to keep the implementation simple with very little code so I went with kafka-streams. I have a really simple topology with a sou