Re: How to always consume from latest offset in kafka-streams

2018-01-19 Thread Matthias J. Sax
That is not supported out-of-box. Configuration "auto.offset.reset" only triggers, if there are not committed offsets and there is KS config to change this behavior. A possible workaround might be (but not sure if I want to recommend this), to increase KafkaStreams commit interval via StreamsConf

Re: How to always consume from latest offset in kafka-streams

2018-01-19 Thread Matt Farmer
That config setting will only work if there are no offsets stored in the consumer offsets target. Something I’ve done in the past is to make the application.id config setting have a random string component to it. So have “my-app-name-[randomchars]” or some such. This ensures that there are neve

How to always consume from latest offset in kafka-streams

2018-01-19 Thread Saloni Vithalani
Our requirement is such that if a kafka-stream app is consuming a partition, it should start it's consumption from latest offset of that partition. This seems like do-able using streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest") Now, let's say using above configuration,