Re: Adding or removing input topics to a Kafka Consumer without downtime

2017-09-05 Thread Håkon Åmdal
Could this be something that only is relevant to the Streams app, then? I've also tried to add a join, but I'm getting the same issues with repartitioning the state store. I've pasted some stack traces below. 19:14:54 ERROR o.a.k.c.c.i.ConsumerCoordinator - User provided listener org.apache.kafk

Re: Adding or removing input topics to a Kafka Consumer without downtime

2017-09-05 Thread Jeff Widman
I haven't played with the Streams app, but I know at the consumer group level that consumers who are part of the same group can have different subscriptions. So at the consumer group level at least this is theoretically possible to have a rolling upgrade (and in fact was purposefully designed to su

Adding or removing input topics to a Kafka Consumer without downtime

2017-09-05 Thread Håkon Åmdal
Hi there, In our company, we are running multiple Kafka Streams app that are deployed in a red/black fashion. A deployment means starting a new set of hosts, run them in parallel with the old hosts until they pass the health check, and then scale down the old hosts. This approach works very well a