Hi there, In our company, we are running multiple Kafka Streams app that are deployed in a red/black fashion. A deployment means starting a new set of hosts, run them in parallel with the old hosts until they pass the health check, and then scale down the old hosts. This approach works very well as we can deploy application changes without any downtime.
However, as far as I understand, there is no way we can run a new and old version in parallel if they don’t consume from the same set of topics. Consumers will try to rebalance tasks between the two different application version, causing errors like the one below: Uncaught exception: Thread xyz-StreamThread-1 stopped unexpectedly after Assigned partition foo-1 for non-subscribed topic regex pattern; subscription pattern is bar To mitigate the problem, we have so far scaled down the old cluster to 0 instances before deploying the new application if we’re adding or removing input topics. However, this causes service downtime which we no longer can accept. I’m curious to hear if anyone are experiencing the same issues, or if anyone have any thoughts or opinions? Are we doing something wrong, or is this something that can be solved by the Kafka Consumer client? Thanks, Håkon
