Hey Kafka lovers, I have a lot of questioning around the roll-out and the update of kafka streams app (continuous deployment fashion).
We are used to ship our apps in dockers quite frequently (our current orchestrator is AWS ECS). In the context of streams app, i understand we should be much more careful to avoid messing up internal state. Each app update could potentially change a computation of an internal state, change how we compute some output values, etc. What do you usually do ? change the app.id ? use the application reset tool ? Application reset only manages internal states. What about all the rest downstream ? (We will have a couple of databases downstream). Do you manually reset all this stuff ? How do you manage orchestration of app shutdown, downstream cleanup, start of the new version in a distributed context ? Is there a way to query kafka broker about running apps to wait the shutdown of the previous ? How do you automate all this ? With a bunch of scripts ? Are you able to test (like in automated CI test) that kind of update ? I know that 's a lot of questions ... :-/ If you're experienced in the topic and have 10-15mn of time to spare, I'd be happy to have a quick talk over google-meet or anything. This could be easier to discuss than to write a long email. If you have any resources, books, posts, conference talks, I'd be happy (if they are about real-life™ apps... not really interested in hello-worlds...) Thanks a lot ! Mathieu D
