Hi, I have a rather complicated kafka streams application involving multiple joins, aggregates, maps etc. At a certain point, parts of the data needs to be removed throughout the entire streams topology, both in the topics, the changelogs and the rocksdb state stores.
Managing this requires a lot of effort and things get very complex. F.e. when a KStream has a null value and is aggregated, you first need to convert it into some optional value instead b/c aggregates ignore nulls. Is there a better way or a way that does not impact all the existing streaming logic? I was thinking about having an out-of-bound process that sends null values to all topics with the correct keys. I could then filter out all null values before doing the rest of the existing stream logic. Would that make sense? I can send null values to all my topics, but how do I get the changelog topics created by kafka-streams. And what about the state store? Best regards Jan
