Hello, I'm wondering how to design a KStreams or regular Kafka application that can hold of processing of messages until a future time.
This related to EU's data protection regulation: we can store raw messages for a given time; afterwards we have to store the anonymised message. So, I was thinking about branching the stream, anonymise the messages into a waiting topic and than continue from there until the retention time passes. But that approach has some caveats: - This is not an exact solution as order of events is not guaranteed: we might encounter a message that triggers the stop processing while some events arriving later should normally still pass - how to stop properly stop processing if we encounter a message that indicates to not continue? - ... Are there better know solutions or best practices to delay message processing with Kafka streams / consumers+producers? Thanks for any insights/help here! -wim
