Hello Wim, Not sure if I understand your motivations for delayed processing, could you elaborate a bit more? Do you want to process raw messages, or do you want to process anonymised messages?
Guozhang On Thu, Mar 8, 2018 at 12:35 PM, Wim Van Leuven < wim.vanleu...@highestpoint.biz> wrote: > Hello, > > I'm wondering how to design a KStreams or regular Kafka application that > can hold of processing of messages until a future time. > > This related to EU's data protection regulation: we can store raw messages > for a given time; afterwards we have to store the anonymised message. So, I > was thinking about branching the stream, anonymise the messages into a > waiting topic and than continue from there until the retention time passes. > > But that approach has some caveats: > > - This is not an exact solution as order of events is not guaranteed: we > might encounter a message that triggers the stop processing while some > events arriving later should normally still pass > - how to stop properly stop processing if we encounter a message that > indicates to not continue? > - ... > > Are there better know solutions or best practices to delay message > processing with Kafka streams / consumers+producers? > > Thanks for any insights/help here! > -wim > -- -- Guozhang
