Hello Guozhang, we ingest messages that we outgest is user facing datastore, after some additional processing. Due to GDPR, we can only retain that unanonymised data for a maximum period of time. Let's say 6 months.
So, right before sending the data to the out topic, we'll branch the data into an anonymisation leg of the processing chain. Now, we want to keep the messages in that chain until the expiration of the unanonymised messages. At that point we want to send the anonymous record to the outgoing topic so that the anonymous message replaces the unanonymised version. Does that make the problem/idea more clear? -- wim On Fri, 9 Mar 2018 at 00:47 Guozhang Wang <wangg...@gmail.com> wrote: > Hello Wim, > > Not sure if I understand your motivations for delayed processing, could you > elaborate a bit more? Do you want to process raw messages, or do you want > to process anonymised messages? > > > Guozhang > > > On Thu, Mar 8, 2018 at 12:35 PM, Wim Van Leuven < > wim.vanleu...@highestpoint.biz> wrote: > > > Hello, > > > > I'm wondering how to design a KStreams or regular Kafka application that > > can hold of processing of messages until a future time. > > > > This related to EU's data protection regulation: we can store raw > messages > > for a given time; afterwards we have to store the anonymised message. > So, I > > was thinking about branching the stream, anonymise the messages into a > > waiting topic and than continue from there until the retention time > passes. > > > > But that approach has some caveats: > > > > - This is not an exact solution as order of events is not guaranteed: > we > > might encounter a message that triggers the stop processing while some > > events arriving later should normally still pass > > - how to stop properly stop processing if we encounter a message that > > indicates to not continue? > > - ... > > > > Are there better know solutions or best practices to delay message > > processing with Kafka streams / consumers+producers? > > > > Thanks for any insights/help here! > > -wim > > > > > > -- > -- Guozhang >
