You should also be able to manage this with a compacted topic. If you give each message a unique key you'd then be able to delete, or overwrite specific records. Kafka will delete them from disk when compaction runs. If you need to partition for ordering purposes you'd need to use a custom partitioner that extracts a partition key from the unique key before it does the hash.
B On Sun, Nov 26, 2017 at 10:40 AM Wim Van Leuven < wim.vanleu...@highestpoint.biz> wrote: > Thanks, Lars, for the most interesting read! > > > > On Sun, 26 Nov 2017 at 00:38 Lars Albertsson <la...@mapflat.com> wrote: > > > Hi David, > > > > You might find this presentation useful: > > https://www.slideshare.net/lallea/protecting-privacy-in-practice > > > > It explains privacy building blocks primarily in a batch processing > > context, but most of the principles are applicable for stream > > processing as well, e.g. splitting non-PII and PII data ("ejected > > record" slide), encrypting PII data ("lost key" slide). > > > > Regards, > > > > > > > > Lars Albertsson > > Data engineering consultant > > www.mapflat.com > > https://twitter.com/lalleal > > +46 70 7687109 <+46%2070%20768%2071%2009> <+46%2070%20768%2071%2009> > > Calendar: http://www.mapflat.com/calendar > > > > > > On Wed, Nov 22, 2017 at 7:46 PM, David Espinosa <espi...@gmail.com> > wrote: > > > Hi all, > > > I would like to double check with you how we want to apply some GDPR > into > > > my kafka topics. In concrete the "right to be forgotten", what forces > us > > to > > > delete some data contained in the messages. So not deleting the > message, > > > but editing it. > > > For doing that, my intention is to replicate the topic and apply a > > > transformation over it. > > > I think that frameworks like Kafka Streams or Apache Storm. > > > > > > Did anybody had to solve this problem? > > > > > > Thanks in advance. > > >
