Hi Christian, disclaimer: IANAL, so take everything with a grain of salt from the legal perspective, I'm sharing the experience I have handling PII data with Kafka in an ecommerce system, so your requirements may differ.
I'm not sure how your system is designed but in general from a data management perspective you can consider the following: If you have a non-log-compacted topic, after `retention.ms` the message (along with the PII) gets deleted from the Kafka message store without any further action, which should satisfy GDPR requirements: - you are handling PII in Kafka for a limited amount of time - you are processing the data for the given purpose it was given - the data will automatically be deleted without any further steps If you have a downstream system, you should also be able to publish a message through Kafka so that the downstream system executes its delete processes - if required. We implemented a similar process where we published an AnonymizeOrder event, which instructed downstream systems to anonymize the order data in their own data store. If you have a log-compacted topic: - yes, I have the same understanding as you have on the active segment. - You can set the segment.ms <https://kafka.apache.org/documentation/#segment.ms> property to force the compaction to occur within an expected timeframe. In general what I understand is true in both cases that Kafka gives you good enough guarantees to either remove the old message after retention.ms milliseconds or execute the topic compaction after segment.ms time that it is unnecessary to try to figure out more specifically in what exact moment the data is deleted. Setting these configurations should give you enough guarantee that the data removal will occur - if not, that imo should be considered a bug and reported back to the project. >From the GDPR point-of-view if you set these values reasonably low enough (couple of days) that should be acceptable as you have one month to comply with the delete request (see https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/individual-rights/right-to-erasure/#:~:text=The%20GDPR%20introduces%20a%20right,to%20respond%20to%20a%20request. ) I hope this helps! Kind regards, Sandor On Wed, 19 Aug 2020 at 18:43, Apolloni, Christian < christian.apoll...@baloise.ch> wrote: > On 2020/08/19 16:15:40, Nemeth Sandor <s...@gmail.com> wrote: > > Hi Christian,> > > Hi, thanks for your reply. > > > depending on how your Kafka topics are configured, you have 2 different> > > options:> > > > > a) if you have a non-log-compacted then you can set the message > retention> > > on the topic to the desired value. In that case the message will be > deleted> > > by Kafka after the retention period expires. (the config value is `> > > retention.ms` I think)> > > That's what we thought too at first as solution, but we likely cannot set > the retention low enough. > > > b) if you use Kafka as a log store with topics having infinite > retention,> > > then one common solution is to send a so-called tombstone record (a > record> > > with the same key containing only GDPR compatible data with the > sensitive> > > information removed), and let Kafka take care of the removal using log> > > compaction.> > > We also thought about this, but as far as we understood there is no real > guarantee that the compaction completes in a given time for all messages in > the topic. From what we understood compaction can be delayed by the > messages still being in the active segment and/or the compaction thread > pool being too busy. > > It's also unclear to us how we can know that the compaction has completed > for all relevant messages and that we can safely report to our "coordinator > system" that the next system can start its own deletion process safely. > > Kind Regards, > > -- > Christian Apolloni > > > Disclaimer: The contents of this email and any attachment thereto are > intended exclusively for the attention of the addressee(s). The email and > any such attachment(s) may contain information that is confidential and > protected on the strength of professional, official or business secrecy > laws and regulations or contractual obligations. Should you have received > this email by mistake, you may neither make use of nor divulge the contents > of the email or of any attachment thereto. In such a case, please inform > the email's sender and delete the message and all attachments without delay > from your systems. > You can find our e-mail disclaimer statement in other languages under > http://www.baloise.ch/email_disclaimer >
