Hi, Deletion of messages in Kafka shouldn't affect Flink state in general. Probably, some operator in your pipeline is re-reading the topic and overwrites the state, dropping what was deleted by Kafka. Could you share the code?
Regards, Roman On Tue, Feb 23, 2021 at 7:12 AM bat man <tintin0...@gmail.com> wrote: > Hi, > > I have 2 streams one event data and the other rules. I broadcast the rules > stream and then key the data stream on event type. The connected stream is > processed thereafter. > We faced an issue where the rules data in the topic got deleted because of > Kafka retention policy. > Post this the existing rules data also got dropped in the broadcast state > and the processing stopped. > > As per my understanding the rules which were present in broadcast state > should still exist even if the data was deleted in Kafka as the rules dats > was already processed and stored in state map. > > PS: I’m reusing the rules stream as broadcast later in processing as well. > Could this be an issue? > > Thanks, > Hemant >
