Hi Thulasiram,
Thanks for your interest in Kafka Streams!
Regarding the communication between clients and brokers,
Kafka brokers cannot send requests to clients, currently. Everything is
pulled by the clients. So the brokers cannot let the clients know about
deletions of events by sending them a notification. If we keep this
pull-only model, the notifications would need to be piggy-backed on a
response of an existing RPC that the client calls or a new RPC would
need to be introduced for notifications that the client calls and that
returns notifications back to the client.
Regarding the deletion of data, as far as I know, the Kafka brokers do
not have any notion of records when they delete topic segments when the
segments exceed the configured retention time. Maybe they know the
offsets of the deleted records but Kafka Streams does not store the
offsets of the records in the state stores. So there is no way to map
the deleted offsets in the changelog topic to the records stored in
state store.
Of course, there is always a solution to all of these challenges, but
the question is if they are worthwhile.
I believe a client-side solution with TTL is probably better suited for
your use case. There are solutions/examples out there for this.
Feel free to share a more detailed technical design in your solution.
Best,
Bruno
On 07.01.25 06:43, Thulasi Ram wrote:
Hi All,
I'd like to propose a KIP that transforms state store cleanup from a
time-driven to event-driven approach by introducing changelog delete
notifications.
Problem:
Currently, state stores have no way to know when records are deleted from
their changelog topics due to retention. This leads to:
- Resource-intensive periodic scans
- Blind cleanup operations
- Inefficient resource utilization
We face this at significant scale:
- State stores with 25B+ records
- Daily ingestion of 50M records per store
- Retention periods from 2 days to 5 years
Proposed Solution:
Introduce a notification mechanism when records are deleted from changelog
topics, enabling:
- Event-driven cleanup instead of time-based scans
- Targeted deletion of specific records
- Better resource utilization
Would love to get the community's thoughts on:
1. Viability of this approach
2. Implementation considerations (callbacks vs alternatives)
3. Potential impact on broker performance
If there's interest, I can share more detailed technical design.
Looking forward to your feedback.
Best regards,
Thulasiram V
