Hi All, I'd like to propose a KIP that transforms state store cleanup from a time-driven to event-driven approach by introducing changelog delete notifications.
Problem: Currently, state stores have no way to know when records are deleted from their changelog topics due to retention. This leads to: - Resource-intensive periodic scans - Blind cleanup operations - Inefficient resource utilization We face this at significant scale: - State stores with 25B+ records - Daily ingestion of 50M records per store - Retention periods from 2 days to 5 years Proposed Solution: Introduce a notification mechanism when records are deleted from changelog topics, enabling: - Event-driven cleanup instead of time-based scans - Targeted deletion of specific records - Better resource utilization Would love to get the community's thoughts on: 1. Viability of this approach 2. Implementation considerations (callbacks vs alternatives) 3. Potential impact on broker performance If there's interest, I can share more detailed technical design. Looking forward to your feedback. Best regards, Thulasiram V
