Hello Kafka Developers, I would like to start discussing KIP-1277: Support Delayed Message in Kafka.
A common queue scheduling feature is delayed messages where the message is not supposed to be delivered or consumed right away. The use case is a large influx of messages or other activities happening in the system at the moment, the message producer wants to make sure the messages are being consumed/processed in a little bit later time or the message consumption is being spread over a period of time. Another common use is message retry handling (e.g. retries in the new Kafka Queue feature), when a message consumer/worker cannot process the message due to some transient failures in external systems, usually the worker wants to unacknowledge the message and retry it later. The retry is usually more ideal to be scheduled at a later time usually with some exponential backoff time interval. Since Kafka was lacking support for message scheduling or delayed message delivery, users have turned to other queuing systems for these features. For example, users have been using AWS SQS delayed message / delayed topic to deliver messages later within a short time frame (e.g. within 15 minutes) and use DynamoDB or traditional database tables for the delayed message for longer delayed duration. We are proposing to implement delayed messages for Kafka to fill in this feature gap. Similar to SQS, the proposal is also focusing on messages delayed delivery for a short time window (up to 15 minutes late). In our use cases, most of the message late delivery is within 1 minute. KIP-1277 can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1277%3A+Support+Delayed+Message+in+Kafka Looking forward to suggestions and feedback :) Best, Henry Cai and Tom Thornton
