Delayed delivery is indeed a frequently needed feature. Especially because many people are coming to kafka from other messaging brokers, like ActiveMQ, that have delayed and scheduled message support built into the broker. Kafka is a a very powerful and complex beast, and At Dow Jones we prioritize Kafka itself as a pure messaging broker. We want ALL of Kafka resources to be dedicated to message delivery. The way we solved for delayed and scheduled delivery, we created a dedicated scheduler topic and a consumer set that checks for DESTINATION_TOPIC, DELAY, AT, and SCHEDULE headers that either represent the time since message creation to delay the message for, specific time to release the message at, or a cron like expression that would create a message at regular intervals. We have a standalone Apache Camel app that consumes from that topic, uses camel quartz integration to schedule it, and ultimately produces the preserved message into the destination topic when the time is right. This allows to keep Kafka very performant, not waste kafka's resources on storing and scheduling the delayed messages, and ultimate scale message streaming and message scheduling independently, Higher performance and lower cost. Especially at high volumes. Would we be implementing scheduling this way if it was available out of the box with Kafka. Probably no. But it is a working alternative. Maybe shipping scheduler as a companion package, similar to connect, is another approach? Max Fortun
> On Jan 30, 2026, at 10:56 AM, Colt McNealy via dev <[email protected]> > wrote: > > Hi Henry, > > Thanks for the KIP. This is quite interesting and it fills a use-case that > many people ask about. > > One thing we've done at LittleHorse is use a simple Kafka Streams topology > to simulate delayed production. We store messages in a state store until > their maturation time and then forward them onwards from a punctuator. > Overall, it's about 20 lines of code total to do this, and it works great. > > The advantages of your KIP over that simple solution are that a user of > 1277 wouldn't have to worry about managing a Streams topology (which > is...hard sometimes). On the other hand, the implementation seems like a > heavy lift, especially when reconciling multiple batches when tiered > storage is enabled. Is it worth the additional effort? > > Colt McNealy > Founder, www.littlehorse.io > > On 2026/01/30 07:34:09 Henry Haiying Cai via dev wrote: >> Hello Kafka Developers, >> >> I would like to start discussing KIP-1277: Support Delayed Message in > Kafka. >> >> A common queue scheduling feature is delayed messages where the message > is not supposed to be delivered or consumed right away. The use case is a > large influx of messages or other activities happening in the system at the > moment, the message producer wants to make sure the messages are being > consumed/processed in a little bit later time or the message consumption is > being spread over a period of time. Another common use is message retry > handling (e.g. retries in the new Kafka Queue feature), when a message > consumer/worker cannot process the message due to some transient failures > in external systems, usually the worker wants to unacknowledge the message > and retry it later. The retry is usually more ideal to be scheduled at a > later time usually with some exponential backoff time interval. >> >> Since Kafka was lacking support for message scheduling or delayed message > delivery, users have turned to other queuing systems for these features. > For example, users have been using AWS SQS delayed message / delayed topic > to deliver messages later within a short time frame (e.g. within 15 > minutes) and use DynamoDB or traditional database tables for the delayed > message for longer delayed duration. >> >> We are proposing to implement delayed messages for Kafka to fill in this > feature gap. Similar to SQS, the proposal is also focusing on messages > delayed delivery for a short time window (up to 15 minutes late). In our > use cases, most of the message late delivery is within 1 minute. >> >> KIP-1277 can be found here: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1277%3A+Support+Delayed+Message+in+Kafka >> >> Looking forward to suggestions and feedback :) >> >> Best, >> Henry Cai and Tom Thornton >>
