Re: [DISCUSS] KIP-1277: Support Delayed Message in Kafka

[Matthias J. Sax]

Hello Henry,

skimming over the KIP and discussion thread, I am not sure if this is 
something we want to add to Kafka? At least no in the current proposed way.

While Kafka (via KIP-932) now supports "share groups", Kafka is still a 
log, not a queue. The term "Queues for Kafka" is mainly a marketing 
phrase and KIP-932 deliberately introduced the technically correctly 
term "share groups".

I agree to all concerns raised to far. I have doubts if it would be the 
right direction to transform Kafka into a hybrid log/queuing system? In 
my experience, systems stay strong if they focus on their core 
capabilities, and expanding the scope of supported use-cases too much, 
can actually do more harm than good?

In the end, KIP-932 did follow the path to _not_ change Kafka. Topics 
are still logs. There is no special "queuing topics"; the core change 
which was added (and this by itself is a massive change and was a 
multi-year effort), is the ability to do per-message acks. The design of 
KIP-932 is really tailored to Kafka's architecture; KIP-932 does not try 
to change Kafka in its core.

In the end, at least how I see it personally, KIP-932 is a kafkaesque 
way to mimic some queuing system behavior. And I mean this in a very 
positive way! The goal was never to make Kafka a queue.

I fully acknowledge the desire to add a "delayed message deliver" 
feature. Transforming Kafka into a queuing system might take it a step 
too far thought?

Just my 2ct.


-Matthias




On 2/11/26 3:37 PM, Henry Haiying Cai via dev wrote:
Thanks for various comments/suggestions on the KIP from the community, that 
demonstrate a pain point many people are facing when Kafka is lacking support 
for some generic queuing feature in delayed/scheduled messages.

Among the suggestions, we think the idea of setting the delay on the 
topic/queue level is the most easy to implement since the queue's message FIFO 
insertion order will be maintained and there is no merge sorting needed on the 
dequeue time. The downside of this design is we need to create multiple of 
those queues with different delay setting and there is a latency degradation 
since the message will needs to be routed to a queue which has a higher delay 
setting than the original message delay setting, we are working with our 
application team to see whether this workaround is acceptable.

Although people might feel there are quite many changes needed to support this 
delayed message feature, the basic implementation idea is not complex. Adding 
an Index and doing a sorting based on message delay time during dequeue. This 
idea can be easily extended to support other queueing feature e.g. priority 
queue support. In the case of priority queue, the dequeue sorting order will be 
based on the message's priority score instead of delay time.  If we use this 
opportunity to support delayed/scheduled message with arbitrary delay length 
and other dequeue features such as priority queue, it might be a worthwhile 
extension to Kafka to keep parity with alternative queueing solutions such as 
RabbitMQ/Redis/SQS.

Here are some specific replies:

For David Arthur on the question of use cases for delayed messages and whether 
it can be solved by consumer throttling. Delayed/scheduled message is a common 
queueing feature and implemented in several other queuing system (e.g. 
SQS/Redis/RabbitMQ). Besides a few use cases I mentioned in the KIP, other 
people also use delayed/scheduled message to target message delivery/processing 
at certain time of the day to maximum the message visibility, to schedule a 
message to be delivered at a later time when other dependencies for the message 
processing are believed to be more likely satisfied, or the writer of the 
message wants to have more control on when the message should be consumed. Not 
all these cases can be solved by consumer read throttling and Kafka's consumer 
throttling support is also very basic.

For Jian Fu on the suggestion of using multiple delayed topic/queues and have 
the delay setting on the topic/queue level. This is a very intriguing 
suggestion since it would simplify the design a lot, the queue's message FIFO 
insertion order will be maintained and there is no merge sorting needs to 
happen on the dequeue time. The downside of the design is we need to create 
multiple of those queues and there is a latency degradation that the message 
will needs to be routed to a queue which has a higher delay setting than the 
original message delay setting, we are working with our application team to see 
whether this workaround is acceptable.

For Andrew on the question on support of transaction and exact-once delivery. 
We chose not to support transactions in the initial implementation because the 
semantics on delayed message inside transaction is not clear. Maybe the 
semantics is all the messages in that transaction will get the same delay 
settings. We can open up for more discussions on this if people feel there is 
use cases to combine delayed messages and normal messages in the same 
transaction. On the question of whether we need to decompress the whole message 
batch to get individual message's delay setting, the answer is no. There is a 
section in the KIP on the batch processing. Although the design is using the 
delay setting on a message to illustrate the use case and flow but the 
implementation will be setting the delay on the batch level for performance 
reason. All the messages in the same producer batch will be delay delivered at 
the same time. The producer will set the batch delay either explicitly or 
implicitly (meaning the producerBatch code will auto adjust the batch delay 
setting based on message delays inside the batch and in some cases it might 
decide to split the batch if the message delays insides in the same batch are 
too far apart).

For Chia-ping's idea on having client hold/buffer the message, this is a good 
idea and we can iterate more on this approach. But as Andrew mentioned, this 
hold/buffer mechanism will affect other normal message delivery. And also we 
intended the delay message for all type of messages, not just the messages for 
the share group. Another cons on this approach is we the logic on the client 
will be more complex and we would have to implement this logic for clients 
writing in different programming languages.  I think the current Kafka 
community stand on the client side design is to keep the client logic light and 
do the heavy lifting on the broker side.  On the concern of introducing a new 
DelayAfter index file, we can avoid generating this index file by sorting the 
log segment based on delayAfter value when we close the current active log 
segment file.  This way there is no extra index file need to be uploaded to 
tiered storage either.

For Colt/Max's idea of using an external system (e.g. Kafka stream or Apache 
Camel) to handle the delay/scheduling portion, this is a common workaround. 
However this introduce the dependency on building and maintaining another 
service. Our client is not on Java stack and they already have a working 
version using SQS. They are more interested to have an architecture removing 
the secondary scheduling system.

Henry Cai and Tom




On Tuesday, February 3, 2026 at 07:49:29 AM PST, David Arthur via dev 
<dev@kafka.apache.org> wrote:





I would be interested to hear more specifics about the use cases. I wonder
how much can be solved with existing or planned features in Kafka.

In the KIP, two are mentioned:

1) The use case is a large influx of messages or other activities happening
in the system at the moment, the message producer wants to make sure the
messages are being consumed/processed in a little bit later time or the
message consumption is being spread over a period of time.

This just sounds like how normal topics work. A large number of messages
are written into a topic, and they are consumed later on as quickly as the
system allows (limited by the brokers and consumers). Would consumer
throttling be enough to satisfy this case?


2) Another common use is message retry handling (e.g. retries in the new
Kafka Queue feature), when a message consumer/worker cannot process the
message due to some transient failures in external systems, usually the
worker wants to unacknowledge the message and retry it later.

"Unacknowledge" definitely sounds like KIP-932 concern. I believe retry
with backoff is something planned for QfK.


I don't deny that this sounds like a useful feature, but I'd like to know
more details about what problems we're solving.

Thanks!
David





On Tue, Feb 3, 2026 at 6:27 AM jian fu <fujian1...@gmail.com> wrote:

Hi Henry and  All,

Thanks for the KIP. I believe this is a feature that many people want to
have.

However, if the design target is set to 15 minutes, then, when considering
the complexity of the currently proposed solution, the cost–benefit ratio
may not be as compelling as we might hope.

That said, if our primary goal is simply to provide a delay feature with
some reasonable constraints, perhaps we could consider supporting a fixed
delay feature per topic — for instance, a topic configured with a 5-minute
delay or a 10-minute delay.

This could help avoid the need to handle multiple different delay durations
within a single topic. In that case, a fixed consumption delay could be
implemented on the client side by pausing polling, which would likely
require only limited additional code.

Anyway, I may have strayed slightly from the original focus of the
discussion, and I apologize for that.

Finally, thank you very much for raising such an interesting and
thought-provoking topic !


Regards

Jian


Andrew Schofield <schofie...@apache.org> 于2026年2月1日周日 02:22写道：

Hi Henry and others,
Thanks for the KIP. I see there’s already been lively discussion.

My initial thought is that the first version of the KIP doesn’t work with
a lot of existing Kafka features and breaks a lot of assumptions. For
example, the offset order of consumed records is no longer monotonically
increasing, there’s no support for EOS because there’s no longer a
specific
offset that cleanly delineates completed transactions from open
transactions, and so on.

I think that one of the principles of a successful KIP of this kind is
that it embraces all of the Kafka features which went before. This is why
KIP-932 supports isolation level and tiered storage. There’s also one
other
relevant detail of KIP-932 which is that the broker never looks within
the
record batches. We don’t want to decompress compressed batches on the
broker, so we do not iterate over the records. I worry about having a
per-record delay and building an index accordingly.

You mention using delayed message support as a way of introducing delay
into message retries with share groups. I think that could be more
readily
achieved in the existing share-partition code.

Chia-Ping’s suggestion for essentially considering this as an enhancement
for share groups is interesting. Regular consumers would be unaware of
the
delay and they would still get records in offset order. Share consumers
are
not guaranteed to get records in offset order so I think we have more
flexibility there. But I see a couple of problems with this too. First,
if
delayed records are acquired by share consumers when they are not yet
intended to be delivered, they will be eating into the in-flight record
limit for the partition. If there are lots of delayed records ahead of
records with no delay, they could temporarily prevent delivery of the
undelayed records. Second, I don’t believe restricting this feature to
share groups is quite what was intended. I think you were aiming for a
general-purpose feature for delayed messages in Kafka.

Max suggested a scheduler topic. That possibly could work, but it gets
tricky to support features like transactions. What if an application
puts a
record with a long delay and a short delay in the same transaction? Are
control records written onto the scheduler topic? How do we atomically
move
records from the scheduler topic onto the user’s topic, because I would
say
that the user’s expected quality-of-service (EOS, idempotent produce)
would
have to be honoured by the copying process? I think that a per-producer
delay is better with this design, because all records put in a
transaction
would have consistent delays and thus could be grouped together. This
would
be an interesting problem to solve. There’s also the fact that the
leadership of the user’s topic-partitions is not going to be the same as
the leadership of the scheduler topic-partitions. There would be
inter-broker calls, just as there are for DLQs for share groups.

Hopefully the comments spark some ideas. This is quite a tough problem to
solve neatly in Kafka.

Thanks,
Andrew

On 2026/01/31 07:18:05 Chia-Ping Tsai wrote:
Hi Henry and everyone,

Thanks for the KIP! Delayed messaging is indeed a highly requested
feature.

I've been following the development of **KIP-932 (Share Groups)** and
the recent **KIP-1222 (Acquisition lock timeout renewal)**, and I’m
wondering if we can achieve the same goal by leveraging Share Consumers
on
the client side, rather than modifying the Broker's storage engine.

The main concern with KIP-1277 is the complexity introduced to the
Broker (new indexes, modifying LogSegment logic) and the potential
performance penalty of breaking Zero-Copy (since the broker needs to
unpack
batches to filter out delayed messages).

**Alternative Proposal: Client-Side Delay with Share Groups**

With KIP-932 and KIP-1222, we might have enough primitives to build a
robust **Client-Side Delay** mechanism that keeps the Broker stateless
and
efficient:

1. **Protocol**: Producers simply add a `__kafka_delay_timestamp`
header. The Broker treats it as a normal message and delivers it via
Zero-Copy fetch.

2. **Client-Side Filtering**: The Share Consumer SDK parses the header.
* If `now >= timestamp`: Process immediately.
* If `now < timestamp`: **Do not** return to the user application.

3. **The "Hold & Renew" Strategy**:
* Instead of `RELEASE`-ing the message back to the broker (which causes
busy loops), the Consumer buffers the record internally.
* Crucially, it uses the **`RENEW` action (from KIP-1222)** to maintain
the acquisition lock without processing the message.
* This effectively "hides" the message from other consumers while
preventing redelivery.

4. **Graceful Degradation (Memory Protection)**:
* To address the concern of buffering 1M+ delayed messages causing OOM:
* The Consumer can implement an eviction policy: Drop the **payload**
but keep the **Offset + Timestamp** in a lightweight PriorityQueue.
* The Consumer continues to `RENEW` the lock for these offsets
(extremely low overhead).
* When the time comes, if the payload was dropped, issue a `RELEASE` to
re-fetch the data from the broker.

**Trade-offs:**

* **Pros:**
* **Zero Broker Changes:** No new index files, no changes to log
segments.
* **Performance:** Preserves Zero-Copy for fetches. The Broker only
handles lightweight metadata (Renew RPCs).
* **Flexibility:** Logic resides in the Client SDK.


* **Cons:**
* **Traffic:** Consumers download delayed data ahead of time. (However,
for the "short time window" use case mentioned in KIP-1277, this seems
acceptable).
* **Head-of-Line Blocking:** If the *entire* batch is delayed messages,
it occupies the "In-flight" limit. However, KIP-1277 also targets "short
delays," so the impact might be manageable or solved via "Parking Lot"
topics for long delays.

Given that KIP-1277 also focuses on **short delays (up to 15 mins)**,
the Share Group approach seems to cover the use case well without the
heavy
lift on the storage engine.

Would love to hear your thoughts on whether this direction has been
considered.

Best,
Chia-Ping

On 2026/01/30 07:34:09 Henry Haiying Cai via dev wrote:
Hello Kafka Developers,

I would like to start discussing KIP-1277: Support Delayed Message in
Kafka.

A common queue scheduling feature is delayed messages where the
message is not supposed to be delivered or consumed right away.  The use
case is a large influx of messages or other activities happening in the
system at the moment, the message producer wants to make sure the
messages
are being consumed/processed in a little bit later time or the message
consumption is being spread over a period of time.  Another common use is
message retry handling (e.g. retries in the new Kafka Queue feature),
when
a message consumer/worker cannot process the message due to some
transient
failures in external systems, usually the worker wants to unacknowledge
the
message and retry it later.  The retry is usually more ideal to be
scheduled at a later time usually with some exponential backoff time
interval.

Since Kafka was lacking support for message scheduling or delayed
message delivery, users have turned to other queuing systems for these
features. For example, users have been using AWS SQS delayed message /
delayed topic to deliver messages later within a short time frame (e.g.
within 15 minutes) and use DynamoDB or traditional database tables for
the
delayed message for longer delayed duration.

We are proposing to implement delayed messages for Kafka to fill in
this feature gap. Similar to SQS, the proposal is also focusing on
messages
delayed delivery for a short time window (up to 15 minutes late).  In our
use cases, most of the message late delivery is within 1 minute.

KIP-1277 can be found here:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-1277%3A+Support+Delayed+Message+in+Kafka

Looking forward to suggestions and feedback :)

Best,
Henry Cai and Tom Thornton