Hi all,

As you may know, we've been working hard to graduate KIP-848 from
preview in Apache Kafka 3.8 to production-ready in Apache Kafka 4.0 —
exciting news!

KIP-848, the next-generation rebalance protocol, encompasses several
significant updates in the consumer and group coordinator areas,
including:
- A completely new rebalance protocol for the consumer.
- A redesigned group coordinator.
- A deep refactoring of the consumer to introduce a background thread
for handling all I/O operations, instead of performing them on the
application thread. Note that this refactor is only applied when the
new rebalance protocol is enabled — an important detail.

Starting with Apache Kafka 4.0:
- The new group coordinator will be the only option.
- The new rebalance protocol will be enabled by default on the server side.
- The consumer will continue to use the old rebalance protocol by
default for backward compatibility with brokers older than 4.0.

Our initial plan was to make the new rebalance protocol the default in
the Consumer in Apache Kafka 4.0. However, after much discussion, we
decided not to proceed with this approach — despite the feature being
ready. The key concern is that the consumer shipped with Apache Kafka
4.0 would not work by default with brokers older than 4.0, which we
believe would result in a poor user experience. Realistically, most
deployments won’t upgrade to 4.0 immediately, and we anticipate a
gradual adoption curve. Moreover, we feel that introducing such a
change without prior notice (e.g. a deprecation warning) in earlier
releases isn't ideal.

We also considered implementing an automatic fallback mechanism in the
consumer. However, the significant differences between the two
protocols made this unfeasible. For example:
- Static membership does not function identically in both protocols.
- Consumer-triggered rebalances are not supported in the new protocol.
- Default assignors are different.
- Metrics and logs are different.
- Some configs are no longer supported.
- There are small behavioral differences due to the new internal
architecture of the Consumer.

Given these differences, we believe users must explicitly choose when
to migrate from the old protocol to the new protocol as already
planned in the KIP. To aid this process, we are providing a clear
online upgrade path.

Instead, our revised plan is as follows:
- We will officially deprecate the old rebalance protocol in a 4.x
release (likely 4.1) and introduce the appropriate deprecation
message.
- We plan to make the new rebalance protocol the default in Apache
Kafka 5.0, giving the community sufficient time to adopt 4.x brokers
before the switch.
- We will drop the possibility of using the old rebalance protocol
from the Consumer in Apache Kafka 6.0.

I will formalize this revised plan in a KIP to discuss it further with
the community.

Regards,
David, on behalf of the KIP-848 team

P.S.: I started this new thread to ensure this update isn’t unnoticed
in the long KIP-848 discussion thread.

Reply via email to