Hi,

recently a user reported an issue upgrading a Kafka Streams application
from 2.2.0 to 2.2.1 (cf
https://mail-archives.apache.org/mod_mbox/kafka-users/201908.mbox/<CAG8RrxkDAhCrGQ1%3DpBpX-Jcxagk_xkmRk4MBQeZYAP6ZGHoK_w%40mail.gmail.com>)

After some investigation, we identified
https://issues.apache.org/jira/browse/KAFKA-7895 to be the root cause of
the problem.

The fix for KAFKA-7895 is using message headers and thus requires broker
version 0.11.0 (or newer) and message format 0.11 (or newer). Hence,
while a Kafka Streams application version 2.2.0 is compatible to older
brokers (0.10.1 and 0.10.2) and only requires message format 0.10, the
backward compatibility was broken accidentally in 2.2.1.

The fix is also contained in 2.3.0 release and cherry-picked to 2.1
branch (2.1.2 is not released yet, and thus 2.1 users are not affected
as this point).

Note: only users that use `suppress()` operator in their program are
affected.

We should not break streams-broker backward compatibility in bug-fix
releases at all and avoid for minor releases. However, it seems
necessary to have the fix in 2.3.0 though -- otherwise, `suppress()` is
effectively useless and it does not seem to be a good idea to fix the
bug only in the next major release. Hence, trading-off some backward
compatibility in a minor release seems to be acceptable for this case,
considering that 0.11.0 was release 2 years ago.

For 2.2.1, it is more challenging to decide how to move forward, because
we should not have broken streams-broker compatibility but 2.2.1 is
already released and we can only react after the fact.

  From my point of view, the best way is to keep the fix and update the
release notes and documentation accordingly. The main reason for my
suggestions is that we would expect a majority of users to be on 0.11.0
brokers already and the fix will work for them -- reverting the fix in
2.2.2 seems to be worse for all those users on newer broker versions. We
also know that `suppress()` is a highly demanded feature and a lot of
people waited for a fix.

  The expected minority of users that are on 0.10.1 / 0.10.2 brokers, or
newer brokers but still on message format 0.10 would either need to stay
on Kafka Streams 2.2.0 or upgrade their brokers/message format
accordingly. However, upgrading brokers/message format is de-facto
already required for 2.2.1 and thus keeping the fix would not make the
situation worse.

For 2.1, I would suggest to revert the fix to make sure we don't break
streams-broker compatibility for 2.1.x users. If those users need the
fix for `suppress()` they need to upgrade to 2.2.1/2.3.0 or newer and
make sure their brokers are on 0.11.0 with message format 0.11, too.


TL;DR; the proposal is:

(1) revert the fix for KAFKA-7895 in 2.1 branch
(2) keep the fix for KAFKA-7895 in 2.2.1 and 2.3.0

Impact:

 - Kafka Streams 2.1.x and 2.2.0 applications are backward compatible
back to 0.10.1 brokers, requiring message format 0.10
 - Kafka Streams 2.2.2 / 2.3.0 application (or newer) are backward
compatible back to 0.11.0 brokers, requiring message format 0.11


Please let us know what you think about this proposal.


-Matthias



Attachment: signature.asc
Description: OpenPGP digital signature

Reply via email to