Hi, recently a user reported an issue upgrading a Kafka Streams application from 2.2.0 to 2.2.1 (cf https://mail-archives.apache.org/mod_mbox/kafka-users/201908.mbox/<CAG8RrxkDAhCrGQ1%3DpBpX-Jcxagk_xkmRk4MBQeZYAP6ZGHoK_w%40mail.gmail.com>)
After some investigation, we identified https://issues.apache.org/jira/browse/KAFKA-7895 to be the root cause of the problem. The fix for KAFKA-7895 is using message headers and thus requires broker version 0.11.0 (or newer) and message format 0.11 (or newer). Hence, while a Kafka Streams application version 2.2.0 is compatible to older brokers (0.10.1 and 0.10.2) and only requires message format 0.10, the backward compatibility was broken accidentally in 2.2.1. The fix is also contained in 2.3.0 release and cherry-picked to 2.1 branch (2.1.2 is not released yet, and thus 2.1 users are not affected as this point). Note: only users that use `suppress()` operator in their program are affected. We should not break streams-broker backward compatibility in bug-fix releases at all and avoid for minor releases. However, it seems necessary to have the fix in 2.3.0 though -- otherwise, `suppress()` is effectively useless and it does not seem to be a good idea to fix the bug only in the next major release. Hence, trading-off some backward compatibility in a minor release seems to be acceptable for this case, considering that 0.11.0 was release 2 years ago. For 2.2.1, it is more challenging to decide how to move forward, because we should not have broken streams-broker compatibility but 2.2.1 is already released and we can only react after the fact. From my point of view, the best way is to keep the fix and update the release notes and documentation accordingly. The main reason for my suggestions is that we would expect a majority of users to be on 0.11.0 brokers already and the fix will work for them -- reverting the fix in 2.2.2 seems to be worse for all those users on newer broker versions. We also know that `suppress()` is a highly demanded feature and a lot of people waited for a fix. The expected minority of users that are on 0.10.1 / 0.10.2 brokers, or newer brokers but still on message format 0.10 would either need to stay on Kafka Streams 2.2.0 or upgrade their brokers/message format accordingly. However, upgrading brokers/message format is de-facto already required for 2.2.1 and thus keeping the fix would not make the situation worse. For 2.1, I would suggest to revert the fix to make sure we don't break streams-broker compatibility for 2.1.x users. If those users need the fix for `suppress()` they need to upgrade to 2.2.1/2.3.0 or newer and make sure their brokers are on 0.11.0 with message format 0.11, too. TL;DR; the proposal is: (1) revert the fix for KAFKA-7895 in 2.1 branch (2) keep the fix for KAFKA-7895 in 2.2.1 and 2.3.0 Impact: - Kafka Streams 2.1.x and 2.2.0 applications are backward compatible back to 0.10.1 brokers, requiring message format 0.10 - Kafka Streams 2.2.2 / 2.3.0 application (or newer) are backward compatible back to 0.11.0 brokers, requiring message format 0.11 Please let us know what you think about this proposal. -Matthias
signature.asc
Description: OpenPGP digital signature
