[
https://issues.apache.org/jira/browse/KAFKA-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax reopened KAFKA-9323:
------------------------------------
> Refactor Streams' upgrade system tests
> ---------------------------------------
>
> Key: KAFKA-9323
> URL: https://issues.apache.org/jira/browse/KAFKA-9323
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: A. Sophie Blee-Goldman
> Priority: Major
>
> With the introduction of version probing in 2.0 and cooperative rebalancing
> in 2.4, the specific upgrade path depends heavily on the to & from version.
> This can be a complex operation, and we should make sure to test a realistic
> upgrade scenario across all possible combinations. The current system tests
> have gaps however, which have allowed bugs in the upgrade path to slip by
> unnoticed for several versions.
> Our current system tests include a "simple upgrade downgrade" test, a
> metadata upgrade test, a version probing test, and a cooperative upgrade
> test. This has a few drawbacks and current issues:
> a) only the version probing test tests "forwards compatibility" (upgrade from
> latest to future version)
> b) nothing tests version probing "backwards compatibility" (upgrade from
> older version to latest), except:
> c) the cooperative rebalancing test actually happens to involve a version
> probing step, and so coincidentally DOES test VP (but only starting with 2.4)
> d) each test itself tries to test the upgrade across different versions,
> meaning there may be overlap and/or unnecessary tests. For example, currently
> both the metadata_upgrade and cooperative_upgrade tests will test the upgrade
> of 1.0 -> 2.4
> e) worse, a number of (to, from) pairs are not tested according to the
> correct upgrade path at all, which has lead to easily reproducible bugs
> slipping past for several versions.
> f) we have a test_simple_upgrade_downgrade test which does not actually do a
> downgrade, and for some reason only tests upgrading within the range [0.10.1
> - 1.1]
> g) as new versions are released, it is unclear to those not directly involved
> in these tests and/or projects whether and what needs to be updated (eg
> should this new version be added to the cooperative test? should the old
> version be aded to the metadata test?)
> We should definitely fill in the testing gap here, but how to do so is of
> course up for discussion.
> I would propose to refactor the upgrade tests, and rather than maintain
> different lists of versions to pass as input to each different test, we
> should have a single matrix that we update with each new version that
> specifies which upgrade path that version combination actually requires. We
> can then loop through each version combination and test only the actual
> upgrade path that users would actually need to follow. This way we can be
> sure we are not missing anything, as each and every possible upgrade would be
> tested.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
