Hello all,
I would like to propose a change in how the Flink master interacts with
Flink-shaded.
TL;DR: Release snapshot artifacts for flink-shaded, and have the Flink
master rely on specific snapshot versions for earlier dependency bumps.
Aa a project we have come to the general conclusion that dependencies
should be bumped as early in the release cycle as possible. This both
prevents cases where some undefined amount of work is still waiting for
as when we want to release the next version (working against the goal of
always being in a releasable state), and it gives us more time to
evaluate the stability and performance of system. Finally it gives us
ample time to look for alternatives if an issue is found.
Currently, this conclusion is at odds with how we handle flink-shaded.
Flink has always relied on flink-shaded artifacts that went through a
proper release cycle. However, since we want to create as few releases
as possible due to the overhead/noise/etc., flink-shaded releases are
typically relegated to the end of the release cycle.
This is particularly troublesome since flink-shaded dependencies are
used in the core of Flink, and hence usage of them cannot be avoided.
As a compromise between these 2 goals I propose the following:
- we deploy SNAPSHOT artifacts for flink-shaded for every change made
- every deployed artifact has a unique version, that is automatically
set via maven (=> no overhead on our side)
- once such an artifact is released we update the Flink dependency to
point to this _specific_ flink-shaded snapshot artifact
- to be clear, this is a manual step, which implies that things
cannot break all of a sudden because something was pushed to flink-shaded
- once the Flink release cycle ends, we publish a proper flink-shaded
release, and change the Flink dependency in the release branch accordingly
This should give us the best of both worlds: We have as few releases as
necessary (at most 1 per Flink release cycle), but can update the
dependencies in Flink as soon as possible.
Furthermore, this can also be considered a test run for how multiple
repos with the same release cycle could be developed in sync with each
other.
Let me know what you think.
Regards,
Chesnay
