Thanks a lot for your proposal, I'm generally open to the idea I have a few questions: a) Does this mean that we are changing flink-shaded to deploy snapshot artifacts to Apache's snapshot maven repository, and change Flink's parent pom to point to this snapshot repo? b) How do you plan to generate the unique SNAPSHOT version on CI? Will we increment the version on every push to flink-shaded:master ? c) How do the unique SNAPSHOT versions relate to the final release versions?
On Mon, Apr 12, 2021 at 1:48 PM Konstantin Knauf <kna...@apache.org> wrote: > Sounds good. +1 > > On Mon, Apr 12, 2021 at 1:23 PM Chesnay Schepler <ches...@apache.org> > wrote: > > > Hello all, > > > > I would like to propose a change in how the Flink master interacts with > > Flink-shaded. > > > > TL;DR: Release snapshot artifacts for flink-shaded, and have the Flink > > master rely on specific snapshot versions for earlier dependency bumps. > > > > > > Aa a project we have come to the general conclusion that dependencies > > should be bumped as early in the release cycle as possible. This both > > prevents cases where some undefined amount of work is still waiting for > > as when we want to release the next version (working against the goal of > > always being in a releasable state), and it gives us more time to > > evaluate the stability and performance of system. Finally it gives us > > ample time to look for alternatives if an issue is found. > > > > Currently, this conclusion is at odds with how we handle flink-shaded. > > Flink has always relied on flink-shaded artifacts that went through a > > proper release cycle. However, since we want to create as few releases > > as possible due to the overhead/noise/etc., flink-shaded releases are > > typically relegated to the end of the release cycle. > > This is particularly troublesome since flink-shaded dependencies are > > used in the core of Flink, and hence usage of them cannot be avoided. > > > > As a compromise between these 2 goals I propose the following: > > - we deploy SNAPSHOT artifacts for flink-shaded for every change made > > - every deployed artifact has a unique version, that is automatically > > set via maven (=> no overhead on our side) > > - once such an artifact is released we update the Flink dependency to > > point to this _specific_ flink-shaded snapshot artifact > > - to be clear, this is a manual step, which implies that things > > cannot break all of a sudden because something was pushed to flink-shaded > > - once the Flink release cycle ends, we publish a proper flink-shaded > > release, and change the Flink dependency in the release branch > accordingly > > > > This should give us the best of both worlds: We have as few releases as > > necessary (at most 1 per Flink release cycle), but can update the > > dependencies in Flink as soon as possible. > > Furthermore, this can also be considered a test run for how multiple > > repos with the same release cycle could be developed in sync with each > > other. > > > > Let me know what you think. > > > > Regards, > > > > Chesnay > > > > > > -- > > Konstantin Knauf > > https://twitter.com/snntrable > > https://github.com/knaufk >
