@Robert We can workaround the snapshot limit issue fairly easily; this
limit is imposed per version, so if we modify the version to include the
commit hash this limit does not apply. This should also make it easier
to work with from the Flink side because a commit hash is easier to
copy&paste than figuring out which timestamp you need.
@Till Yes, the release process involves too many manual steps, and I
don't want to spend time on it unless I'm sure we don't need another
one. Many of these cannot be automated AFAIK, like the book-keeping at
reporter.apache.org, preparing and handling the voting threads,
preparing the flink-web PR, managing the snapshot repository, and so on.
As for removing the SNAPSHOT version, we can enable an enforcer rule to
forbid snapshot dependencies. It's a good safeguard in any case. We can
furthermore automate the removal, and do it as part of the release
branch creation.
On 4/13/2021 10:47 AM, Till Rohrmann wrote:
Thanks for creating this proposal Chesnay. I do understand the problem
you want to fix.
What I am wondering is why we don't release flink-shaded more often.
Does the release process cause too much overhead? If this is the case,
then we could look into what is causing the overhead and whether we
can improve the situation. Concerning the noise, I personally don't
see it as a problem.
My main concern is that it can easily slip our minds to change the
flink-shaded SNAPSHOT version to a non SNAPSHOT version and that it
introduces another manual step. If we forget to change the version,
then the Flink release does not build against a stable set of
dependencies. Moreover, I also second Robert's concern that a single
commit to flink-shaded can then break downstream projects (Flink in
this case) if we rely on the SNAPSHOT builds. Having to scan poms for
some references sounds like an indicator that this might not be the
most straight forward approach.
Cheers,
Till
On Tue, Apr 13, 2021 at 9:26 AM Robert Metzger <rmetz...@apache.org
<mailto:rmetz...@apache.org>> wrote:
Thanks a lot for your responses.
I didn't know that you can explicitly refer to the timestamped
snapshots of
the artifacts. The limitation to the last 2 snapshots means that a
push to
flink-shaded can break our main CI? This sounds very fragile to
me, given
that the setup itself is probably a bit uncommon and difficult to
understand.
Maybe we should add an automated check to flink-shaded that warns
if a PR
would break Flink's CI? (by checking out flink and scanning the
poms for
references to a timestamp-to-be-deleted)
Or we ask Infra to keep more than two snapshots for flink-shaded?
On Mon, Apr 12, 2021 at 4:41 PM Chesnay Schepler
<ches...@apache.org <mailto:ches...@apache.org>> wrote:
> a) yes.
> b) maven by default adds a timestamp to snapshot artifacts that
we can
> use. The apache repository retains the last 2 snapshots, so we
do need
> to keep things in sync a fair bit, but there are rarely commits
made in
> flink-shaded that I don't think this will be a problem.
> c) a-SNAPSHOT-uniquesuffix => a.0
>
> On 4/12/2021 3:07 PM, Robert Metzger wrote:
> > Thanks a lot for your proposal, I'm generally open to the idea
> >
> > I have a few questions:
> > a) Does this mean that we are changing flink-shaded to deploy
snapshot
> > artifacts to Apache's snapshot maven repository, and change
Flink's
> parent
> > pom to point to this snapshot repo?
> > b) How do you plan to generate the unique SNAPSHOT version on
CI? Will we
> > increment the version on every push to flink-shaded:master ?
> > c) How do the unique SNAPSHOT versions relate to the final release
> versions?
> >
> >
> >
> >
> > On Mon, Apr 12, 2021 at 1:48 PM Konstantin Knauf
<kna...@apache.org <mailto:kna...@apache.org>>
> wrote:
> >
> >> Sounds good. +1
> >>
> >> On Mon, Apr 12, 2021 at 1:23 PM Chesnay Schepler
<ches...@apache.org <mailto:ches...@apache.org>>
> >> wrote:
> >>
> >>> Hello all,
> >>>
> >>> I would like to propose a change in how the Flink master
interacts with
> >>> Flink-shaded.
> >>>
> >>> TL;DR: Release snapshot artifacts for flink-shaded, and have
the Flink
> >>> master rely on specific snapshot versions for earlier
dependency bumps.
> >>>
> >>>
> >>> Aa a project we have come to the general conclusion that
dependencies
> >>> should be bumped as early in the release cycle as possible.
This both
> >>> prevents cases where some undefined amount of work is still
waiting for
> >>> as when we want to release the next version (working against
the goal
> of
> >>> always being in a releasable state), and it gives us more
time to
> >>> evaluate the stability and performance of system. Finally it
gives us
> >>> ample time to look for alternatives if an issue is found.
> >>>
> >>> Currently, this conclusion is at odds with how we handle
flink-shaded.
> >>> Flink has always relied on flink-shaded artifacts that went
through a
> >>> proper release cycle. However, since we want to create as
few releases
> >>> as possible due to the overhead/noise/etc., flink-shaded
releases are
> >>> typically relegated to the end of the release cycle.
> >>> This is particularly troublesome since flink-shaded
dependencies are
> >>> used in the core of Flink, and hence usage of them cannot be
avoided.
> >>>
> >>> As a compromise between these 2 goals I propose the following:
> >>> - we deploy SNAPSHOT artifacts for flink-shaded for every
change made
> >>> - every deployed artifact has a unique version, that is
automatically
> >>> set via maven (=> no overhead on our side)
> >>> - once such an artifact is released we update the Flink
dependency to
> >>> point to this _specific_ flink-shaded snapshot artifact
> >>> - to be clear, this is a manual step, which implies
that things
> >>> cannot break all of a sudden because something was pushed to
> flink-shaded
> >>> - once the Flink release cycle ends, we publish a proper
flink-shaded
> >>> release, and change the Flink dependency in the release branch
> >> accordingly
> >>> This should give us the best of both worlds: We have as few
releases as
> >>> necessary (at most 1 per Flink release cycle), but can
update the
> >>> dependencies in Flink as soon as possible.
> >>> Furthermore, this can also be considered a test run for how
multiple
> >>> repos with the same release cycle could be developed in sync
with each
> >>> other.
> >>>
> >>> Let me know what you think.
> >>>
> >>> Regards,
> >>>
> >>> Chesnay
> >>>
> >>>
> >> --
> >>
> >> Konstantin Knauf
> >>
> >> https://twitter.com/snntrable <https://twitter.com/snntrable>
> >>
> >> https://github.com/knaufk <https://github.com/knaufk>
> >>
>
>
