What CI resources do you actually intend use? Asking since the ASF GHA
resources are afaik quite overloaded.
On 05/01/2022 11:48, Martijn Visser wrote:
Hi everyone,
I wanted to summarise the email thread and see if there are any open items
that still need to be discussed, before we can finalise the discussion in
this email thread:
1. About having multi connectors in one repo or each connector in its own
repository
As explained by @Arvid Heise <ar...@apache.org> we ultimately propose to
have a single repository per connector, which seems to be favoured in the
community.
2. About having the connector repositories under ASF or not.
The consensus is that all connectors would remain under the ASF.
I think we can categorise the questions or concerns that are brought
forward as the following one:
3. How would we set up the testing?
We need to make sure that we provide a proper testing framework, which
means that we provide a public Source- and Sink testing framework. As
mentioned extensively in the thread, we need to make sure that the
necessary interfaces are properly annotated and at least @PublicEvolving.
This also includes the test infrastructure, like MiniCluster. For the
latter, we don't know exactly yet how to balance having publicly available
test infrastructure vs being able to iterate inside of Flink, but we can
all agree this has to be solved.
For testing infrastructure, we would like to use Github Actions. In the
current state, it probably makes sense for a connector repo to follow the
branching strategy of Flink. That will ensure a match between the released
connector and Flink version. This should change when all the Flink
interfaces have stabilised so you can use a connector with multiple Flink
versions. That means that we should have a nightly build test for:
- The `main` branch of the connector (which would be the unreleased
version) against the `master` branch of Flink (the unreleased version of
Flink).
- Any supported `release-X.YY` branch of the connector against the
`release-X.YY` branch of Flink.
We should also have a smoke test E2E tests in Flink (one for DataStream,
one for Table, one for SQL, one for Python) which loads all the connectors
and does an arbitrary test (post data on source, load into Flink, sink
output and compare that output is as expected.
4. How would we integrate documentation?
Documentation for a connector should probably end up in the connector
repository. The Flink website should contain one entrance to all connectors
(so not the current approach where we have connectors per DataStream API,
Table API etc). Each connector documentation should end up as one menu item
in connectors, containing all necessary information for all DataStream,
Table, SQL and Python implementations.
5. Which connectors should end up in the external connector repo?
I'll open up a separate thread on this topic to have a parallel discussion
on that. We should reach consensus on both threads before we can move
forward on this topic as a whole.
Best regards,
Martijn
On Fri, 10 Dec 2021 at 04:47, Thomas Weise <t...@apache.org> wrote:
+1 for repo per connector from my side also
Thanks for trying out the different approaches.
Where would the common/infra pieces live? In a separate repository
with its own release?
Thomas
On Thu, Dec 9, 2021 at 12:42 PM Till Rohrmann <trohrm...@apache.org>
wrote:
Sorry if I was a bit unclear. +1 for the single repo per connector
approach.
Cheers,
Till
On Thu, Dec 9, 2021 at 5:41 PM Till Rohrmann <trohrm...@apache.org>
wrote:
+1 for the single repo approach.
Cheers,
Till
On Thu, Dec 9, 2021 at 3:54 PM Martijn Visser <mart...@ververica.com>
wrote:
I also agree that it feels more natural to go with a repo for each
individual connector. Each repository can be made available at
flink-packages.org so users can find them, next to referring to them
in
documentation. +1 from my side.
On Thu, 9 Dec 2021 at 15:38, Arvid Heise <ar...@apache.org> wrote:
Hi all,
We tried out Chesnay's proposal and went with Option 2.
Unfortunately,
we
experienced tough nuts to crack and feel like we hit a dead end:
- The main pain point with the outlined Frankensteinian connector
repo
is
how to handle shared code / infra code. If we have it in some
<common>
branch, then we need to merge the common branch in the connector
branch
on
update. However, it's unclear to me how improvements in the common
branch
that naturally appear while working on a specific connector go back
into
the common branch. You can't use a pull request from your branch or
else
your connector code would poison the connector-less common branch.
So
you
would probably manually copy the files over to a common branch and
create a
PR branch for that.
- A weird solution could be to have the common branch as a
submodule in
the
repo itself (if that's even possible). I'm sure that this setup
would
blow
up the minds of all newcomers.
- Similarly, it's mandatory to have safeguards against code from
connector
A poisoning connector B, common, or main. I had some similar setup
in
the
past and code from two "distinct" branch types constantly swept
over.
- We could also say that we simply release <common> independently
and
just
have a maven (SNAPSHOT) dependency on it. But that would create a
weird
flow if you need to change in common where you need to constantly
switch
branches back and forth.
- In general, Frankensteinian's approach is very switch intensive.
If
you
maintain 3 connectors and need to fix 1 build stability each at the
same
time (quite common nowadays for some reason) and you have 2 review
rounds,
you need to switch branches 9 times ignoring changes to common.
Additionally, we still have the rather user/dev unfriendly main
that is
mostly empty. I'm also not sure we can generate an overview
README.md to
make it more friendly here because in theory every connector branch
should
be based on main and we would get merge conflicts.
I'd like to propose once again to go with individual repositories.
- The only downside that we discussed so far is that we have more
initial
setup to do. Since we organically grow the number of
connector/repositories
that load is quite distributed. We can offer templates after
finding a
good
approach that can even be used by outside organizations.
- Regarding secrets, I think it's actually an advantage that the
Kafka
connector has no access to the AWS secrets. If there are secrets to
be
shared across connectors, we can and should use Azure's Variable
Groups
(I
have used it in the past to share Nexus creds across repos). That
would
also make rotation easy.
- Working on different connectors would be rather easy as all
modern IDE
support multiple repo setups in the same project. You still need to
do
multiple releases in case you update common code (either accessed
through
Nexus or git submodule) and you want to release your connector.
- There is no difference in respect to how many CI runs there in
both
approaches.
- Individual repositories also have the advantage of allowing
external
incubation. Let's assume someone builds connector A and hosts it in
their
organization (very common setup). If they want to contribute the
code to
Flink, we could simply transfer the repository into ASF after
ensuring
Flink coding standards. Then we retain git history and Github
issues.
Is there any point that I'm missing?
On Fri, Nov 26, 2021 at 1:32 PM Chesnay Schepler <
ches...@apache.org>
wrote:
For sharing workflows we should be able to use composite actions.
We'd
have the main definition files in the flink-connectors repo, that
we
also need to tag/release, which other branches/repos can then
import.
These are also versioned, so we don't have to worry about
accidentally
breaking stuff.
These could also be used to enforce certain standards / interfaces
such
that we can automate more things (e.g., integration into the Flink
documentation).
It is true that Option 2) and dedicated repositories share a lot
of
properties. While I did say in an offline conversation that we in
that
case might just as well use separate repositories, I'm not so sure
anymore. One repo would make administration a bit easier, for
example
secrets wouldn't have to be applied to each repo (we wouldn't want
certain secrets to be set up organization-wide).
I overall also like that one repo would present a single access
point;
you can't "miss" a connector repo, and I would hope that having
it as
one repo would nurture more collaboration between the connectors,
which
after all need to solve similar problems.
It is a fair point that the branching model would be quite weird,
but
I
think that would subside pretty quickly.
Personally I'd go with Option 2, and if that doesn't work out we
can
still split the repo later on. (Which should then be a trivial
matter
of
copying all <connector>/* branches and renaming them).
On 26/11/2021 12:47, Till Rohrmann wrote:
Hi Arvid,
Thanks for updating this thread with the latest findings. The
described
limitations for a single connector repo sound suboptimal to me.
* Option 2. sounds as if we try to simulate multi connector
repos
inside
of
a single repo. I also don't know how we would share code
between the
different branches (sharing infrastructure would probably be
easier
though). This seems to have the same limitations as dedicated
repos
with
the downside of having a not very intuitive branching model.
* Isn't option 1. kind of a degenerated version of option 2.
where
we
have
some unrelated code from other connectors in the individual
connector
branches?
* Option 3. has the downside that someone creating a release
has to
release
all connectors. This means that she either has to sync with the
different
connector maintainers or has to be able to release all
connectors on
her
own. We are already seeing in the Flink community that releases
require
quite good communication/coordination between the different
people
working
on different Flink components. Given our goals to make connector
releases
easier and more frequent, I think that coupling different
connector
releases might be counter-productive.
To me it sounds not very practical to mainly use a mono
repository
w/o
having some more advanced build infrastructure that, for
example,
allows
to
have different git roots in different connector directories.
Maybe
the
mono
repo can be a catch all repository for connectors that want to
be
released
in lock-step (Option 3.) with all other connectors the repo
contains.
But
for connectors that get changed frequently, having a dedicated
repository
that allows independent releases sounds preferable to me.
What utilities and infrastructure code do you intend to share?
Using
git
submodules can definitely be one option to share code. However,
it
might
also be ok to depend on flink-connector-common artifacts which
could
make
things easier. Where I am unsure is whether git submodules can
be
used
to
share infrastructure code (e.g. the .github/workflows) because
you
need
these files in the repo to trigger the CI infrastructure.
Cheers,
Till
On Thu, Nov 25, 2021 at 1:59 PM Arvid Heise <ar...@apache.org>
wrote:
Hi Brian,
Thank you for sharing. I think your approach is very valid and
is
in
line
with what I had in mind.
Basically Pravega community aligns the connector releases with
the
Pravega
mainline release
This certainly would mean that there is little value in
coupling
connector
versions. So it's making a good case for having separate
connector
repos.
and maintains the connector with the latest 3 Flink
versions(CI
will
publish snapshots for all these 3 branches)
I'd like to give connector devs a simple way to express to
which
Flink
versions the current branch is compatible. From there we can
generate
the
compatibility matrix automatically and optionally also create
different
releases per supported Flink version. Not sure if the latter is
indeed
better than having just one artifact that happens to run with
multiple
Flink versions. I guess it depends on what dependencies we are
exposing. If
the connector uses flink-connector-base, then we probably need
separate
artifacts with poms anyways.
Best,
Arvid
On Fri, Nov 19, 2021 at 10:55 AM Zhou, Brian <b.z...@dell.com>
wrote:
Hi Arvid,
For branching model, the Pravega Flink connector has some
experience
what
I would like to share. Here[1][2] is the compatibility matrix
and
wiki
explaining the branching model and releases. Basically Pravega
community
aligns the connector releases with the Pravega mainline
release,
and
maintains the connector with the latest 3 Flink versions(CI
will
publish
snapshots for all these 3 branches).
For example, recently we have 0.10.1 release[3], and in maven
central
we
need to upload three artifacts(For Flink 1.13, 1.12, 1.11) for
0.10.1
version[4].
There are some alternatives. Another solution that we once
discussed
but
finally got abandoned is to have a independent version just
like
the
current CDC connector, and then give a big compatibility
matrix to
users.
We think it would be too confusing when the connector
develops. On
the
contrary, we can also do the opposite way to align with Flink
version
and
maintain several branches for different system version.
I would say this is only a fairly-OK solution because it is a
bit
painful
for maintainers as cherry-picks are very common and releases
would
require
much work. However, if neither systems do not have a nice
backward
compatibility, there seems to be no comfortable solution to
the
their
connector.
[1]
https://github.com/pravega/flink-connectors#compatibility-matrix
[2]
https://github.com/pravega/flink-connectors/wiki/Versioning-strategy-for-Flink-connector
[3]
https://github.com/pravega/flink-connectors/releases/tag/v0.10.1
[4]
https://search.maven.org/search?q=pravega-connectors-flink
Best Regards,
Brian
Internal Use - Confidential
-----Original Message-----
From: Arvid Heise <ar...@apache.org>
Sent: Friday, November 19, 2021 4:12 PM
To: dev
Subject: Re: [DISCUSS] Creating an external connector
repository
[EXTERNAL EMAIL]
Hi everyone,
we are currently in the process of setting up the
flink-connectors
repo
[1] for new connectors but we hit a wall that we currently
cannot
take:
branching model.
To reiterate the original motivation of the external connector
repo:
We
want to decouple the release cycle of a connector with Flink.
However,
if
we want to support semantic versioning in the connectors with
the
ability
to introduce breaking changes through major version bumps and
support
bugfixes on old versions, then we need release branches
similar to
how
Flink core operates.
Consider two connectors, let's call them kafka and hbase. We
have
kafka
in
version 1.0.X, 1.1.Y (small improvement), 2.0.Z (config
option)
change
and
hbase only on 1.0.A.
Now our current assumption was that we can work with a
mono-repo
under
ASF
(flink-connectors). Then, for release-branches, we found 3
options:
1. We would need to create some ugly mess with the cross
product
of
connector and version: so you have kafka-release-1.0,
kafka-release-1.1,
kafka-release-2.0, hbase-release-1.0. The main issue is not
the
amount
of
branches (that's something that git can handle) but there the
state
of
kafka is undefined in hbase-release-1.0. That's a call for
desaster
and
makes releasing connectors very cumbersome (CI would only
execute
and
publish hbase SNAPSHOTS on hbase-release-1.0).
2. We could avoid the undefined state by having an empty
master
and
each
release branch really only holds the code of the connector.
But
that's
also
not great: any user that looks at the repo and sees no
connector
would
assume that it's dead.
3. We could have synced releases similar to the CDC connectors
[2].
That
means that if any connector introduces a breaking change, all
connectors
get a new major. I find that quite confusing to a user if
hbase
gets
a
new
release without any change because kafka introduced a breaking
change.
To fully decouple release cycles and CI of connectors, we
could
add
individual repositories under ASF (flink-connector-kafka,
flink-connector-hbase). Then we can apply the same branching
model as
before. I quickly checked if there are precedences in the
apache
community
for that approach and just by scanning alphabetically I found
cordova
with
70 and couchdb with 77 apache repos respectively. So it
certainly
seems
like other projects approached our problem in that way and the
apache
organization is okay with that. I currently expect max 20
additional
repos
for connectors and in the future 10 max each for formats and
filesystems
if
we would also move them out at some point in time. So we
would be
at
a
total of 50 repos.
Note for all options, we need to provide a compability matrix
that we
aim
to autogenerate.
Now for the potential downsides that we internally discussed:
- How can we ensure common infra structure code, utilties, and
quality?
I propose to add a flink-connector-common that contains all
these
things
and is added as a git submodule/subtree to the repos.
- Do we implicitly discourage connector developers to maintain
more
than
one connector with a fragmented code base?
That is certainly a risk. However, I currently also see few
devs
working
on more than one connector. However, it may actually help
keeping
the
devs
that maintain a specific connector on the hook. We could use
github
issues
to track bugs and feature requests and a dev can focus his
limited
time
on
getting that one connector right.
So WDYT? Compared to some intermediate suggestions with split
repos,
the
big difference is that everything remains under Apache
umbrella
and
the
Flink community.
[1]
https://urldefense.com/v3/__https://github.com/apache/flink-connectors__;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-XjpYgXzxxweh4$
[github[.]com] [2]
https://urldefense.com/v3/__https://github.com/ververica/flink-cdc-connectors/__;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-XjpYgXzgoPGA8$
[github[.]com]
On Fri, Nov 12, 2021 at 3:39 PM Arvid Heise <ar...@apache.org
wrote:
Hi everyone,
I created the flink-connectors repo [1] to advance the
topic. We
would
create a proof-of-concept in the next few weeks as a special
branch
that I'd then use for discussions. If the community agrees
with
the
approach, that special branch will become the master. If
not, we
can
reiterate over it or create competing POCs.
If someone wants to try things out in parallel, just make
sure
that
you are not accidentally pushing POCs to the master.
As a reminder: We will not move out any current connector
from
Flink
at this point in time, so everything in Flink will remain as
is
and
be
maintained there.
Best,
Arvid
[1]
https://urldefense.com/v3/__https://github.com/apache/flink-connectors
__;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-XjpYgXzxxweh4
$ [github[.]com]
On Fri, Oct 29, 2021 at 6:57 PM Till Rohrmann <
trohrm...@apache.org
wrote:
Hi everyone,
From the discussion, it seems to me that we have different
opinions
whether to have an ASF umbrella repository or to host them
outside
of
the ASF. It also seems that this is not really the problem
to
solve.
Since there are many good arguments for either approach, we
could
simply start with an ASF umbrella repository and see how
people
adopt
it. If the individual connectors cannot move fast enough or
if
people
prefer to not buy into the more heavy-weight ASF processes,
then
they
can host the code also somewhere else. We simply need to
make
sure
that these connectors are discoverable (e.g. via
flink-packages).
The more important problem seems to be to provide common
tooling
(testing, infrastructure, documentation) that can easily be
reused.
Similarly, it has become clear that the Flink community
needs to
improve on providing stable APIs. I think it is not
realistic to
first complete these tasks before starting to move
connectors to
dedicated repositories. As Stephan said, creating a
connector
repository will force us to pay more attention to API
stability
and
also to think about which testing tools are required.
Hence, I
believe that starting to add connectors to a different
repository
than apache/flink will help improve our connector tooling
(declaring
testing classes as public, creating a common test utility
repo,
creating a repo
template) and vice versa. Hence, I like Arvid's proposed
process as
it will start kicking things off w/o letting this effort
fizzle
out.
Cheers,
Till
On Thu, Oct 28, 2021 at 11:44 AM Stephan Ewen <
se...@apache.org
wrote:
Thank you all, for the nice discussion!
From my point of view, I very much like the idea of
putting
connectors
in a
separate repository. But I would argue it should be part of
Apache
Flink,
similar to flink-statefun, flink-ml, etc.
I share many of the reasons for that:
- As argued many times, reduces complexity of the Flink
repo,
increases
response times of CI, etc.
- Much lower barrier of contribution, because an
unstable
connector
would
not de-stabilize the whole build. Of course, we would need
to
make
sure
we
set this up the right way, with connectors having
individual CI
runs,
build
status, etc. But it certainly seems possible.
I would argue some points a bit different than some cases
made
before:
(a) I believe the separation would increase connector
stability.
Because it
really forces us to work with the connectors against the
APIs
like
any external developer. A mono repo is somehow the wrong
thing
if
you in practice want to actually guarantee stable internal
APIs at
some layer.
Because the mono repo makes it easy to just change
something on
both
sides
of the API (provider and consumer) seamlessly.
Major refactorings in Flink need to keep all connector API
contracts intact, or we need to have a new version of the
connector
API.
(b) We may even be able to go towards more lightweight and
automated releases over time, even if we stay in Apache
Flink
with
that repo.
This isn't yet fully aligned with the Apache release
policies,
yet,
but there are board discussions about whether there can be
bot-triggered releases (by dependabot) and how that could
fit
into
the Apache process.
This doesn't seem to be quite there just yet, but seeing
that
those
start
is a good sign, and there is a good chance we can do some
things
there.
I am not sure whether we should let bots trigger releases,
because
a
final
human look at things isn't a bad thing, especially given
the
popularity
of
software supply chain attacks recently.
I do share Chesnay's concerns about complexity in tooling,
though.
Both release tooling and test tooling. They are not
incompatible
with that approach, but they are a task we need to tackle
during
this change which will add additional work.
On Tue, Oct 26, 2021 at 10:31 AM Arvid Heise <
ar...@apache.org
wrote:
Hi folks,
I think some questions came up and I'd like to address the
question of
the
timing.
Could you clarify what release cadence you're thinking of?
There's
quite
a big range that fits "more frequent than Flink"
(per-commit,
daily, weekly, bi-weekly, monthly, even bi-monthly).
The short answer is: as often as needed:
- If there is a CVE in a dependency and we need to bump
it -
release immediately.
- If there is a new feature merged, release soonish. We
may
collect a
few
successive features before a release.
- If there is a bugfix, release immediately or soonish
depending
on
the
severity and if there are workarounds available.
We should not limit ourselves; the whole idea of
independent
releases
is
exactly that you release as needed. There is no release
planning
or anything needed, you just go with a release as if it
was an
external artifact.
(1) is the connector API already stable?
From another discussion thread [1], connector API is far
from
stable.
Currently, it's hard to build connectors against multiple
Flink
versions.
There are breaking API changes both in 1.12 -> 1.13 and
1.13
->
1.14
and
maybe also in the future versions, because Table
related
APIs
are
still
@PublicEvolving and new Sink API is still @Experimental.
The question is: what is stable in an evolving system? We
recently discovered that the old SourceFunction needed to
be
refined such that cancellation works correctly [1]. So
that
interface is in Flink since
7
years, heavily used also outside, and we still had to
change
the
contract
in a way that I'd expect any implementer to recheck their
implementation.
It might not be necessary to change anything and you can
probably
change
the the code for all Flink versions but still, the
interface
was
not
stable
in the closest sense.
If we focus just on API changes on the unified interfaces,
then
we
expect
one more change to Sink API to support compaction. For
Table
API,
there
will most likely also be some changes in 1.15. So we could
wait
for
1.15.
But I'm questioning if that's really necessary because we
will
add
more
functionality beyond 1.15 without breaking API. For
example,
we
may
add
more unified connector metrics. If you want to use it in
your
connector,
you have to support multiple Flink versions anyhow. So
rather
then
focusing
the discussion on "when is stuff stable", I'd rather
focus on
"how
can we
support building connectors against multiple Flink
versions"
and
make
it
as
painless as possible.
Chesnay pointed out to use different branches for
different
Flink
versions
which sounds like a good suggestion. With a mono-repo, we
can't
use branches differently anyways (there is no way to have
release
branches
per
connector without chaos). In these branches, we could
provide
shims to simulate future features in older Flink versions
such
that code-wise,
the
source code of a specific connector may not diverge
(much).
For
example,
to
register unified connector metrics, we could simulate the
current
approach
also in some utility package of the mono-repo.
I see the stable core Flink API as a prerequisite for
modularity.
And
for connectors it is not just the source and sink API
(source
being stable as of 1.14), but everything that is
required to
build and maintain a connector downstream, such as the
test
utilities and infrastructure.
That is a very fair point. I'm actually surprised to see
that
MiniClusterWithClientResource is not public. I see it
being
used
in
all
connectors, especially outside of Flink. I fear that as
long
as
we do
not
have connectors outside, we will not properly annotate and
maintain
these
utilties in a classic hen-and-egg-problem. I will outline
an
idea
at
the
end.
the connectors need to be adopted and require at least
one
release
per
Flink minor release.
However, this will make the releases of connectors
slower,
e.g.
maintain
features for multiple branches and release multiple
branches.
I think the main purpose of having an external connector
repository
is
in
order to have "faster releases of connectors"?
Imagine a project with a complex set of dependencies.
Let's
say
Flink
version A plus Flink reliant dependencies released by
other
projects (Flink-external connectors, Beam, Iceberg, Hudi,
..).
We don't want
a
situation where we bump the core Flink version to B and
things
fall apart (interface changes, utilities that were
useful but
not public, transitive dependencies etc.).
Yes, that's why I wanted to automate the processes more
which
is
not
that
easy under ASF. Maybe we automate the source provision
across
supported
versions and have 1 vote thread for all versions of a
connector?
From the perspective of CDC connector maintainers, the
biggest
advantage
of
maintaining it outside of the Flink project is that:
1) we can have a more flexible and faster release cycle
2) we can be more liberal with committership for
connector
maintainers
which can also attract more committers to help the
release.
Personally, I think maintaining one connector repository
under
the
ASF
may
not have the above benefits.
Yes, I also feel that ASF is too restrictive for our
needs.
But
it
feels
like there are too many that see it differently and I
think we
need
(2) Flink testability without connectors.
This is a very good question. How can we guarantee the
new
Source
and
Sink
API are stable with only test implementation?
We can't and shouldn't. Since the connector repo is
managed by
Flink,
a
Flink release manager needs to check if the Flink
connectors
are
actually
working prior to creating an RC. That's similar to how
flink-shaded
and
flink core are related.
So here is one idea that I had to get things rolling. We
are
going to address the external repo iteratively without
compromising what we
already
have:
1.Phase, add new contributions to external repo. We use
that
time
to
setup
infra accordingly and optimize release processes. We will
identify
test
utilities that are not yet public/stable and fix that.
2.Phase, add ports to the new unified interfaces of
existing
connectors.
That requires a previous Flink release to make utilities
stable.
Keep
old
interfaces in flink-core.
3.Phase, remove old interfaces in flink-core of some
connectors
(tbd
at a
later point).
4.Phase, optionally move all remaining connectors (tbd at
a
later
point).
I'd envision having ~3 months between the starting the
different
phases.
WDYT?
[1]
https://urldefense.com/v3/__https://issues.apache.org/jira/browse
/FLINK-23527__;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgd
ke_-XjpYgX2sIvAP4$ [issues[.]apache[.]org]
On Thu, Oct 21, 2021 at 7:12 AM Kyle Bendickson <
k...@tabular.io
wrote:
Hi all,
My name is Kyle and I’m an open source developer
primarily
focused
on
Apache Iceberg.
I’m happy to help clarify or elaborate on any aspect of
our
experience
working on a relatively decoupled connector that is
downstream
and
pretty
popular.
I’d also love to be able to contribute or assist in any
way I
can.
I don’t mean to thread jack, but are there any meetings
or
community
sync
ups, specifically around the connector APIs, that I might
join
/ be
invited
to?
I did want to add that even though I’ve experienced some
of
the
pain
points
of integrating with an evolving system / API (catalog
support
is
generally
speaking pretty new everywhere really in this space), I
also
agree personally that you shouldn’t slow down development
velocity too
much
for
the sake of external connector. Getting to a performant
and
stable
place
should be the primary goal, and slowing that down to
support
stragglers
will (in my personal opinion) always be a losing game.
Some
folks
will
simply stay behind on versions regardless until they
have to
upgrade.
I am working on ensuring that the Iceberg community stays
within 1-2 versions of Flink, so that we can help provide
more
feedback or
contribute
things that might make our ability to support multiple
Flink
runtimes /
versions with one project / codebase and minimal to no
reflection
(our
desired goal).
If there’s anything I can do or any way I can be of
assistance,
please
don’t hesitate to reach out. Or find me on ASF slack 😀
I greatly appreciate your general concern for the needs
of
downstream
connector integrators!
Cheers
Kyle Bendickson (GitHub: kbendick) Open Source Developer
kyle
[at] tabular [dot] io
On Wed, Oct 20, 2021 at 11:35 AM Thomas Weise <
t...@apache.org>
wrote:
Hi,
I see the stable core Flink API as a prerequisite for
modularity.
And
for connectors it is not just the source and sink API
(source
being
stable as of 1.14), but everything that is required to
build
and maintain a connector downstream, such as the test
utilities and infrastructure.
Without the stable surface of core Flink, changes will
leak
into downstream dependencies and force lock step
updates.
Refactoring across N repos is more painful than a single
repo. Those with experience developing downstream of
Flink
will know the pain, and
that
isn't limited to connectors. I don't remember a Flink
"minor
version"
update that was just a dependency version change and
did not
force other downstream changes.
Imagine a project with a complex set of dependencies.
Let's
say
Flink
version A plus Flink reliant dependencies released by
other
projects
(Flink-external connectors, Beam, Iceberg, Hudi, ..). We
don't
want a
situation where we bump the core Flink version to B and
things
fall
apart (interface changes, utilities that were useful
but not
public,
transitive dependencies etc.).
The discussion here also highlights the benefits of
keeping
certain
connectors outside Flink. Whether that is due to
difference
in developer community, maturity of the connectors,
their
specialized/limited usage etc. I would like to see that
as a
sign
of
a
growing ecosystem and most of the ideas that Arvid has
put
forward would benefit further growth of the connector
ecosystem.
As for keeping connectors within Apache Flink: I prefer
that
as
the
path forward for "essential" connectors like FileSource,
KafkaSource,
... And we can still achieve a more flexible and faster
release
cycle.
Thanks,
Thomas
On Wed, Oct 20, 2021 at 3:32 AM Jark Wu <
imj...@gmail.com>
wrote:
Hi Konstantin,
the connectors need to be adopted and require at least
one
release
per
Flink minor release.
However, this will make the releases of connectors
slower,
e.g.
maintain
features for multiple branches and release multiple
branches.
I think the main purpose of having an external
connector
repository
is
in
order to have "faster releases of connectors"?
From the perspective of CDC connector maintainers, the
biggest
advantage
of
maintaining it outside of the Flink project is that:
1) we can have a more flexible and faster release cycle
2) we can be more liberal with committership for
connector
maintainers
which can also attract more committers to help the
release.
Personally, I think maintaining one connector
repository
under
the
ASF
may
not have the above benefits.
Best,
Jark
On Wed, 20 Oct 2021 at 15:14, Konstantin Knauf <
kna...@apache.org>
wrote:
Hi everyone,
regarding the stability of the APIs. I think everyone
agrees
that
connector APIs which are stable across minor versions
(1.13->1.14)
are
the
mid-term goal. But:
a) These APIs are still quite young, and we shouldn't
make
them
@Public
prematurely either.
b) Isn't this *mostly* orthogonal to where the
connector
code
lives?
Yes,
as long as there are breaking changes, the connectors
need to
be
adopted
and require at least one release per Flink minor
release.
Documentation-wise this can be addressed via a
compatibility
matrix
for
each connector as Arvid suggested. IMO we shouldn't
block
this
effort
on
the stability of the APIs.
Cheers,
Konstantin
On Wed, Oct 20, 2021 at 8:56 AM Jark Wu
<imj...@gmail.com>
wrote:
Hi,
I think Thomas raised very good questions and would
like
to
know
your
opinions if we want to move connectors out of flink
in
this
version.
(1) is the connector API already stable?
Separate releases would only make sense if the core
Flink
surface
is
fairly stable though. As evident from Iceberg (and
also
Beam),
that's
not the case currently. We should probably focus on
addressing
the
stability first, before splitting code. A success
criteria
could
be
that we are able to build Iceberg and Beam against
multiple
Flink
versions w/o the need to change code. The goal would
be
that
no
connector breaks when we make changes to Flink core.
Until
that's
the
case, code separation creates a setup where 1+1 or
N+1
repositories
need to move lock step.
From another discussion thread [1], connector API
is far
from
stable.
Currently, it's hard to build connectors against
multiple
Flink
versions.
There are breaking API changes both in 1.12 -> 1.13
and
1.13
->
1.14
and
maybe also in the future versions, because Table
related
APIs
are
still
@PublicEvolving and new Sink API is still
@Experimental.
(2) Flink testability without connectors.
Flink w/o Kafka connector (and few others) isn't
viable. Testability of Flink was already brought up,
can we
really
certify a Flink core release without Kafka
connector?
Maybe
those
connectors that are used in Flink e2e tests to
validate
functionality
of core Flink should not be broken out?
This is a very good question. How can we guarantee
the
new
Source
and
Sink
API are stable with only test implementation?
Best,
Jark
On Tue, 19 Oct 2021 at 23:56, Chesnay Schepler <
ches...@apache.org>
wrote:
Could you clarify what release cadence you're
thinking
of?
There's
quite
a big range that fits "more frequent than Flink"
(per-commit,
daily,
weekly, bi-weekly, monthly, even bi-monthly).
On 19/10/2021 14:15, Martijn Visser wrote:
Hi all,
I think it would be a huge benefit if we can
achieve
more
frequent
releases
of connectors, which are not bound to the release
cycle
of
Flink
itself.
I
agree that in order to get there, we need to have
stable
interfaces
which
are trustworthy and reliable, so they can be safely
used
by
those
connectors. I do think that work still needs to be
done
on
those
interfaces, but I am confident that we can get
there
from a
Flink
perspective.
I am worried that we would not be able to achieve
those
frequent
releases
of connectors if we are putting these connectors
under
the
Apache
umbrella,
because that means that for each connector release
we
have
to
follow
the
Apache release creation process. This requires a
lot
of
manual
steps
and
prohibits automation and I think it would be hard
to
scale
out
frequent
releases of connectors. I'm curious how others
think
this
challenge
could
be solved.
Best regards,
Martijn
On Mon, 18 Oct 2021 at 22:22, Thomas Weise <
t...@apache.org>
wrote:
Thanks for initiating this discussion.
There are definitely a few things that are not
optimal
with
our
current management of connectors. I would not
necessarily
characterize
it as a "mess" though. As the points raised so far
show, it
isn't
easy
to find a solution that balances competing
requirements
and
leads to
a
net improvement.
It would be great if we can find a setup that
allows for
connectors
to
be released independently of core Flink and that
each
connector
can
be
released separately. Flink already has separate
releases (flink-shaded), so that by itself isn't a
new thing.
Per-connector
releases would need to allow for more frequent
releases
(without
the
baggage that a full Flink release comes with).
Separate releases would only make sense if the core
Flink
surface is
fairly stable though. As evident from Iceberg (and
also
Beam),
that's
not the case currently. We should probably focus
on
addressing
the
stability first, before splitting code. A success
criteria
could
be
that we are able to build Iceberg and Beam against
multiple
Flink
versions w/o the need to change code. The goal
would be
that
no
connector breaks when we make changes to Flink
core.
Until
that's the
case, code separation creates a setup where 1+1 or
N+1
repositories
need to move lock step.
Regarding some connectors being more important for
Flink
than
others:
That's a fact. Flink w/o Kafka connector (and few
others)
isn't
viable. Testability of Flink was already brought
up,
can we
really
certify a Flink core release without Kafka
connector?
Maybe
those
connectors that are used in Flink e2e tests to
validate
functionality
of core Flink should not be broken out?
Finally, I think that the connectors that move
into
separate
repos
should remain part of the Apache Flink project.
Larger
organizations
tend to approve the use of and contribution to
open
source
at
the
project level. Sometimes it is everything ASF.
More
often
it
is
"Apache Foo". It would be fatal to end up with a
patchwork
of
projects
with potentially different licenses and governance
to
arrive
at a
working Flink setup. This may mean we prioritize
usability
over
developer convenience, if that's in the best
interest of
Flink
as a
whole.
Thanks,
Thomas
On Mon, Oct 18, 2021 at 6:59 AM Chesnay Schepler <
ches...@apache.org
wrote:
Generally, the issues are reproducibility and
control.
Stuffs completely broken on the Flink side for a
week?
Well
then so
are
the connector repos.
(As-is) You can't go back to a previous version
of
the
snapshot.
Which
also means that checking out older commits can be
problematic
because
you'd still work against the latest snapshots,
and
they
not
be
compatible with each other.
On 18/10/2021 15:22, Arvid Heise wrote:
I was actually betting on snapshots versions.
What are
the
limits?
Obviously, we can only do a release of a 1.15
connector
after
1.15
is
release.
--
Konstantin Knauf
https://urldefense.com/v3/__https://twitter.com/snntrable
__;!!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-
XjpYgX5MUy9M4$ [twitter[.]com]
https://urldefense.com/v3/__https://github.com/knaufk__;!
!LpKI!2a1uSGfMmwc8HNwqBUIGtFPzLHP5m9yS0sC3n3IpLgdke_-XjpY
gXyX8u50S$ [github[.]com]
