Thanks for driving this discussion Arvid! I think this will be one giant leap for Flink community. Externalizing connectors would give connector developers more freedom in developing, releasing and maintaining, which can attract more developers for contributing their connectors and expand the Flink ecosystems.
Considering the position for hosting connectors, I prefer to use an individual organization outside Apache umbrella. If we keep all connectors under Apache, I think there’s not quite difference comparing keeping them in the Flink main repo. Connector developers still require permissions from Flink committers to contribute, and release process should follow Apache rules, which are against our initial motivations of externalizing connectors. Using an individual Github organization will maximum the freedom provided to developers. An ideal structure in my mind would be like "github.com/flink-connectors/flink-connector-xxx". The new established flink-extended org might be another choice, but considering the amount of connectors, I prefer to use an individual org for connectors to avoid flushing other repos under flink-extended. In the meantime, we need to provide a well-established standard / guideline for contributing connectors, including CI, testing, docs (maybe we can’t provide resources for running them, but we should give enough guide on how to setup one) to keep the high quality of connectors. I’m happy to help building these fundamental bricks. Also since Kafka connector is widely used among Flink users, we can make Kafka connector a “model” of how to build and contribute a well-qualified connector into Flink ecosystem, and we can still use this trusted one for Flink E2E tests. Again I believe this will definitely boost the expansion of Flink ecosystem. Very excited to see the progress! Best, Qingsheng Ren On Oct 15, 2021, 8:47 PM +0800, Arvid Heise <ar...@apache.org>, wrote: > Dear community, > Today I would like to kickstart a series of discussions around creating an > external connector repository. The main idea is to decouple the release cycle > of Flink with the release cycles of the connectors. This is a common approach > in other big data analytics projects and seems to scale better than the > current approach. In particular, it will yield the following changes. > • Faster releases of connectors: New features can be added more quickly, > bugs can be fixed immediately, and we can have faster security patches in > case of direct or indirect (through dependencies) security flaws. • New > features can be added to old Flink versions: If the connector API didn’t > change, the same connector jar may be used with different Flink versions. > Thus, new features can also immediately be used with older Flink versions. A > compatibility matrix on each connector page will help users to find suitable > connector versions for their Flink versions. • More activity and > contributions around connectors: If we ease the contribution and development > process around connectors, we will see faster development and also more > connectors. Since that heavily depends on the chosen approach discussed > below, more details will be shown there. • An overhaul of the connector page: > In the future, all known connectors will be shown on the same page in a > similar layout independent of where they reside. They could be hosted on > external project pages (e.g., Iceberg and Hudi), on some company page, or may > stay within the main Flink reposi tory. Connectors may receive some sort > of quality seal such that users can quickly access the production-readiness > and we could also add which community/company promises which kind of support. > • If we take out (some) connectors out of Flink, Flink CI will be faster and > Flink devs will experience less build stabilities (which mostly come from > connectors). That would also speed up Flink development. > Now I’d first like to collect your viewpoints on the ideal state. Let’s first > recap which approaches, we currently have: > • We have half of the connectors in the main Flink repository. Relatively > few of them have received updates in the past couple of months. • Another > large chunk of connectors are in Apache Bahir. It recently has seen the first > release in 3 years. • There are a few other (Apache) projects that maintain a > Flink connector, such as Apache Iceberg, Apache Hudi, and Pravega. • A few > connectors are listed on company-related repositories, such as Apache Pulsar > on StreamNative and CDC connectors on Ververica. > My personal observation is that having a repository per connector seems to > increase the activity on a connector as it’s easier to maintain. For example, > in Apache Bahir all connectors are built against the same Flink version, > which may not be desirable when certain APIs change; for example, > SinkFunction will be eventually deprecated and removed but new Sink interface > may gain more features. > Now, I'd like to outline different approaches. All approaches will allow you > to host your connector on any kind of personal, project, or company > repository. We still want to provide a default place where users can > contribute their connectors and hopefully grow a community around it. The > approaches are: > 1. Create a mono-repo under the Apache umbrella where all connectors will > reside, for example, github.com/apache/flink-connectors. That repository > needs to follow its rules: No GitHub issues, no Dependabot or similar tools, > and a strict manual release process. It would be under the Flink community, > such that Flink committers can write to that repository but no-one else. 2. > Create a GitHub organization with small repositories, for example > github.com/flink-connectors. Since it’s not under the Apache umbrella, we are > free to use whatever process we deem best (up to a future discussion). Each > repository can have a shared list of maintainers + connector specific > committers. We can provide more automation. We may even allow different > licenses to incorporate things like a connector to Oracle that cannot be > released under ASL. 3. ??? <- please provide your additional approaches > In both cases, we will provide opinionated module/repository templates based > on a connector testing framework and guidelines. Depending on the approach, > we may need to enforce certain things. > I’d like to first focus on what the community would ideally seek and minimize > the discussions around legal issues, which we would discuss later. For now, > I’d also like to postpone the discussion if we move all or only a subset of > connectors from Flink to the new default place as it seems to be orthogonal > to the fundamental discussion. > PS: If the external repository for connectors is successful, I’d also like to > move out other things like formats, filesystems, and metric reporters in the > far future. So I’m actually aiming for github.com/(apache/)flink-packages. > But again this discussion is orthogonal to the basic one. > PPS: Depending on the chosen approach, there may be synergies with the > recently approved flink-extended organization.
