Hi all,
I agree with Konstantin, mentoring is important especially on this new
connector framework. Long time maintenance is even more important.
I could not mentor you on this topic because I'm not a committer on the
Flink project and because I don't know Pub/Sub tech. That being said I
have on blog under writing to share what I learnt while authoring the
Cassandra connector with the new source framework. I think it could be
useful as a first learning step and to avoid some caveats.
Regarding the FLIP, as that you already developed the connector inside
Google, I understand why you gave the whole code inside the FLIP (there
is no better doc than code) but I think that giving the big
architectural components and decisions would help the discussion/vote.
Also, I did not review the code but just took a quick look at the Google
techs coupling:
- you need to replace the Google headers by the ASF v2 ones
- when possible, try to use the equivalent Flink / JDK / ASF libs
instead of the Google ones (futures, collections, safe guard
annotations, autovalue etc...)
Finally, as a hint, I think you could take a look at the first commits
of the Apache Beam project when DataFlow SDK was donated to the ASF and
see what was done there to make the code ASF friendly.
Best
Etienne
Le 09/03/2023 à 09:45, Konstantin Knauf a écrit :
Hi Daniel,
I think, it would be great to have a PubSub Lite Connector in Flink. Before
you put this proposal up for a vote, though, we need feedback from a
Committer who would review and help maintain it going forward. Ideally,
this Committer would guide one or more contributors from Google to
Committership so that Google could step up and maintain Flink's PubSub and
PubSub Lite Connector in the future. For this, it would be good to
understand how you envision the involvement of the PubSub Lite team at
Google.
I am specifically sensitive on this topic, because the PubSub connector has
lacked attention and maintenance for a long time. There was also a very
short-lived interested by Google in the past to contribute a Google PubSub
Connector [1].
Best,
Konstantin
[1] https://issues.apache.org/jira/browse/FLINK-22380
Am Mi., 8. März 2023 um 14:45 Uhr schrieb Etienne Chauchot <
echauc...@apache.org>:
Hi,
I agree with Ryan, even if clients might be totally different the
backend technologies are the same so hosting them in the same repo makes
sense. Similar thinking made us put all the Cassandra related connectors
in the same cassandra repo.
Etienne
Le 02/03/2023 à 14:43, Daniel Collins a écrit :
Hello Ryan,
Unfortunately there's not much shared logic between the two- the clients
have to look fundamentally different since the Pub/Sub Lite client
exposes
partitions to the split level for repeatable reads.
I have no objection to this living in the same repo as the Pub/Sub
connector, if this is an easier way forward than setting up a new repo,
sounds good to me. The Pub/Sub team is organizationally close to us, and
is
looking into providing more support for the flink connector in the near
future.
-Daniel
On Thu, Mar 2, 2023 at 3:26 AM Ryan Skraba <ryan.skr...@aiven.io.invalid
wrote:
Hello Daniel! Quite a while ago, I started porting the Pub/Sub
connector
(from an existing PR) to the new source API in the new
flink-connector-gcp-pubsub repository [PR2]. As Martijn mentioned,
there
hasn't been a lot of attention on this connector; any community
involvement
would be appreciated!
Instead of considering this a new connector, is there an opportunity
here
to offer the two variants (Pub/Sub and Pub/Sub Lite) as different
artifacts
in that same repo? Is there much common logic that can be shared
between
the two? I'm not as familiar as I should be with Lite, but I do recall
that they share many concepts and _some_ dependencies.
All my best, Ryan
On Wed, Mar 1, 2023 at 11:21 PM Daniel Collins
<dpcoll...@google.com.invalid>
wrote:
Hello all,
I'd like to start an official discuss thread for adding a Pub/Sub Lite
Connector to Flink. We've had requests from our users to add flink
support,
and are willing to maintain and support this connector long term from
the
product team.
The proposal is https://cwiki.apache.org/confluence/x/P51bDg, what
would
be
people's thoughts on adding this connector?
-Daniel
