Hi Daniel, > I don't know how to get to this point, it sounds like more of an organizational constraint than a technical one though- who is responsible for the same role for the standard Pub/Sub connector? I'm working with the Pub/Sub team right now on prioritization of supporting the flink connector and converting it to support the recommended delivery mechanism for that service.
It's not so much an organizational constraint, it's more if there are one or more committers in the Flink community who have the bandwidth to help with reviewing and merging a new connector. The PubSub connector has pretty much been unmaintained for the past couple of years; I have done a couple of outreaches to Google, but those were unfruitful. I'm hoping that someone from the Flink committers has the bandwidth for helping you out. @All, if you have bandwidth, please come forward. > I imagine our involvement would be similar to support for our self-managed client libraries I think that sounds fine. One question I have is if you envision that the code of https://github.com/googleapis/java-pubsublite-flink moves to https://github.com/apache/flink-connector-gcp-pubsub, would the repository https://github.com/googleapis/java-pubsublite-flink be removed or do you propose to have both of them exist? I would be in favour of having one, but wanted to check with you. Best regards, Martijn On Tue, Mar 14, 2023 at 4:09 AM Daniel Collins <dpcoll...@google.com.invalid> wrote: > Hi all, > > Thank you for the feedback. Responses inline. > > > we need feedback from a Committer who would review and help maintain it > going forward. Ideally, this Committer would guide one or more contributors > from Google to Committership so that Google could step up and maintain > Flink > 's PubSub and PubSub Lite Connector in the future. > > I don't know how to get to this point, it sounds like more of an > organizational constraint than a technical one though- who is responsible > for the same role for the standard Pub/Sub connector? I'm working with the > Pub/Sub team right now on prioritization of supporting the flink connector > and converting it to support the recommended delivery mechanism for that > service. > > > For this, it would be good to understand how you envision the involvement > of the PubSub Lite team at Google. > > I imagine our involvement would be similar to support for our self-managed > client libraries, we would field feature requests and dependency update > requests as they come in, as well as debugging any bug reports, and > improving the library as new service features arrive. We would want to work > with the flink community to support new features that our users find > useful. Our team did not contribute the Pub/Sub connector, but as more of > our customers come to use flink, we would like to bring it up to par both > in performance and supportability of the rest of our supported clients. > > > I think that giving the big architectural components and decisions would > help the discussion/vote. > > I will add discussion of these components to the proposal. > > > you need to replace the Google headers by the ASF v2 ones > > This is moot unless the proposal to merge is accepted. > > > when possible, try to use the equivalent Flink / JDK / ASF libs instead > of the Google ones > > look at the first commits of the Apache Beam project > > I've been an active contributor in the past to the apache beam project- I > don't think anything that I've used here would prevent merging this to the > beam project, and much of it is needed to interact with the client library. > > -Daniel > > On Mon, Mar 13, 2023 at 5:26 AM Etienne Chauchot <echauc...@apache.org> > wrote: > > > Hi all, > > > > I agree with Konstantin, mentoring is important especially on this new > > connector framework. Long time maintenance is even more important. > > > > I could not mentor you on this topic because I'm not a committer on the > > Flink project and because I don't know Pub/Sub tech. That being said I > > have on blog under writing to share what I learnt while authoring the > > Cassandra connector with the new source framework. I think it could be > > useful as a first learning step and to avoid some caveats. > > > > Regarding the FLIP, as that you already developed the connector inside > > Google, I understand why you gave the whole code inside the FLIP (there > > is no better doc than code) but I think that giving the big > > architectural components and decisions would help the discussion/vote. > > > > Also, I did not review the code but just took a quick look at the Google > > techs coupling: > > > > - you need to replace the Google headers by the ASF v2 ones > > > > - when possible, try to use the equivalent Flink / JDK / ASF libs > > instead of the Google ones (futures, collections, safe guard > > annotations, autovalue etc...) > > > > Finally, as a hint, I think you could take a look at the first commits > > of the Apache Beam project when DataFlow SDK was donated to the ASF and > > see what was done there to make the code ASF friendly. > > > > Best > > > > Etienne > > > > Le 09/03/2023 à 09:45, Konstantin Knauf a écrit : > > > Hi Daniel, > > > > > > I think, it would be great to have a PubSub Lite Connector in Flink. > > Before > > > you put this proposal up for a vote, though, we need feedback from a > > > Committer who would review and help maintain it going forward. Ideally, > > > this Committer would guide one or more contributors from Google to > > > Committership so that Google could step up and maintain Flink's PubSub > > and > > > PubSub Lite Connector in the future. For this, it would be good to > > > understand how you envision the involvement of the PubSub Lite team at > > > Google. > > > > > > I am specifically sensitive on this topic, because the PubSub connector > > has > > > lacked attention and maintenance for a long time. There was also a very > > > short-lived interested by Google in the past to contribute a Google > > PubSub > > > Connector [1]. > > > > > > Best, > > > > > > Konstantin > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-22380 > > > > > > Am Mi., 8. März 2023 um 14:45 Uhr schrieb Etienne Chauchot < > > > echauc...@apache.org>: > > > > > >> Hi, > > >> > > >> I agree with Ryan, even if clients might be totally different the > > >> backend technologies are the same so hosting them in the same repo > makes > > >> sense. Similar thinking made us put all the Cassandra related > connectors > > >> in the same cassandra repo. > > >> > > >> Etienne > > >> > > >> Le 02/03/2023 à 14:43, Daniel Collins a écrit : > > >>> Hello Ryan, > > >>> > > >>> Unfortunately there's not much shared logic between the two- the > > clients > > >>> have to look fundamentally different since the Pub/Sub Lite client > > >> exposes > > >>> partitions to the split level for repeatable reads. > > >>> > > >>> I have no objection to this living in the same repo as the Pub/Sub > > >>> connector, if this is an easier way forward than setting up a new > repo, > > >>> sounds good to me. The Pub/Sub team is organizationally close to us, > > and > > >> is > > >>> looking into providing more support for the flink connector in the > near > > >>> future. > > >>> > > >>> -Daniel > > >>> > > >>> On Thu, Mar 2, 2023 at 3:26 AM Ryan Skraba > > <ryan.skr...@aiven.io.invalid > > >>> > > >>> wrote: > > >>> > > >>>> Hello Daniel! Quite a while ago, I started porting the Pub/Sub > > >> connector > > >>>> (from an existing PR) to the new source API in the new > > >>>> flink-connector-gcp-pubsub repository [PR2]. As Martijn mentioned, > > >> there > > >>>> hasn't been a lot of attention on this connector; any community > > >> involvement > > >>>> would be appreciated! > > >>>> > > >>>> Instead of considering this a new connector, is there an opportunity > > >> here > > >>>> to offer the two variants (Pub/Sub and Pub/Sub Lite) as different > > >> artifacts > > >>>> in that same repo? Is there much common logic that can be shared > > >> between > > >>>> the two? I'm not as familiar as I should be with Lite, but I do > > recall > > >>>> that they share many concepts and _some_ dependencies. > > >>>> > > >>>> All my best, Ryan > > >>>> > > >>>> > > >>>> On Wed, Mar 1, 2023 at 11:21 PM Daniel Collins > > >>>> <dpcoll...@google.com.invalid> > > >>>> wrote: > > >>>> > > >>>>> Hello all, > > >>>>> > > >>>>> I'd like to start an official discuss thread for adding a Pub/Sub > > Lite > > >>>>> Connector to Flink. We've had requests from our users to add flink > > >>>> support, > > >>>>> and are willing to maintain and support this connector long term > from > > >> the > > >>>>> product team. > > >>>>> > > >>>>> The proposal is https://cwiki.apache.org/confluence/x/P51bDg, what > > >> would > > >>>>> be > > >>>>> people's thoughts on adding this connector? > > >>>>> > > >>>>> -Daniel > > >>>>> > > > > > >
