Hi, Since PubSub Lite is deprecated, I think we can safely close this FLIP and not proceed further [1]
Best regards, Martijn [1] https://cloud.google.com/pubsub/docs/choosing-pubsub-or-lite On Sun, Jun 9, 2024 at 12:23 AM Ahmed Hamdy <hamdy10...@gmail.com> wrote: > Hi all, > I wonder if we can revive the discussion here. > It has been discussed in another thread[1] that the lack of maintenance for > connector repos as in gcp-pubsub should be a motive to promote more > committers who actually commit to maintaining the repos instead of > migrating/adding connectors out of support of the community, and I don't > see another blocker for such FLIP to progress given the effort done by > Daniel on the external repo. > I do have comments on the FLIP itself, and am happy to drive changes > myself If Daniel is no longer available but this doesn't seem the core > point of the discussion at the moment. > > I would love to hear your thoughts. > > 1-https://lists.apache.org/thread/63wdvo5hvr6koc3wzbqy2kw4krhmkfbx > Best Regards > Ahmed Hamdy > > > On Fri, 17 Mar 2023 at 15:58, Daniel Collins <dpcoll...@google.com.invalid > > > wrote: > > > > would the repository ... be removed ... ? > > > > Yes, I would remove it once it is merged into a version of flink that is > > supported by GCP dataproc. It exists now (and I am creating releases and > > maven artifacts for it) to unblock users in the interim period. > > > > -Daniel > > > > On Thu, Mar 16, 2023 at 3:32 PM Martijn Visser <martijnvis...@apache.org > > > > wrote: > > > > > Hi Daniel, > > > > > > > I don't know how to get to this point, it sounds like more of an > > > organizational constraint than a technical one though- who is > responsible > > > for the same role for the standard Pub/Sub connector? I'm working with > > the > > > Pub/Sub team right now on prioritization of supporting the flink > > connector > > > and converting it to support the recommended delivery mechanism for > that > > > service. > > > > > > It's not so much an organizational constraint, it's more if there are > one > > > or more committers in the Flink community who have the bandwidth to > help > > > with reviewing and merging a new connector. The PubSub connector has > > pretty > > > much been unmaintained for the past couple of years; I have done a > couple > > > of outreaches to Google, but those were unfruitful. > > > > > > I'm hoping that someone from the Flink committers has the bandwidth for > > > helping you out. @All, if you have bandwidth, please come forward. > > > > > > > I imagine our involvement would be similar to support for our > > > self-managed client libraries > > > > > > I think that sounds fine. > > > > > > One question I have is if you envision that the code of > > > https://github.com/googleapis/java-pubsublite-flink moves to > > > https://github.com/apache/flink-connector-gcp-pubsub, would the > > repository > > > https://github.com/googleapis/java-pubsublite-flink be removed or do > you > > > propose to have both of them exist? I would be in favour of having one, > > but > > > wanted to check with you. > > > > > > Best regards, > > > > > > Martijn > > > > > > On Tue, Mar 14, 2023 at 4:09 AM Daniel Collins > > > <dpcoll...@google.com.invalid> > > > wrote: > > > > > > > Hi all, > > > > > > > > Thank you for the feedback. Responses inline. > > > > > > > > > we need feedback from a Committer who would review and help > maintain > > it > > > > going forward. Ideally, this Committer would guide one or more > > > contributors > > > > from Google to Committership so that Google could step up and > maintain > > > > Flink > > > > 's PubSub and PubSub Lite Connector in the future. > > > > > > > > I don't know how to get to this point, it sounds like more of an > > > > organizational constraint than a technical one though- who is > > responsible > > > > for the same role for the standard Pub/Sub connector? I'm working > with > > > the > > > > Pub/Sub team right now on prioritization of supporting the flink > > > connector > > > > and converting it to support the recommended delivery mechanism for > > that > > > > service. > > > > > > > > > For this, it would be good to understand how you envision the > > > involvement > > > > of the PubSub Lite team at Google. > > > > > > > > I imagine our involvement would be similar to support for our > > > self-managed > > > > client libraries, we would field feature requests and dependency > update > > > > requests as they come in, as well as debugging any bug reports, and > > > > improving the library as new service features arrive. We would want > to > > > work > > > > with the flink community to support new features that our users find > > > > useful. Our team did not contribute the Pub/Sub connector, but as > more > > of > > > > our customers come to use flink, we would like to bring it up to par > > both > > > > in performance and supportability of the rest of our supported > clients. > > > > > > > > > I think that giving the big architectural components and decisions > > > would > > > > help the discussion/vote. > > > > > > > > I will add discussion of these components to the proposal. > > > > > > > > > you need to replace the Google headers by the ASF v2 ones > > > > > > > > This is moot unless the proposal to merge is accepted. > > > > > > > > > when possible, try to use the equivalent Flink / JDK / ASF libs > > instead > > > > of the Google ones > > > > > look at the first commits of the Apache Beam project > > > > > > > > I've been an active contributor in the past to the apache beam > > project- I > > > > don't think anything that I've used here would prevent merging this > to > > > the > > > > beam project, and much of it is needed to interact with the client > > > library. > > > > > > > > -Daniel > > > > > > > > On Mon, Mar 13, 2023 at 5:26 AM Etienne Chauchot < > echauc...@apache.org > > > > > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > I agree with Konstantin, mentoring is important especially on this > > new > > > > > connector framework. Long time maintenance is even more important. > > > > > > > > > > I could not mentor you on this topic because I'm not a committer on > > the > > > > > Flink project and because I don't know Pub/Sub tech. That being > said > > I > > > > > have on blog under writing to share what I learnt while authoring > the > > > > > Cassandra connector with the new source framework. I think it could > > be > > > > > useful as a first learning step and to avoid some caveats. > > > > > > > > > > Regarding the FLIP, as that you already developed the connector > > inside > > > > > Google, I understand why you gave the whole code inside the FLIP > > (there > > > > > is no better doc than code) but I think that giving the big > > > > > architectural components and decisions would help the > > discussion/vote. > > > > > > > > > > Also, I did not review the code but just took a quick look at the > > > Google > > > > > techs coupling: > > > > > > > > > > - you need to replace the Google headers by the ASF v2 ones > > > > > > > > > > - when possible, try to use the equivalent Flink / JDK / ASF libs > > > > > instead of the Google ones (futures, collections, safe guard > > > > > annotations, autovalue etc...) > > > > > > > > > > Finally, as a hint, I think you could take a look at the first > > commits > > > > > of the Apache Beam project when DataFlow SDK was donated to the ASF > > and > > > > > see what was done there to make the code ASF friendly. > > > > > > > > > > Best > > > > > > > > > > Etienne > > > > > > > > > > Le 09/03/2023 à 09:45, Konstantin Knauf a écrit : > > > > > > Hi Daniel, > > > > > > > > > > > > I think, it would be great to have a PubSub Lite Connector in > > Flink. > > > > > Before > > > > > > you put this proposal up for a vote, though, we need feedback > from > > a > > > > > > Committer who would review and help maintain it going forward. > > > Ideally, > > > > > > this Committer would guide one or more contributors from Google > to > > > > > > Committership so that Google could step up and maintain Flink's > > > PubSub > > > > > and > > > > > > PubSub Lite Connector in the future. For this, it would be good > to > > > > > > understand how you envision the involvement of the PubSub Lite > team > > > at > > > > > > Google. > > > > > > > > > > > > I am specifically sensitive on this topic, because the PubSub > > > connector > > > > > has > > > > > > lacked attention and maintenance for a long time. There was also > a > > > very > > > > > > short-lived interested by Google in the past to contribute a > Google > > > > > PubSub > > > > > > Connector [1]. > > > > > > > > > > > > Best, > > > > > > > > > > > > Konstantin > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-22380 > > > > > > > > > > > > Am Mi., 8. März 2023 um 14:45 Uhr schrieb Etienne Chauchot < > > > > > > echauc...@apache.org>: > > > > > > > > > > > >> Hi, > > > > > >> > > > > > >> I agree with Ryan, even if clients might be totally different > the > > > > > >> backend technologies are the same so hosting them in the same > repo > > > > makes > > > > > >> sense. Similar thinking made us put all the Cassandra related > > > > connectors > > > > > >> in the same cassandra repo. > > > > > >> > > > > > >> Etienne > > > > > >> > > > > > >> Le 02/03/2023 à 14:43, Daniel Collins a écrit : > > > > > >>> Hello Ryan, > > > > > >>> > > > > > >>> Unfortunately there's not much shared logic between the two- > the > > > > > clients > > > > > >>> have to look fundamentally different since the Pub/Sub Lite > > client > > > > > >> exposes > > > > > >>> partitions to the split level for repeatable reads. > > > > > >>> > > > > > >>> I have no objection to this living in the same repo as the > > Pub/Sub > > > > > >>> connector, if this is an easier way forward than setting up a > new > > > > repo, > > > > > >>> sounds good to me. The Pub/Sub team is organizationally close > to > > > us, > > > > > and > > > > > >> is > > > > > >>> looking into providing more support for the flink connector in > > the > > > > near > > > > > >>> future. > > > > > >>> > > > > > >>> -Daniel > > > > > >>> > > > > > >>> On Thu, Mar 2, 2023 at 3:26 AM Ryan Skraba > > > > > <ryan.skr...@aiven.io.invalid > > > > > >>> > > > > > >>> wrote: > > > > > >>> > > > > > >>>> Hello Daniel! Quite a while ago, I started porting the > Pub/Sub > > > > > >> connector > > > > > >>>> (from an existing PR) to the new source API in the new > > > > > >>>> flink-connector-gcp-pubsub repository [PR2]. As Martijn > > > mentioned, > > > > > >> there > > > > > >>>> hasn't been a lot of attention on this connector; any > community > > > > > >> involvement > > > > > >>>> would be appreciated! > > > > > >>>> > > > > > >>>> Instead of considering this a new connector, is there an > > > opportunity > > > > > >> here > > > > > >>>> to offer the two variants (Pub/Sub and Pub/Sub Lite) as > > different > > > > > >> artifacts > > > > > >>>> in that same repo? Is there much common logic that can be > > shared > > > > > >> between > > > > > >>>> the two? I'm not as familiar as I should be with Lite, but I > do > > > > > recall > > > > > >>>> that they share many concepts and _some_ dependencies. > > > > > >>>> > > > > > >>>> All my best, Ryan > > > > > >>>> > > > > > >>>> > > > > > >>>> On Wed, Mar 1, 2023 at 11:21 PM Daniel Collins > > > > > >>>> <dpcoll...@google.com.invalid> > > > > > >>>> wrote: > > > > > >>>> > > > > > >>>>> Hello all, > > > > > >>>>> > > > > > >>>>> I'd like to start an official discuss thread for adding a > > Pub/Sub > > > > > Lite > > > > > >>>>> Connector to Flink. We've had requests from our users to add > > > flink > > > > > >>>> support, > > > > > >>>>> and are willing to maintain and support this connector long > > term > > > > from > > > > > >> the > > > > > >>>>> product team. > > > > > >>>>> > > > > > >>>>> The proposal is https://cwiki.apache.org/confluence/x/P51bDg > , > > > what > > > > > >> would > > > > > >>>>> be > > > > > >>>>> people's thoughts on adding this connector? > > > > > >>>>> > > > > > >>>>> -Daniel > > > > > >>>>> > > > > > > > > > > > > > > > > > > > > >
