Thanks Mazlum, this sounds great. I think there are two ways we can proceed if we decide to integrate the Asgarde library into Beam.
(1) Directly import the code into Beam without significant modifications and/or a review (though we may add tests). (2) Go through a design/code review to determine whether this is the best approach for implementing error handling / DLQ in Beam transforms or whether there are other alternatives/modifications to Asgarde we want to consider. If we do (1) I prefer adding Asgarde as a separate Gradle module in Beam. We can later integrate it into the core module after a design/code review. Thank, Cham On Tue, Sep 12, 2023 at 10:26 AM Mazlum TOSUN <mazlum.to...@gmail.com> wrote: > Hello Austin and everyone, > > I am open for discussion. > > My first intention with Asgarde was to help the Beam community, because > Dead Letter Queue is so important in Beam and all the data pipeline > frameworks. > When I worked with Beam on production with my customers, we needed to > catch errors with side outputs and dead letter queue. > > This library really helped us to keep a less verbose code while applying > all the error handling logic, that is error prone and verbose if it is > repeated. > > As Kennet said, my intention was to stay as close as possible to Beam, > with a Wrapper and a Failure Monad on top of a PCollection, to handle all > the code and complexity for try catch blocks and side output. > > For the governance, even if I am the creator of this library, the most > important isn't me but the community and to help the community. > If the best solution to help the community is including the library > directly on Beam, we can go in this direction, with of course your reviews > and recommendations. > > Then the library will belong to the community and we will continue to > improve it. > > For the decision about the best place, I will comply with the majority. > > Best regards, > > Mazlum > > On Mon, Sep 11, 2023 at 11:15 PM Austin Bennett <aus...@apache.org> wrote: > >> @Mazlum TOSUN <mazlum.to...@gmail.com> -- you and I have spoken a few >> times about this. it'd be good for you to comment here on list, on any of >> your concerns with governance, and/or other thoughts. Ex: if you think >> contributing asgarde directly is the thing [ or perhaps expressing any >> interest helping write/contribute the relevant functionality into beam ... >> it is possible that by adding the actual functionality into beam - like >> Kenn's mentioned 'other place' we could make asgarde as an separate add-on >> obsolete ]. >> >> >> >> On Fri, Sep 8, 2023 at 8:55 AM Kenneth Knowles <k...@apache.org> wrote: >> >>> For anyone who hasn't clicked over the Asgarde, my TL;DR description of >>> it is that it adds the "failure monad" aka "andThen" style error/result >>> handling on top of chaining of PCollections. So it is at a similar level of >>> abstraction of our basic transforms and generally useful for chaining >>> dead-letter side outputs. It is no more or less appropriate for the core >>> SDK than, say, the Project/Filter/Join transforms, or Watch, etc. If we >>> actually aspired to have a thin core with the accessories like that in >>> another place, then it should go to that other place. >>> >>> Kenn >>> >>> On Fri, Sep 8, 2023 at 11:24 AM Daniel Collins via dev < >>> dev@beam.apache.org> wrote: >>> >>>> > until we *require* Asgard on a core transform, it shouldn't be in the >>>> main repo >>>> >>>> I don't think this is necessarily true if it solves end user use cases. >>>> If there is a specific transform that solves a specific use case, we could >>>> include it in the transforms folder for end-users, even if it isn't >>>> utilized in the I/Os at present. Hence the suggestion to take the most >>>> promising transforms and propose adding them with documentation, apis and >>>> rationale. >>>> >>>> -Daniel >>>> >>>> On Fri, Sep 8, 2023 at 11:20 AM Robert Burke <rob...@frantil.com> >>>> wrote: >>>> >>>>> I would say until we *require* Asgard on a core transform, it >>>>> shouldn't be in the main repo. >>>>> >>>>> Incorporating something before there's a need for it is premature >>>>> abstraction. We can't do things because they *might* be useful. Let's see >>>>> concrete places where they are useful, or we're already having a similar >>>>> need solved a different way. >>>>> >>>>> Beam is complicated by itself, and we do encourage multiple ways of >>>>> solving problems, but that says to me that having an out of repo ecosystem >>>>> is the right path, rather than incorporation. >>>>> >>>>> On Fri, Sep 8, 2023, 8:14 AM Daniel Collins via dev < >>>>> dev@beam.apache.org> wrote: >>>>> >>>>>> I think there are a lot of interesting and relatively isolated >>>>>> components of the project, it might make sense to write per-transform one >>>>>> pagers for isolated things like the most useful pieces (just basically >>>>>> copying the documentation and justifying the API) instead of doing a >>>>>> one-shot import or having it live forever in an external project. >>>>>> >>>>>> -Daniel >>>>>> >>>>>> On Fri, Sep 8, 2023 at 11:10 AM Kenneth Knowles <k...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> I agree with everyone about "not everything has to be in the Beam >>>>>>> repo". I really like the idea of having a clearer "ecosystem" section of >>>>>>> the website, which is sort of started at >>>>>>> https://beam.apache.org/community/integrations/ but that is not >>>>>>> very prominent. >>>>>>> >>>>>>> Agree with John though. The transforms in Asgarde could potentially >>>>>>> be used in Beam. Potentially best accomplished by just adding them as >>>>>>> transforms to the core Java SDK? >>>>>>> >>>>>>> Kenn >>>>>>> >>>>>>> On Wed, Sep 6, 2023 at 1:46 PM John Casey via dev < >>>>>>> dev@beam.apache.org> wrote: >>>>>>> >>>>>>>> Agreed on documentation and on keeping it in a separate repo. >>>>>>>> >>>>>>>> We have a few pretty significant beam extensions (scio and Dataflow >>>>>>>> Templates also come to mind) that Beam should highlight, but are >>>>>>>> separate >>>>>>>> repos for their own governance, contributions, and release reasons. >>>>>>>> >>>>>>>> The difference with Asgarde is that we might want to use it in Beam >>>>>>>> itself, which makes it more reasonable to include in the main repo. >>>>>>>> >>>>>>>> On Tue, Sep 5, 2023 at 8:36 PM Robert Bradshaw via dev < >>>>>>>> dev@beam.apache.org> wrote: >>>>>>>> >>>>>>>>> I think this is a great library. I'm on the fence of whether it >>>>>>>>> makes sense to include with Beam proper vs. be a library that builds >>>>>>>>> on top >>>>>>>>> of Beam. (Would there be benefits of tighter integration? There is the >>>>>>>>> maintenance/loss of governance issue.) I am definitely not on the >>>>>>>>> side that >>>>>>>>> the entire Beam ecosystem needs to be distributed/maintained by Beam >>>>>>>>> itself. >>>>>>>>> >>>>>>>>> Regardless of the direction we go, I think it could make a lot of >>>>>>>>> sense to put pointers to it in our documentation. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Sep 5, 2023 at 7:21 AM Danny McCormick via dev < >>>>>>>>> dev@beam.apache.org> wrote: >>>>>>>>> >>>>>>>>>> I think my only concerns here are around the toil we'll be taking >>>>>>>>>> on, and will we be leaving the asgarde project in a better or worse >>>>>>>>>> place. >>>>>>>>>> >>>>>>>>>> From a release standpoint, we would need to release it with the >>>>>>>>>> same cadence as Beam. Adding asgarde into our standard release >>>>>>>>>> process >>>>>>>>>> seems fairly straightforward, though, so I'm not too worried about >>>>>>>>>> it - >>>>>>>>>> looks like it's basically (1) add a commit like this >>>>>>>>>> <https://github.com/tosun-si/asgarde/commit/432de527d67dc71f06507328319b466b6d0fb56a>, >>>>>>>>>> (2) run this workflow >>>>>>>>>> <https://github.com/tosun-si/asgarde/blob/main/.github/workflows/publish-project.yml>, >>>>>>>>>> and (3) tag/mark the release as released on GitHub. >>>>>>>>>> >>>>>>>>>> In terms of bug fixes and improvements, though, I'm a little >>>>>>>>>> worried that we might be leaving things in a worse state since >>>>>>>>>> Mazlum has >>>>>>>>>> been the only contributor thus far, and he would lose some >>>>>>>>>> governance (and >>>>>>>>>> possibly the ability to commit code on his own). An extra motivated >>>>>>>>>> community member or two could change the math a bit, but I'm not >>>>>>>>>> sure if >>>>>>>>>> there are actually clear advantages to including it in Apache other >>>>>>>>>> than >>>>>>>>>> visibility. Would adding links to our docs calling Asgarde out as an >>>>>>>>>> option >>>>>>>>>> accomplish the same purpose? >>>>>>>>>> >>>>>>>>>> > Let's be careful about whether these tests are included in our >>>>>>>>>> presubmits. Contrib code with flaky tests has been a major pain >>>>>>>>>> point in >>>>>>>>>> the past. >>>>>>>>>> >>>>>>>>>> +1 - I think if we do this I'd vote that it be in a separate repo >>>>>>>>>> (github.com/apache/beam-asgarde made sense to me). >>>>>>>>>> >>>>>>>>>> --------------------------------------- >>>>>>>>>> >>>>>>>>>> Overall, I'm probably a slight -1 to adding this to the Apache >>>>>>>>>> workspace, but +1 to at least adding links from the Beam docs to >>>>>>>>>> Asgarde. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Danny >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Sep 5, 2023 at 12:03 AM Reuven Lax via dev < >>>>>>>>>> dev@beam.apache.org> wrote: >>>>>>>>>> >>>>>>>>>>> Let's be careful about whether these tests are included in our >>>>>>>>>>> presubmits. Contrib code with flaky tests has been a major pain >>>>>>>>>>> point in >>>>>>>>>>> the past. >>>>>>>>>>> >>>>>>>>>>> On Sat, Sep 2, 2023 at 12:02 PM Austin Bennett < >>>>>>>>>>> aus...@apache.org> wrote: >>>>>>>>>>> >>>>>>>>>>>> Wanting us to not miss this. @Mazlum TOSUN >>>>>>>>>>>> <mazlum.to...@gmail.com> is happy to donate Asgarde to >>>>>>>>>>>> our project. >>>>>>>>>>>> >>>>>>>>>>>> It looks like he'd need a SGA and CCLA [ 1 ] on file; anything >>>>>>>>>>>> else? >>>>>>>>>>>> >>>>>>>>>>>> I recalled the donation of Euphoria [ 2 ] , so I looked at >>>>>>>>>>>> those threads [ 3 ] for insights into the process. It didn't >>>>>>>>>>>> look like >>>>>>>>>>>> there was a needed VOTE, so mostly a matter of ensuring necessary >>>>>>>>>>>> signatures, and ideally some sort of consensus [ or non-opposition >>>>>>>>>>>> ] to the >>>>>>>>>>>> donation. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [ 1 ] >>>>>>>>>>>> https://www.apache.org/licenses/contributor-agreements.html >>>>>>>>>>>> [ 2 ] https://beam.apache.org/documentation/sdks/java/euphoria/ >>>>>>>>>>>> [ 3 ] >>>>>>>>>>>> https://lists.apache.org/thread/xzlx4rm2tvc36mmwvhyvtdvsw7bnjscp >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Jun 15, 2023 at 7:05 AM Kerry Donny-Clark via dev < >>>>>>>>>>>> dev@beam.apache.org> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> This looks like an excellent contribution. I can easily >>>>>>>>>>>>> understand the motivation, and I think Beam would benefit from a >>>>>>>>>>>>> higher >>>>>>>>>>>>> level abstraction for error handling. >>>>>>>>>>>>> Kerry >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Jun 14, 2023, 6:31 PM Austin Bennett < >>>>>>>>>>>>> aus...@apache.org> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Beam Devs, >>>>>>>>>>>>>> >>>>>>>>>>>>>> @Mazlum <https://www.linkedin.com/in/mazlum-tosun-900b1812/> >>>>>>>>>>>>>> was suggested to consider donating Asgarde >>>>>>>>>>>>>> <https://github.com/tosun-si/asgarde> to Beam for >>>>>>>>>>>>>> Java/Kotlin error handling to Beam [ see: >>>>>>>>>>>>>> https://2022.beamsummit.org/sessions/error-handling-asgarde/ >>>>>>>>>>>>>> for last year's Beam Summit talk ], he is also the author of >>>>>>>>>>>>>> Pasgard <https://github.com/tosun-si/pasgarde>e [ for Python >>>>>>>>>>>>>> ] and Milgard [ for a simplified Kotlin API ]. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Would Asgarde be a good contribution, something the Beam >>>>>>>>>>>>>> community would be willing to accept? I imagine we might want >>>>>>>>>>>>>> it to live >>>>>>>>>>>>>> at github.com/apache/beam-asgarde ? Or perhaps there is a >>>>>>>>>>>>>> good place in github.com/apache/beam ?? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Especially once/if officially part of Beam, I imagine we'd >>>>>>>>>>>>>> add follow-up items like getting onto the website/docs, and >>>>>>>>>>>>>> related. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>> Austin >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> P.S. This might warrant separate/additional conversations >>>>>>>>>>>>>> for his other libraries, but let's focus any discussion on >>>>>>>>>>>>>> Asgarde for >>>>>>>>>>>>>> now? >>>>>>>>>>>>>> >>>>>>>>>>>>>
