> until we *require* Asgard on a core transform, it shouldn't be in the main repo
I don't think this is necessarily true if it solves end user use cases. If there is a specific transform that solves a specific use case, we could include it in the transforms folder for end-users, even if it isn't utilized in the I/Os at present. Hence the suggestion to take the most promising transforms and propose adding them with documentation, apis and rationale. -Daniel On Fri, Sep 8, 2023 at 11:20 AM Robert Burke <rob...@frantil.com> wrote: > I would say until we *require* Asgard on a core transform, it shouldn't be > in the main repo. > > Incorporating something before there's a need for it is premature > abstraction. We can't do things because they *might* be useful. Let's see > concrete places where they are useful, or we're already having a similar > need solved a different way. > > Beam is complicated by itself, and we do encourage multiple ways of > solving problems, but that says to me that having an out of repo ecosystem > is the right path, rather than incorporation. > > On Fri, Sep 8, 2023, 8:14 AM Daniel Collins via dev <dev@beam.apache.org> > wrote: > >> I think there are a lot of interesting and relatively isolated components >> of the project, it might make sense to write per-transform one pagers for >> isolated things like the most useful pieces (just basically copying the >> documentation and justifying the API) instead of doing a one-shot import or >> having it live forever in an external project. >> >> -Daniel >> >> On Fri, Sep 8, 2023 at 11:10 AM Kenneth Knowles <k...@apache.org> wrote: >> >>> I agree with everyone about "not everything has to be in the Beam repo". >>> I really like the idea of having a clearer "ecosystem" section of the >>> website, which is sort of started at >>> https://beam.apache.org/community/integrations/ but that is not very >>> prominent. >>> >>> Agree with John though. The transforms in Asgarde could potentially be >>> used in Beam. Potentially best accomplished by just adding them as >>> transforms to the core Java SDK? >>> >>> Kenn >>> >>> On Wed, Sep 6, 2023 at 1:46 PM John Casey via dev <dev@beam.apache.org> >>> wrote: >>> >>>> Agreed on documentation and on keeping it in a separate repo. >>>> >>>> We have a few pretty significant beam extensions (scio and Dataflow >>>> Templates also come to mind) that Beam should highlight, but are separate >>>> repos for their own governance, contributions, and release reasons. >>>> >>>> The difference with Asgarde is that we might want to use it in Beam >>>> itself, which makes it more reasonable to include in the main repo. >>>> >>>> On Tue, Sep 5, 2023 at 8:36 PM Robert Bradshaw via dev < >>>> dev@beam.apache.org> wrote: >>>> >>>>> I think this is a great library. I'm on the fence of whether it makes >>>>> sense to include with Beam proper vs. be a library that builds on top of >>>>> Beam. (Would there be benefits of tighter integration? There is the >>>>> maintenance/loss of governance issue.) I am definitely not on the side >>>>> that >>>>> the entire Beam ecosystem needs to be distributed/maintained by Beam >>>>> itself. >>>>> >>>>> Regardless of the direction we go, I think it could make a lot of >>>>> sense to put pointers to it in our documentation. >>>>> >>>>> >>>>> On Tue, Sep 5, 2023 at 7:21 AM Danny McCormick via dev < >>>>> dev@beam.apache.org> wrote: >>>>> >>>>>> I think my only concerns here are around the toil we'll be taking on, >>>>>> and will we be leaving the asgarde project in a better or worse place. >>>>>> >>>>>> From a release standpoint, we would need to release it with the same >>>>>> cadence as Beam. Adding asgarde into our standard release process seems >>>>>> fairly straightforward, though, so I'm not too worried about it - looks >>>>>> like it's basically (1) add a commit like this >>>>>> <https://github.com/tosun-si/asgarde/commit/432de527d67dc71f06507328319b466b6d0fb56a>, >>>>>> (2) run this workflow >>>>>> <https://github.com/tosun-si/asgarde/blob/main/.github/workflows/publish-project.yml>, >>>>>> and (3) tag/mark the release as released on GitHub. >>>>>> >>>>>> In terms of bug fixes and improvements, though, I'm a little worried >>>>>> that we might be leaving things in a worse state since Mazlum has been >>>>>> the >>>>>> only contributor thus far, and he would lose some governance (and >>>>>> possibly >>>>>> the ability to commit code on his own). An extra motivated community >>>>>> member >>>>>> or two could change the math a bit, but I'm not sure if there are >>>>>> actually >>>>>> clear advantages to including it in Apache other than visibility. Would >>>>>> adding links to our docs calling Asgarde out as an option accomplish the >>>>>> same purpose? >>>>>> >>>>>> > Let's be careful about whether these tests are included in our >>>>>> presubmits. Contrib code with flaky tests has been a major pain point in >>>>>> the past. >>>>>> >>>>>> +1 - I think if we do this I'd vote that it be in a separate repo ( >>>>>> github.com/apache/beam-asgarde made sense to me). >>>>>> >>>>>> --------------------------------------- >>>>>> >>>>>> Overall, I'm probably a slight -1 to adding this to the Apache >>>>>> workspace, but +1 to at least adding links from the Beam docs to Asgarde. >>>>>> >>>>>> Thanks, >>>>>> Danny >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Sep 5, 2023 at 12:03 AM Reuven Lax via dev < >>>>>> dev@beam.apache.org> wrote: >>>>>> >>>>>>> Let's be careful about whether these tests are included in our >>>>>>> presubmits. Contrib code with flaky tests has been a major pain point in >>>>>>> the past. >>>>>>> >>>>>>> On Sat, Sep 2, 2023 at 12:02 PM Austin Bennett <aus...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>>> Wanting us to not miss this. @Mazlum TOSUN <mazlum.to...@gmail.com> is >>>>>>>> happy to donate Asgarde to our project. >>>>>>>> >>>>>>>> It looks like he'd need a SGA and CCLA [ 1 ] on file; anything >>>>>>>> else? >>>>>>>> >>>>>>>> I recalled the donation of Euphoria [ 2 ] , so I looked at those >>>>>>>> threads [ 3 ] for insights into the process. It didn't look like >>>>>>>> there >>>>>>>> was a needed VOTE, so mostly a matter of ensuring necessary >>>>>>>> signatures, and >>>>>>>> ideally some sort of consensus [ or non-opposition ] to the donation. >>>>>>>> >>>>>>>> >>>>>>>> [ 1 ] https://www.apache.org/licenses/contributor-agreements.html >>>>>>>> [ 2 ] https://beam.apache.org/documentation/sdks/java/euphoria/ >>>>>>>> [ 3 ] >>>>>>>> https://lists.apache.org/thread/xzlx4rm2tvc36mmwvhyvtdvsw7bnjscp >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jun 15, 2023 at 7:05 AM Kerry Donny-Clark via dev < >>>>>>>> dev@beam.apache.org> wrote: >>>>>>>> >>>>>>>>> This looks like an excellent contribution. I can easily understand >>>>>>>>> the motivation, and I think Beam would benefit from a higher level >>>>>>>>> abstraction for error handling. >>>>>>>>> Kerry >>>>>>>>> >>>>>>>>> On Wed, Jun 14, 2023, 6:31 PM Austin Bennett <aus...@apache.org> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Beam Devs, >>>>>>>>>> >>>>>>>>>> @Mazlum <https://www.linkedin.com/in/mazlum-tosun-900b1812/> was >>>>>>>>>> suggested to consider donating Asgarde >>>>>>>>>> <https://github.com/tosun-si/asgarde> to Beam for Java/Kotlin >>>>>>>>>> error handling to Beam [ see: >>>>>>>>>> https://2022.beamsummit.org/sessions/error-handling-asgarde/ for >>>>>>>>>> last year's Beam Summit talk ], he is also the author of Pasgard >>>>>>>>>> <https://github.com/tosun-si/pasgarde>e [ for Python ] and >>>>>>>>>> Milgard [ for a simplified Kotlin API ]. >>>>>>>>>> >>>>>>>>>> Would Asgarde be a good contribution, something the Beam >>>>>>>>>> community would be willing to accept? I imagine we might want it to >>>>>>>>>> live >>>>>>>>>> at github.com/apache/beam-asgarde ? Or perhaps there is a good >>>>>>>>>> place in github.com/apache/beam ?? >>>>>>>>>> >>>>>>>>>> Especially once/if officially part of Beam, I imagine we'd add >>>>>>>>>> follow-up items like getting onto the website/docs, and related. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Austin >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> P.S. This might warrant separate/additional conversations for >>>>>>>>>> his other libraries, but let's focus any discussion on Asgarde for >>>>>>>>>> now? >>>>>>>>>> >>>>>>>>>
