"The bits we build on top of external components becomes the Apache project that is ALv2 and is distributed through source archives with convenience binaries (in this case, the image layers and YAML files)."
I like this statement from Matt and completely agree with it. On Sun, Sep 6, 2020, 09:40 Kaxil Naik <kaxiln...@gmail.com> wrote: > I will be happy to help too, definitely and important topic where we need > a clear guidance on it (what sources and binaries are allowed etc) > > Regards, > Kaxil > Airflow PMC > > > > On Sun, Sep 6, 2020, 09:37 Jarek Potiuk <ja...@potiuk.com> wrote: > >> Hey Dave, >> >> Thanks for this. Having the ComDev/Infra repository where PMC can publish >> approved "Chart" and clear docs on what it means to follow the Release >> Policy is exactly what we need I think. >> >> We have a heated debate within Airflow PMC and despite exchanging many >> messages and (sometimes heated) arguments we remain deeply divided on >> that. >> For me, that's a clear sign that we need the ASF help to resolve it. >> >> How can I help to make it happen? >> >> I am happy to devote quite a lot of my time - both infrastructure and >> policy discussions/writing as I think this is a very important subject to >> at least some commercial users of ASF software (and ASF being >> 'commercially >> friendly'). >> >> J. >> >> >> >> On Sun, Sep 6, 2020 at 1:19 AM Kaxil Naik <kaxiln...@gmail.com> wrote: >> >> > Great, thanks Dave >> > >> > On Sun, Sep 6, 2020, 00:17 Dave Fisher <wave4d...@comcast.net> wrote: >> > >> > > Every release must be made in SVN on dist.apache.org. All other >> channels >> > > are optional and may or may not have Infra support. >> > > >> > > I view this as a request to have ComDev or Infra manage a >> Gitbox/Github >> > > repository where any PMC can publish a PMC approved Helm Chart release >> > from >> > > their project. >> > > >> > > How to make a Helm Chart compliant with Apache Release Policy would >> part >> > > of the documentation. >> > > >> > > Sent from my iPhone >> > > >> > > > On Sep 5, 2020, at 3:30 PM, Kaxil Naik <kaxiln...@gmail.com> wrote: >> > > > >> > > > >> > > >> >> > > >> The point is that we MUST give the users possibility to (re)build >> > those >> > > > images on their own if they want. We >> > > > >> > > > >> > > > From >> http://www.apache.org/legal/release-policy.html#source-packages, >> > it >> > > > does not say it has to be on PyPI. I will reiterate what I said in >> the >> > > > Github issue, having a binary on PyPI is no guarantee that it will >> stay >> > > > there. Having a binary on Github releases >> > > > https://github.com/jbub/pgbouncer_exporter/tags also does not >> > guarantee >> > > it >> > > > would not change. If the license are correct and the source is >> > available >> > > I >> > > > definitely do not see this as a problem: >> > > > >> > > > Every ASF release MUST contain one or more source packages, which >> MUST >> > be >> > > >> sufficient for a user to build and test the release provided they >> have >> > > >> access to the appropriate platform and tools. >> > > > >> > > > >> > > > >> > > >> On Sat, Sep 5, 2020 at 10:33 PM Jarek Potiuk <ja...@potiuk.com> >> > wrote: >> > > >> >> > > >> @Matt Sicker <boa...@gmail.com> -> agree on both, central chart >> repo, >> > > and >> > > >> per-project sources for charts. Though packaging is a bit different >> > than >> > > >> Docker though. The packages contain tar.gz sources of the chart >> (which >> > > >> usually includes quite complex go templating logic, not just simple >> > yaml >> > > >> files) and those templates include names of the images used. This >> > makes >> > > >> the Helm chart much more close to the -src.tar.gz files we release >> now >> > > than >> > > >> to Docker Images we publish. So IMHO the .tgz part should follow >> the >> > > normal >> > > >> release process with voting @Kaxilnaik <kaxiln...@apache.org>. >> > > >> Unless those are the very same sources of product that are already >> > > formally >> > > >> released. But then @Gaetan Semet previously pointed out as bad >> > practice >> > > >> to bind the two together. >> > > >> IMHO that means that we have to vote on each Chart release. But I'd >> > love >> > > >> what others think - knowing this packaging details. >> > > >> >> > > >> @ermanndimb- my point for images pointed to by the helm chart is >> not >> > > about >> > > >> copying full sources (I think from this discussion I realised where >> > the >> > > >> main confusion was) . This is not needed. I just made a PR to show >> how >> > > this >> > > >> can be done with the pgbouncer-exporter. This is quite a good >> example >> > > to >> > > >> base our discussion on: >> https://github.com/apache/airflow/pull/10759 >> > > >> >> > > >> The point is that we MUST give the users possibility to (re)build >> > those >> > > >> images on their own if they want. They should be able to do that >> > using: >> > > >> >> > > >> a) official base images (DockerHub has the "official image >> program" >> > for >> > > >> platforms. openjdk, python,debian, alpine - those are all official >> > > images >> > > >> in DockerHub terms >> > > >> b) having access to properly licensed source code with some >> > guarantees >> > > >> that it will not disappear ("official" repo, multiple forks in >> Github, >> > > >> heavily used by others, fork under "apache" organisation) . >> > > >> c) instructions where to find the sources and build the images >> > > >> (Dockerfile + necessary scripts to build the image if in case it is >> > not >> > > >> straightforward). Similarly as we provide instructions on building >> the >> > > >> "product". >> > > >> >> > > >> If the first two are available but it's not obvious how the image >> was >> > > built >> > > >> - IMHO we need to provide the user instructions on how to build >> those >> > > >> images (ideally a script that does it automatically). >> > > >> >> > > >> This is all fulfilled by the PR above as an example: >> > > >> a) based on official, alpine image >> > > >> b) the pgbouncer code is properly licenced and we forked the >> sources >> > > >> several times to be sure it won't disappear >> > > >> c) we have a script that builds and publishes the image (image is >> > > published >> > > >> under apache/airflow DockerHub) >> > > >> >> > > >> IMHO - that's a very good example to show the kind of approach we >> can >> > > take, >> > > >> and shows that it isn't difficult to handle it well at all. >> > > >> >> > > >> The alternative that I am afraid of is - that we require our users >> to >> > > use >> > > >> binaries which are not "official" but maintained and controlled by >> > > >> 3rd-party (without a straightforward way of building the binary >> from >> > > >> official images + properly licensed sources + instructions). If we >> do >> > > not >> > > >> give those to the user but we simply point to a 3rd-party binary >> > bimage >> > > - >> > > >> we both endorse the 3rd party, and introduce dependency on the >> > > 3rd-party. >> > > >> The user then has to rely on the 3rd-party to use the Airflow Helm >> > > Chart - >> > > >> which I think is a very bad idea. >> > > >> >> > > >> J. >> > > >> >> > > >> >> > > >>> On Sat, Sep 5, 2020 at 7:42 PM Kaxil Naik <kaxiln...@gmail.com> >> > wrote: >> > > >>> >> > > >>> Does voting needs to happen for "releasing" the Helm chart ? Do we >> > any >> > > >>> guidelines from the ASF around releasing Helm Chart & Dockerfile? >> > > >>> >> > > >>>> On Sat, Sep 5, 2020, 17:36 Matt Sicker <boa...@gmail.com> wrote: >> > > >>> >> > > >>>> If packaging is similar to Docker, then our Helm charts will >> still >> > > >>>> need some third party base images and such for GPL licensed >> system >> > > >>>> dependencies (besides the kernel itself, there's OpenJDK which is >> > > >>>> definitely used in plenty of projects here). The bits we build on >> > top >> > > >>>> of external components becomes the Apache project that is ALv2 >> and >> > is >> > > >>>> distributed through source archives with convenience binaries (in >> > this >> > > >>>> case, the image layers and YAML files). >> > > >>>> >> > > >>>> I do like the idea of having a central Apache Helm chart repo >> where >> > > >>>> releases can be published. That makes it simpler for users to >> > combine >> > > >>>> multiple Apache charts together, too, based on my past experience >> > with >> > > >>>> using Helm (especially with the new decentralized charts >> > distribution >> > > >>>> model they've switched to). I do think it makes sense to try and >> > > >>>> maintain the sources and such for those charts in the individual >> > > >>>> projects since each PMC may have different workflows on how >> that's >> > > >>>> created, and it makes it easier to release alongside the normal >> > > >>>> project's releases. >> > > >>>> >> > > >>>> On Sat, 5 Sep 2020 at 11:13, Daniel Imberman < >> dimber...@apache.org> >> > > >>> wrote: >> > > >>>>> >> > > >>>>> While I understand the necessity for users to have access to all >> > > >> source >> > > >>>> code, I'm a bit concerned about the complexity this places on the >> > > >>>> open-source project. Let's take Airflow as an example. If we >> want to >> > > >> use >> > > >>>> pgbouncer in our home chart are we expected to be able to build >> this >> > > >>>> external project? >> > > >>>>> >> > > >>>>> I think that if this is ASF policy, then maybe we need to modify >> > the >> > > >>>> policy to meet the current ecosystem. Helm charts are made to >> handle >> > > >>>> complex deployments, and expecting a project to own every other >> > piece >> > > >> of >> > > >>>> their ecosystem is a pretty heavy burden. Is there some >> compromise >> > we >> > > >> can >> > > >>>> come to? Can we store certain pieces such as source code, >> > Dockerfile, >> > > >> and >> > > >>>> docker binary in some Apache cold storage to ensure they are >> > > >>> reproduceable? >> > > >>>>> >> > > >>>>> Ideally I'd just like to keep the burden on project maintainers >> to >> > a >> > > >>>> minimum. This could heavily discourage projects from creating >> > official >> > > >>> helm >> > > >>>> charts. >> > > >>>>> >> > > >>>>> >> > > >>>>> On 2020/09/04 15:05:56, Scott Rigby <sc...@r6by.com> wrote: >> > > >>>>>> Jarek, this is fantastic! Please let me know if I/we can help >> with >> > > >>>> this. The Helm team has written tools to make some of these >> > processes >> > > >>>> easier, including steps to preserve former git history, and >> GitHub >> > > >>> Actions >> > > >>>> for CI (chart testing) and CD (releasing chart version packages >> via >> > > >>> GitHub >> > > >>>> release artifacts). >> > > >>>>>> >> > > >>>>>> Bertrand, many orgs are adopting a git repo naming convention >> > > >>>> "helm-charts" to make this explicit. >> > > >>>>>> >> > > >>>>>> Fun follow-up question: a helm repo is really just an index >> (YAML) >> > > >>>> file of metadata in a format the helm client expects, including >> > links >> > > >> to >> > > >>>> packaged tarballs for each immutable version of a chart. The >> idea of >> > > >>> Apache >> > > >>>> mirror hosting is on the table, one issue we have is that the GCP >> > > >> object >> > > >>>> storage buckets that currently host all previous versions of >> stable >> > > and >> > > >>>> incubator repo charts will be deleted after support for these >> repos >> > > >> ends >> > > >>> on >> > > >>>> Nov 13th ( >> > > >>>> >> > > >>> >> > > >> >> > > >> > >> https://github.com/helm/charts/blob/master/README.md#deprecation-timeline >> > > >>> ). >> > > >>>> Would it be possible for Apache to host a mirror of all these >> > packages >> > > >>>> (note, they are all Apache licenced), or only chart packages >> related >> > > to >> > > >>>> Apache software (airflow, solr, spark, etc)? >> > > >>>>>> >> > > >>>>>> Scott >> > > >>>>>> >> > > >>>>>> On 2020/09/04 08:47:30, Jarek Potiuk <ja...@potiuk.com> wrote: >> > > >>>>>>> Big +1 for doing this within existing ASF infrastructure. >> Maybe >> > > >>>> indeed >> > > >>>>>>> current SVN publishing a release snapshot + mirroring is the >> > > >> best >> > > >>>> way. >> > > >>>>>>> >> > > >>>>>>> IMHO such 'released' charts do not have to have commit >> history at >> > > >>> all >> > > >>>>>>> (except 'release' commits) - just release 'snapshots' is >> enough. >> > > >>> For >> > > >>>> all >> > > >>>>>>> the other development uses, whatever the source repository is, >> > > >> you >> > > >>>> will be >> > > >>>>>>> able too install 'development' chart from the sources. >> > > >>>>>>> >> > > >>>>>>> Still having clear guidelines on how to approach Docker images >> > > >> and >> > > >>>>>>> 'rebuildability' by the users from sources is an important >> aspect >> > > >>>> IMHO. >> > > >>>>>>> J. >> > > >>>>>>> >> > > >>>>>>> pt., 4 wrz 2020, 10:15 użytkownik Bertrand Delacretaz < >> > > >>>>>>> bdelacre...@apache.org> napisał: >> > > >>>>>>> >> > > >>>>>>>> Hi, >> > > >>>>>>>> >> > > >>>>>>>> On Fri, Sep 4, 2020 at 9:22 AM Jarek Potiuk < >> ja...@potiuk.com> >> > > >>>> wrote: >> > > >>>>>>>>> ...I think having ASF "apache/chart" for all "officially >> > > >>>> released" helm >> > > >>>>>>>>> charts is the best solution.... >> > > >>>>>>>> >> > > >>>>>>>> Sounds like a good idea but please use a more specific name, >> > > >>>> "charts" >> > > >>>>>>>> is very generic. >> > > >>>>>>>> >> > > >>>>>>>> If distributing those Helm charts does not requires more >> than a >> > > >>> web >> > > >>>>>>>> server (with some metadata files I suppose) that can probably >> > > >> go >> > > >>>> in a >> > > >>>>>>>> specific folder under >> > > >>> https://dist.apache.org/repos/dist/release/ >> > > >>>>>>>> >> > > >>>>>>>> The next question is what kind of traffic is expected there, >> > > >> and >> > > >>>> can >> > > >>>>>>>> the clients which will download those Helm charts handle >> > > >>> redirects >> > > >>>> to >> > > >>>>>>>> distribution mirrors? If yes the mechanism of >> > > >>>>>>>> http://www.apache.org/dev/release-download-pages.html comes >> to >> > > >>>> mind, >> > > >>>>>>>> if you can piggyback on that mirroring infrastructure that's >> a >> > > >>> big >> > > >>>>>>>> plus IMO. >> > > >>>>>>>> >> > > >>>>>>>> I guess the way to move forward is for the people interested >> in >> > > >>>> this >> > > >>>>>>>> to list the technical and licensing/legal requirements on a >> > > >> wiki >> > > >>>> page >> > > >>>>>>>> or equivalent, to discuss with the ASF infrastructure team >> how >> > > >> to >> > > >>>> best >> > > >>>>>>>> support them. >> > > >>>>>>>> >> > > >>>>>>>> -Bertrand >> > > >>>>>>>> >> > > >>>>>>>> >> > > >>>> >> > --------------------------------------------------------------------- >> > > >>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org >> > > >>>>>>>> For additional commands, e-mail: >> dev-h...@community.apache.org >> > > >>>>>>>> >> > > >>>>>>>> >> > > >>>>>>> >> > > >>>>>> >> > > >>>>>> >> > > >> >> --------------------------------------------------------------------- >> > > >>>>>> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org >> > > >>>>>> For additional commands, e-mail: dev-h...@community.apache.org >> > > >>>>>> >> > > >>>>>> >> > > >>>>> >> > > >>>>> >> > --------------------------------------------------------------------- >> > > >>>>> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org >> > > >>>>> For additional commands, e-mail: dev-h...@community.apache.org >> > > >>>>> >> > > >>>> >> > > >>>> >> > > >>>> -- >> > > >>>> Matt Sicker <boa...@gmail.com> >> > > >>>> >> > > >>>> >> > --------------------------------------------------------------------- >> > > >>>> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org >> > > >>>> For additional commands, e-mail: dev-h...@community.apache.org >> > > >>>> >> > > >>>> >> > > >>> >> > > >> >> > > >> >> > > >> -- >> > > >> +48 660 796 129 >> > > >> >> > > >> > > >> > >> >> >> -- >> +48 660 796 129 >> >
