Thanks! Good points Jonathan. I will loop in the OWNERS indeed. I am also in contact with Daniel from Bitnami who manages the Bitnami Airflow image and helm chart and hopefully we can work out an approach where we can work together :). Helm Hub is also good. I updated the AIP to reflect those.
On Mon, Oct 14, 2019 at 3:43 AM Jonathan Miles <j...@cybus.co.uk> wrote: > As a user of the official Helm Chart within The Trade Desk and > contributor of a few things there, I'm happy that the intention is to > reuse it. Like Marcin, I also got the opposite impression from reading > the earlier thread messages, so thanks for clarifying. There are many > valuable contributions to that chart from various people since it was > merged back in December 2018. > > There are two "owners" for the chart registered at the moment: > https://github.com/helm/charts/blob/master/stable/airflow/OWNERS. > Suppose it will be some time until the new image is ready, but probably > worth looping them in by creating a GitHub issue to register the intent > and discuss? Gsemet was the most responsive when I made my contributions. > > Now that there's also Helm Hub - https://hub.helm.sh/charts?q=airflow - > to help deal some of the scale challenges of managing so many charts in > a monorepo, it might also be worth considering moving the chart to its > own repo. It'd be easier to couple the chart to official Airflow > versions and even have multiple branches/tags in parallel. > > Jon > > On 13/10/2019 10:09, Jarek Potiuk wrote: > > It's likely we are both talking about the same :). > > > > I was never proposing to create a new helm chart to Apache Airflow repo. > I > > just proposed that the official docker image is made part of Airflow and > > change the official helm chart to use it. > > > > Official Helm charts are distributed via the helm github repo: > > https://github.com/helm/charts/tree/master/stable/airflow and it should > > remain like that, no doubt about it. Maybe we will have to make a few > > adjustments to the chart itself to make it work with the new Airflow > image, > > but it likely should remain unchanged. There are multiple references to > > puckel image in the chart docs (how to fork etc.) and they should be > > updated to the official image. I believe Daniel Imberman wanted to work > on > > the helm chart once the docker image is released (I am happy to help with > > that as well). > > > > I will make sure to update that in the AIP so that it is clear we are not > > going to have another copy of the chart. > > > > J. > > > > > > On Sun, Oct 13, 2019 at 10:57 AM Marcin Szymański <ms32...@gmail.com> > wrote: > > > >> Maybe I didn't put that clear enough, but my comment referred to the > chart > >> not the image. In fact I've been always running the stable/airflow chart > >> with a custom built image, without any issues. Upgrading it to the > official > >> Airflow image seems like a simple PR, which I'm sure would get accepted. > >> > >> For creating another helm chart I see no benefits, just downsides: > >> - helm repo to maintain > >> - helm repo must be added by the user > >> - chart doesn't go through official helm quality checks > >> - we don't get future helm CI enhancements > >> - confusion among users > >> > >> Next, helm chart releases will have to be aligned not only with Airflow, > >> but also changes to helm and Kubernetes APIs. > >> > >> In short, I believe it's better to upgrade/fix the existing chart than > >> create a new one. > >> > >> On Sun, 13 Oct 2019, 09:39 Jarek Potiuk, <jarek.pot...@polidea.com> > wrote: > >> > >>> Some more comments about Production-ready image and security/patches > >>> aspect. > >>> > >>> I updated the motivation section in AIP-26 > >>> < > >>> > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-26+-+Production-ready+Airflow+Docker+Image+and+helm+chart > >>> to > >>> explain why we are working on community-driven image. I also updated > >>> description about releasing new Airflow image after Python security > >> patches > >>> have been released (hint - we automate it :). > >>> > >>> With the current CI image build process - whenever we rebuild image on > >>> DockerHub we automatically take into account the latest security > patches > >>> released by Python. We can easily make it in the way that we rebuild > and > >>> re-publish even all the already released images to include those > patches > >>> (however we need to include some kind of semver versioning for that I > >>> think). I make sure this is included while working on the POC PR for > >>> PROD-ready image - https://github.com/apache/airflow/pull/6266 > >>> > >>> This solves problems of security patches of the base Python image, but > >> one > >>> thing that came to my mind is whether we should pin dependencies for > >> those > >>> released images (and master image)? This is also related to separate > >>> pinning-dependencies > >>> < > >>> > >> > https://lists.apache.org/thread.html/c643d17833fd3bea534c8c6072cc91e7878348d79dcf756eb94d483f@%3Cdev.airflow.apache.org%3E > >>> thread. > >>> > >>> My questio is - should we regularly re-release old images and re-pin > the > >>> dependencies with security patches? I guess so. > >>> > >>> I am already looking at some ways to automate (optional) pinning of > >>> dependencies and possibly also automating of applying latest security > >>> patches for dependencies. I think however that this should be a > separate > >>> AIP/proposal after we have the production image out and we can release > >> the > >>> production image for testing before we address it. > >>> > >>> I would love to hear your comments. > >>> > >>> J. > >>> > >>> > >>> > >>> On Sun, Oct 13, 2019 at 9:10 AM Jarek Potiuk <jarek.pot...@polidea.com > > > >>> wrote: > >>> > >>>> Sorry - I missed that comment - it was not that you got ignored, I > just > >>>> missed it in the flood of emails I had :(. sorry. > >>>> > >>>> The chart (and corresponding puckel image) is quite ok for the past > but > >>> if > >>>> we want to move forward, we need to make sure that the image, charts > >> etc. > >>>> are driven and managed by the community following release schedule and > >>>> processes of Apache Software Foundation. > >>>> > >>>> The current chart uses (often used) puckel image which was good for > >> quite > >>>> a while but it was not really part of the Apache official community > >>> effort. > >>>> For example one of the rules of releasing software is that any > software > >>>> formally released by the project must be voted by PMC ( > >>>> https://www.apache.org/foundation/how-it-works.html#pmc-members) > >>>> > >>>> By bringing the official image to apache/airflow repository and making > >>>> sure it is part of the release process of Airflow we can release new > >>> images > >>>> at the same time new versions of Airflow get released. Additionally we > >>> can > >>>> provide more maintainability - for example add some more detailed > >>>> instructions and guidelines on how to run Airflow in the production > >>>> environment. We can also make sure we have some optimisations in place > >>> and > >>>> support wider set of audience - hopefully we can get some feedback > from > >>>> people using the official Airflow image/chart and address it longer > >> term. > >>>> Once we incorporate it to our community process, it will be easier for > >>>> everyone to contribute to it - in the same way they contribute to the > >>> code > >>>> of Airflow. > >>>> > >>>> I will update Motivation section in the AIP to include that. > >>>> > >>>> J. > >>>> > >>>> > >>>> > >>>> On Mon, Oct 7, 2019 at 8:06 PM Marcin Szymański <ms32...@gmail.com> > >>> wrote: > >>>>> What are the issues with the existing stable/airflow chart in helm > >>> public > >>>>> repo? I've used it on a number of occasions and it's good enough to > >>> deploy > >>>>> Airflow with Celery executor and run Kubernetes pods from Airflow > >> using > >>>>> in_cluster mode. Not sure if it allows to deploy Kubernetes executor > >>>>> though. > >>>>> > >>>>> On Mon, Oct 7, 2019 at 8:43 AM Jarek Potiuk < > jarek.pot...@polidea.com > >>>>> wrote: > >>>>> > >>>>>> I just created "AIP-26 - Production-ready Airflow Docker Image and > >>> helm > >>>>>> chart > >>>>>> < > >>>>>> > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-26+-+Production-ready+Airflow+Docker+Image+and+helm+chart > >>>>>>> " > >>>>>> proposal and would love community feedback and comments. > >>>>>> > >>>>>> I would love to know what is your expectation from the Docker image > >>> and > >>>>> how > >>>>>> you are using production images currently. > >>>>>> > >>>>>> We were looking at the images available together with Daniel > >> Imberman > >>>>>> (including puckel and Astronomer image) and I started to work on a > >>> draft > >>>>>> POC > >>>>>> <https://github.com/apache/airflow/pull/6266> to incorporate > >> building > >>>>> of > >>>>>> the production-ready image into the CI framework we have. The > >> current > >>>>> POC > >>>>>> uses the same Dockerfile we used for CI images but run with > >>>>>> production-ready parameters. I made a number of refactorings and > >>>>>> simplifications to the build logic and the scripts should be much > >> more > >>>>> sane > >>>>>> now (and unit-testable soon). > >>>>>> > >>>>>> The images are based on Debian buster (Debian 10) LTS released few > >>>>> months > >>>>>> ago (so far we used Debian stretch for CI). It still needs some > >> fixes > >>>>> for > >>>>>> failing tests ( > >> https://travis-ci.org/potiuk/airflow/builds/594281768) > >>>>> but > >>>>>> the image itself can be downloaded from my private repo: `docker > >> pull > >>>>>> potiuk/airflow:master-python3.5`) > >>>>>> > >>>>>> The Docker file is available here: > >>>>>> > >> https://github.com/PolideaInternal/airflow/blob/prod-image/Dockerfile > >>>>>> Once we have good image we will work with Daniel on helm chart so > >> that > >>>>>> Airflow can be easily installed on Kubernetes. > >>>>>> > >>>>>> Any comments and feedback/discussion in the AIP-26 document are > >>> welcome > >>>>> ! > >>>>>> J. > >>>>>> > >>>>>> -- > >>>>>> > >>>>>> Jarek Potiuk > >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>>>>> > >>>>>> M: +48 660 796 129 <+48660796129> > >>>>>> [image: Polidea] <https://www.polidea.com/> > >>>>>> > >>>> > >>>> -- > >>>> > >>>> Jarek Potiuk > >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>>> > >>>> M: +48 660 796 129 <+48660796129> > >>>> [image: Polidea] <https://www.polidea.com/> > >>>> > >>>> > >>> -- > >>> > >>> Jarek Potiuk > >>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>> > >>> M: +48 660 796 129 <+48660796129> > >>> [image: Polidea] <https://www.polidea.com/> > >>> > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
