It's likely we are both talking about the same :). I was never proposing to create a new helm chart to Apache Airflow repo. I just proposed that the official docker image is made part of Airflow and change the official helm chart to use it.
Official Helm charts are distributed via the helm github repo: https://github.com/helm/charts/tree/master/stable/airflow and it should remain like that, no doubt about it. Maybe we will have to make a few adjustments to the chart itself to make it work with the new Airflow image, but it likely should remain unchanged. There are multiple references to puckel image in the chart docs (how to fork etc.) and they should be updated to the official image. I believe Daniel Imberman wanted to work on the helm chart once the docker image is released (I am happy to help with that as well). I will make sure to update that in the AIP so that it is clear we are not going to have another copy of the chart. J. On Sun, Oct 13, 2019 at 10:57 AM Marcin Szymański <ms32...@gmail.com> wrote: > Maybe I didn't put that clear enough, but my comment referred to the chart > not the image. In fact I've been always running the stable/airflow chart > with a custom built image, without any issues. Upgrading it to the official > Airflow image seems like a simple PR, which I'm sure would get accepted. > > For creating another helm chart I see no benefits, just downsides: > - helm repo to maintain > - helm repo must be added by the user > - chart doesn't go through official helm quality checks > - we don't get future helm CI enhancements > - confusion among users > > Next, helm chart releases will have to be aligned not only with Airflow, > but also changes to helm and Kubernetes APIs. > > In short, I believe it's better to upgrade/fix the existing chart than > create a new one. > > On Sun, 13 Oct 2019, 09:39 Jarek Potiuk, <jarek.pot...@polidea.com> wrote: > > > Some more comments about Production-ready image and security/patches > > aspect. > > > > I updated the motivation section in AIP-26 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-26+-+Production-ready+Airflow+Docker+Image+and+helm+chart > > > > > to > > explain why we are working on community-driven image. I also updated > > description about releasing new Airflow image after Python security > patches > > have been released (hint - we automate it :). > > > > With the current CI image build process - whenever we rebuild image on > > DockerHub we automatically take into account the latest security patches > > released by Python. We can easily make it in the way that we rebuild and > > re-publish even all the already released images to include those patches > > (however we need to include some kind of semver versioning for that I > > think). I make sure this is included while working on the POC PR for > > PROD-ready image - https://github.com/apache/airflow/pull/6266 > > > > This solves problems of security patches of the base Python image, but > one > > thing that came to my mind is whether we should pin dependencies for > those > > released images (and master image)? This is also related to separate > > pinning-dependencies > > < > > > https://lists.apache.org/thread.html/c643d17833fd3bea534c8c6072cc91e7878348d79dcf756eb94d483f@%3Cdev.airflow.apache.org%3E > > > > > thread. > > > > My questio is - should we regularly re-release old images and re-pin the > > dependencies with security patches? I guess so. > > > > I am already looking at some ways to automate (optional) pinning of > > dependencies and possibly also automating of applying latest security > > patches for dependencies. I think however that this should be a separate > > AIP/proposal after we have the production image out and we can release > the > > production image for testing before we address it. > > > > I would love to hear your comments. > > > > J. > > > > > > > > On Sun, Oct 13, 2019 at 9:10 AM Jarek Potiuk <jarek.pot...@polidea.com> > > wrote: > > > > > Sorry - I missed that comment - it was not that you got ignored, I just > > > missed it in the flood of emails I had :(. sorry. > > > > > > The chart (and corresponding puckel image) is quite ok for the past but > > if > > > we want to move forward, we need to make sure that the image, charts > etc. > > > are driven and managed by the community following release schedule and > > > processes of Apache Software Foundation. > > > > > > The current chart uses (often used) puckel image which was good for > quite > > > a while but it was not really part of the Apache official community > > effort. > > > For example one of the rules of releasing software is that any software > > > formally released by the project must be voted by PMC ( > > > https://www.apache.org/foundation/how-it-works.html#pmc-members) > > > > > > By bringing the official image to apache/airflow repository and making > > > sure it is part of the release process of Airflow we can release new > > images > > > at the same time new versions of Airflow get released. Additionally we > > can > > > provide more maintainability - for example add some more detailed > > > instructions and guidelines on how to run Airflow in the production > > > environment. We can also make sure we have some optimisations in place > > and > > > support wider set of audience - hopefully we can get some feedback from > > > people using the official Airflow image/chart and address it longer > term. > > > Once we incorporate it to our community process, it will be easier for > > > everyone to contribute to it - in the same way they contribute to the > > code > > > of Airflow. > > > > > > I will update Motivation section in the AIP to include that. > > > > > > J. > > > > > > > > > > > > On Mon, Oct 7, 2019 at 8:06 PM Marcin Szymański <ms32...@gmail.com> > > wrote: > > > > > >> What are the issues with the existing stable/airflow chart in helm > > public > > >> repo? I've used it on a number of occasions and it's good enough to > > deploy > > >> Airflow with Celery executor and run Kubernetes pods from Airflow > using > > >> in_cluster mode. Not sure if it allows to deploy Kubernetes executor > > >> though. > > >> > > >> On Mon, Oct 7, 2019 at 8:43 AM Jarek Potiuk <jarek.pot...@polidea.com > > > > >> wrote: > > >> > > >> > I just created "AIP-26 - Production-ready Airflow Docker Image and > > helm > > >> > chart > > >> > < > > >> > > > >> > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-26+-+Production-ready+Airflow+Docker+Image+and+helm+chart > > >> > >" > > >> > proposal and would love community feedback and comments. > > >> > > > >> > I would love to know what is your expectation from the Docker image > > and > > >> how > > >> > you are using production images currently. > > >> > > > >> > We were looking at the images available together with Daniel > Imberman > > >> > (including puckel and Astronomer image) and I started to work on a > > draft > > >> > POC > > >> > <https://github.com/apache/airflow/pull/6266> to incorporate > building > > >> of > > >> > the production-ready image into the CI framework we have. The > current > > >> POC > > >> > uses the same Dockerfile we used for CI images but run with > > >> > production-ready parameters. I made a number of refactorings and > > >> > simplifications to the build logic and the scripts should be much > more > > >> sane > > >> > now (and unit-testable soon). > > >> > > > >> > The images are based on Debian buster (Debian 10) LTS released few > > >> months > > >> > ago (so far we used Debian stretch for CI). It still needs some > fixes > > >> for > > >> > failing tests ( > https://travis-ci.org/potiuk/airflow/builds/594281768) > > >> but > > >> > the image itself can be downloaded from my private repo: `docker > pull > > >> > potiuk/airflow:master-python3.5`) > > >> > > > >> > The Docker file is available here: > > >> > > https://github.com/PolideaInternal/airflow/blob/prod-image/Dockerfile > > >> > > > >> > Once we have good image we will work with Daniel on helm chart so > that > > >> > Airflow can be easily installed on Kubernetes. > > >> > > > >> > Any comments and feedback/discussion in the AIP-26 document are > > welcome > > >> ! > > >> > > > >> > J. > > >> > > > >> > -- > > >> > > > >> > Jarek Potiuk > > >> > Polidea <https://www.polidea.com/> | Principal Software Engineer > > >> > > > >> > M: +48 660 796 129 <+48660796129> > > >> > [image: Polidea] <https://www.polidea.com/> > > >> > > > >> > > > > > > > > > -- > > > > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
