Some more comments about Production-ready image and security/patches aspect.
I updated the motivation section in AIP-26 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-26+-+Production-ready+Airflow+Docker+Image+and+helm+chart> to explain why we are working on community-driven image. I also updated description about releasing new Airflow image after Python security patches have been released (hint - we automate it :). With the current CI image build process - whenever we rebuild image on DockerHub we automatically take into account the latest security patches released by Python. We can easily make it in the way that we rebuild and re-publish even all the already released images to include those patches (however we need to include some kind of semver versioning for that I think). I make sure this is included while working on the POC PR for PROD-ready image - https://github.com/apache/airflow/pull/6266 This solves problems of security patches of the base Python image, but one thing that came to my mind is whether we should pin dependencies for those released images (and master image)? This is also related to separate pinning-dependencies <https://lists.apache.org/thread.html/c643d17833fd3bea534c8c6072cc91e7878348d79dcf756eb94d483f@%3Cdev.airflow.apache.org%3E> thread. My questio is - should we regularly re-release old images and re-pin the dependencies with security patches? I guess so. I am already looking at some ways to automate (optional) pinning of dependencies and possibly also automating of applying latest security patches for dependencies. I think however that this should be a separate AIP/proposal after we have the production image out and we can release the production image for testing before we address it. I would love to hear your comments. J. On Sun, Oct 13, 2019 at 9:10 AM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > Sorry - I missed that comment - it was not that you got ignored, I just > missed it in the flood of emails I had :(. sorry. > > The chart (and corresponding puckel image) is quite ok for the past but if > we want to move forward, we need to make sure that the image, charts etc. > are driven and managed by the community following release schedule and > processes of Apache Software Foundation. > > The current chart uses (often used) puckel image which was good for quite > a while but it was not really part of the Apache official community effort. > For example one of the rules of releasing software is that any software > formally released by the project must be voted by PMC ( > https://www.apache.org/foundation/how-it-works.html#pmc-members) > > By bringing the official image to apache/airflow repository and making > sure it is part of the release process of Airflow we can release new images > at the same time new versions of Airflow get released. Additionally we can > provide more maintainability - for example add some more detailed > instructions and guidelines on how to run Airflow in the production > environment. We can also make sure we have some optimisations in place and > support wider set of audience - hopefully we can get some feedback from > people using the official Airflow image/chart and address it longer term. > Once we incorporate it to our community process, it will be easier for > everyone to contribute to it - in the same way they contribute to the code > of Airflow. > > I will update Motivation section in the AIP to include that. > > J. > > > > On Mon, Oct 7, 2019 at 8:06 PM Marcin Szymański <ms32...@gmail.com> wrote: > >> What are the issues with the existing stable/airflow chart in helm public >> repo? I've used it on a number of occasions and it's good enough to deploy >> Airflow with Celery executor and run Kubernetes pods from Airflow using >> in_cluster mode. Not sure if it allows to deploy Kubernetes executor >> though. >> >> On Mon, Oct 7, 2019 at 8:43 AM Jarek Potiuk <jarek.pot...@polidea.com> >> wrote: >> >> > I just created "AIP-26 - Production-ready Airflow Docker Image and helm >> > chart >> > < >> > >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-26+-+Production-ready+Airflow+Docker+Image+and+helm+chart >> > >" >> > proposal and would love community feedback and comments. >> > >> > I would love to know what is your expectation from the Docker image and >> how >> > you are using production images currently. >> > >> > We were looking at the images available together with Daniel Imberman >> > (including puckel and Astronomer image) and I started to work on a draft >> > POC >> > <https://github.com/apache/airflow/pull/6266> to incorporate building >> of >> > the production-ready image into the CI framework we have. The current >> POC >> > uses the same Dockerfile we used for CI images but run with >> > production-ready parameters. I made a number of refactorings and >> > simplifications to the build logic and the scripts should be much more >> sane >> > now (and unit-testable soon). >> > >> > The images are based on Debian buster (Debian 10) LTS released few >> months >> > ago (so far we used Debian stretch for CI). It still needs some fixes >> for >> > failing tests (https://travis-ci.org/potiuk/airflow/builds/594281768) >> but >> > the image itself can be downloaded from my private repo: `docker pull >> > potiuk/airflow:master-python3.5`) >> > >> > The Docker file is available here: >> > https://github.com/PolideaInternal/airflow/blob/prod-image/Dockerfile >> > >> > Once we have good image we will work with Daniel on helm chart so that >> > Airflow can be easily installed on Kubernetes. >> > >> > Any comments and feedback/discussion in the AIP-26 document are welcome >> ! >> > >> > J. >> > >> > -- >> > >> > Jarek Potiuk >> > Polidea <https://www.polidea.com/> | Principal Software Engineer >> > >> > M: +48 660 796 129 <+48660796129> >> > [image: Polidea] <https://www.polidea.com/> >> > >> > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
