Maybe I didn't put that clear enough, but my comment referred to the chart
not the image. In fact I've been always running the stable/airflow chart
with a custom built image, without any issues. Upgrading it to the official
Airflow image seems like a simple PR, which I'm sure would get accepted.
For creating another helm chart I see no benefits, just downsides:
- helm repo to maintain
- helm repo must be added by the user
- chart doesn't go through official helm quality checks
- we don't get future helm CI enhancements
- confusion among users
Next, helm chart releases will have to be aligned not only with Airflow,
but also changes to helm and Kubernetes APIs.
In short, I believe it's better to upgrade/fix the existing chart than
create a new one.
On Sun, 13 Oct 2019, 09:39 Jarek Potiuk, <jarek.pot...@polidea.com> wrote:
Some more comments about Production-ready image and security/patches
aspect.
I updated the motivation section in AIP-26
<
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-26+-+Production-ready+Airflow+Docker+Image+and+helm+chart
to
explain why we are working on community-driven image. I also updated
description about releasing new Airflow image after Python security
patches
have been released (hint - we automate it :).
With the current CI image build process - whenever we rebuild image on
DockerHub we automatically take into account the latest security patches
released by Python. We can easily make it in the way that we rebuild and
re-publish even all the already released images to include those patches
(however we need to include some kind of semver versioning for that I
think). I make sure this is included while working on the POC PR for
PROD-ready image - https://github.com/apache/airflow/pull/6266
This solves problems of security patches of the base Python image, but
one
thing that came to my mind is whether we should pin dependencies for
those
released images (and master image)? This is also related to separate
pinning-dependencies
<
https://lists.apache.org/thread.html/c643d17833fd3bea534c8c6072cc91e7878348d79dcf756eb94d483f@%3Cdev.airflow.apache.org%3E
thread.
My questio is - should we regularly re-release old images and re-pin the
dependencies with security patches? I guess so.
I am already looking at some ways to automate (optional) pinning of
dependencies and possibly also automating of applying latest security
patches for dependencies. I think however that this should be a separate
AIP/proposal after we have the production image out and we can release
the
production image for testing before we address it.
I would love to hear your comments.
J.
On Sun, Oct 13, 2019 at 9:10 AM Jarek Potiuk <jarek.pot...@polidea.com>
wrote:
Sorry - I missed that comment - it was not that you got ignored, I just
missed it in the flood of emails I had :(. sorry.
The chart (and corresponding puckel image) is quite ok for the past but
if
we want to move forward, we need to make sure that the image, charts
etc.
are driven and managed by the community following release schedule and
processes of Apache Software Foundation.
The current chart uses (often used) puckel image which was good for
quite
a while but it was not really part of the Apache official community
effort.
For example one of the rules of releasing software is that any software
formally released by the project must be voted by PMC (
https://www.apache.org/foundation/how-it-works.html#pmc-members)
By bringing the official image to apache/airflow repository and making
sure it is part of the release process of Airflow we can release new
images
at the same time new versions of Airflow get released. Additionally we
can
provide more maintainability - for example add some more detailed
instructions and guidelines on how to run Airflow in the production
environment. We can also make sure we have some optimisations in place
and
support wider set of audience - hopefully we can get some feedback from
people using the official Airflow image/chart and address it longer
term.
Once we incorporate it to our community process, it will be easier for
everyone to contribute to it - in the same way they contribute to the
code
of Airflow.
I will update Motivation section in the AIP to include that.
J.
On Mon, Oct 7, 2019 at 8:06 PM Marcin Szymański <ms32...@gmail.com>
wrote:
What are the issues with the existing stable/airflow chart in helm
public
repo? I've used it on a number of occasions and it's good enough to
deploy
Airflow with Celery executor and run Kubernetes pods from Airflow
using
in_cluster mode. Not sure if it allows to deploy Kubernetes executor
though.
On Mon, Oct 7, 2019 at 8:43 AM Jarek Potiuk <jarek.pot...@polidea.com
wrote:
I just created "AIP-26 - Production-ready Airflow Docker Image and
helm
chart
<
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-26+-+Production-ready+Airflow+Docker+Image+and+helm+chart
"
proposal and would love community feedback and comments.
I would love to know what is your expectation from the Docker image
and
how
you are using production images currently.
We were looking at the images available together with Daniel
Imberman
(including puckel and Astronomer image) and I started to work on a
draft
POC
<https://github.com/apache/airflow/pull/6266> to incorporate
building
of
the production-ready image into the CI framework we have. The
current
POC
uses the same Dockerfile we used for CI images but run with
production-ready parameters. I made a number of refactorings and
simplifications to the build logic and the scripts should be much
more
sane
now (and unit-testable soon).
The images are based on Debian buster (Debian 10) LTS released few
months
ago (so far we used Debian stretch for CI). It still needs some
fixes
for
failing tests (
https://travis-ci.org/potiuk/airflow/builds/594281768)
but
the image itself can be downloaded from my private repo: `docker
pull
potiuk/airflow:master-python3.5`)
The Docker file is available here:
https://github.com/PolideaInternal/airflow/blob/prod-image/Dockerfile
Once we have good image we will work with Daniel on helm chart so
that
Airflow can be easily installed on Kubernetes.
Any comments and feedback/discussion in the AIP-26 document are
welcome
!
J.
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
