I am ok with users building their own packages. T.
On Mon, Feb 10, 2020 at 1:47 PM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > I think it should be a deliberate effort for releasing - with voting. We > are releasing the source code and IMHO it should follow the same rules as > releasing airflow itself. > With this change - anyone will be able to build and prepare their own .whl > packages and install them locally, so I do not think there is a need to > automatically release those packages? > > However releasing them in PyPi should be quite an important event as pypi > releases are supposed to be used by users not developers. > > J. > > On Mon, Feb 10, 2020 at 11:16 AM Tomasz Urbaszek < > tomasz.urbas...@polidea.com> wrote: > > > I think as long as we follow: > > > The only people who are supposed to know about such developer resources > > are individuals actively participating in development or following the > dev > > list and thus aware of the conditions placed on unreleased materials. > > > > we should be ok. My impression is that people are usually aware of > > what "nightly build" means and what are the risks. But it's just a > > suggestion that I made thinking about all those people who contribute > > integration and can't use it "officialy" for let say the following 2 > > months. I was also thinking about this result > > > > > https://www.digitalocean.com/currents/december-2019/#generational-expectations-for-open-source-maintenance > > :) > > > > T. > > > > On Mon, Feb 10, 2020 at 10:52 AM Ash Berlin-Taylor <a...@apache.org> > wrote: > > > > > > That might be a grey area according to my reading of the Apache release > > policies: > > > > > > https://apache.org/legal/release-policy.html#publication > > > > During the process of developing software and preparing a release, > > various packages are made available to the development community for > > testing purposes. Projects MUST direct outsiders towards official > releases > > rather than raw source repositories, nightly builds, snapshots, release > > candidates, or any other similar packages. The only people who are > supposed > > to know about such developer resources are individuals actively > > participating in development or following the dev list and thus aware of > > the conditions placed on unreleased materials. > > > On Feb 10 2020, at 9:49 am, Tomasz Urbaszek < > tomasz.urbas...@polidea.com> > > wrote: > > > > As per the frequency of releases maybe we can consider "nightly > > > > builds" for providers? In this way any contributed hook/operator will > > > > be pip-installable in 24h, so users can start to use it = test it. > > > > This can help us reduce the number of releases with unworking > > > > integrations. > > > > > > > > Tomek > > > > > > > > On Mon, Feb 10, 2020 at 12:11 AM Jarek Potiuk < > > jarek.pot...@polidea.com> wrote: > > > > > > > > > > TL;DR; I wanted to discuss the approach we are going to take for > > backported > > > > > providers packages. This is important for PMCs to decide about how > > we are > > > > > going to make release process for it, but I wanted to make it > public > > > > > discussion so that anyone else can chime-in and we can discuss it > as > > a > > > > > community. > > > > > > > > > > *Context* > > > > > As explained in the other thread - we are close to have > > releasable/tested > > > > > backport packages for Airflow 1.10.* series for "providers" > > > > > operators/hooks/packages. The main purpose of those backport > > packages is to > > > > > let users migrate to the new operators before they migrate to 2.0.* > > version > > > > > of Airflow. > > > > > > > > > > The 2.0 version is still some time in the future, and we have a > > number of > > > > > operators/hooks/sensors implemented that are not actively > used/tests > > > > > because they are in master version. There are a number of changes > > and fixes > > > > > only implemented in master/2.0 so it would be great to use them in > > 1.10 - > > > > > to use the new features but also to test the master versions as > > early as > > > > > possible. > > > > > > > > > > Another great property of the backport packages is that they can be > > used to > > > > > ease migration process - users can install the > > "apache-airflow-providers" > > > > > package and start using the new operators without migrating to a > new > > > > > Airflow. They can incrementally move all their DAGs to use the new > > > > > "providers" package and only after all that is migrated they can > > migrate > > > > > Airflow to 2.0 when they are ready. That allows to have a smooth > > migration > > > > > path for those users. > > > > > > > > > > *Testing* > > > > > The issue we have with those packages is that we are not 100% sure > > if the > > > > > "providers" operators will work with any 1.10.* airflow version. > > There were > > > > > no fundamental changes and they SHOULD work - but we never know > > until we > > > > > test. > > > > > > > > > > Some preliminary tests with subset of GCP operators show that the > > operators > > > > > work out-of-the box. We have a big set of "system" tests for "GCP" > > > > > operators that we will run semi-automatically and make sure that > all > > GCP > > > > > operators are working fine. This is already a great compatibility > > test (GCP > > > > > operators are about 1/3 of all operators for Airflow). But also the > > > > > approach used in GCP system tests can be applied to other > operators. > > > > > > > > > > I plan to have a matrix of "compatibilities" in > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Backported+providers+packages+for+Airflow+1.10.*+series > > > > > and > > > > > ask community to add/run tests with other packages as well. It > > should be > > > > > rather easy to add system tests for other systems - following the > > way it is > > > > > implemented for GCP. > > > > > > > > > > *Releases* > > > > > I think the most important decision is how we are going to release > > the > > > > > packages. This is where PMCs have to decide I think as we have > legal > > > > > responsibility for releasing Apache Airflow official software. > > > > > > > > > > What we have now (after the PRs get merged) - wheel and source > > packages > > > > > build automatically in Travis CI and uploaded to file.io ephemeral > > storage. > > > > > The builds upload all the packages there - one big "providers" > > package and > > > > > separate packages for each "provider". > > > > > > > > > > It would be great if we can officially publish packages for > > backporting in > > > > > pypi however and here where we have to agree on the > > > > > process/versioning/cadence. > > > > > > > > > > We can follow the same process/keys etc as for releasing the main > > airflow > > > > > package, but I think it can be a bit more relaxed in terms of > > testing - and > > > > > we can release it more often (as long as there will be new changes > in > > > > > providers). Those packages might be released on "as-is" basis - > > without > > > > > guarantee that they work for all operators/hooks/sensors - and > > without > > > > > guarantee that they will work for all 1.10.* versions. We can have > > the > > > > > "compatibility" statement/matrix in our wiki where people who > tested > > some > > > > > package might simply state that it works for them. At Polidea we > can > > assume > > > > > stewardship on the GCP packages and test them using our automated > > system > > > > > tests for every release for example - maybe others can assume > > > > > stewardship for other providers. > > > > > > > > > > For that - we will need some versioning/release policy. I would say > > a CalVer > > > > > <https://calver.org/> approach might work best (YYYY.MM.DD). And > to > > make it > > > > > simple we should release one "big" providers package with all > > providers in. > > > > > We can have roughly monthly cadence for it. > > > > > > > > > > But I am also open to any suggestions here. > > > > > Please let me know what you think. > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Jarek Potiuk > > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > > > > -- > > > > Tomasz Urbaszek > > > > Polidea | Software Engineer > > > > > > > > M: +48 505 628 493 > > > > E: tomasz.urbas...@polidea.com > > > > > > > > Unique Tech > > > > Check out our projects! > > > > > > > > > > > > > -- > > > > Tomasz Urbaszek > > Polidea | Software Engineer > > > > M: +48 505 628 493 > > E: tomasz.urbas...@polidea.com > > > > Unique Tech > > Check out our projects! > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > -- Tomasz Urbaszek Polidea <https://www.polidea.com/> | Software Engineer M: +48 505 628 493 <+48505628493> E: tomasz.urbas...@polidea.com <tomasz.urbasz...@polidea.com> Unique Tech Check out our projects! <https://www.polidea.com/our-work>
