I think as long as we follow: > The only people who are supposed to know about such developer resources are > individuals actively participating in development or following the dev list > and thus aware of the conditions placed on unreleased materials.
we should be ok. My impression is that people are usually aware of what "nightly build" means and what are the risks. But it's just a suggestion that I made thinking about all those people who contribute integration and can't use it "officialy" for let say the following 2 months. I was also thinking about this result https://www.digitalocean.com/currents/december-2019/#generational-expectations-for-open-source-maintenance :) T. On Mon, Feb 10, 2020 at 10:52 AM Ash Berlin-Taylor <a...@apache.org> wrote: > > That might be a grey area according to my reading of the Apache release > policies: > > https://apache.org/legal/release-policy.html#publication > > During the process of developing software and preparing a release, various > > packages are made available to the development community for testing > > purposes. Projects MUST direct outsiders towards official releases rather > > than raw source repositories, nightly builds, snapshots, release > > candidates, or any other similar packages. The only people who are supposed > > to know about such developer resources are individuals actively > > participating in development or following the dev list and thus aware of > > the conditions placed on unreleased materials. > On Feb 10 2020, at 9:49 am, Tomasz Urbaszek <tomasz.urbas...@polidea.com> > wrote: > > As per the frequency of releases maybe we can consider "nightly > > builds" for providers? In this way any contributed hook/operator will > > be pip-installable in 24h, so users can start to use it = test it. > > This can help us reduce the number of releases with unworking > > integrations. > > > > Tomek > > > > On Mon, Feb 10, 2020 at 12:11 AM Jarek Potiuk <jarek.pot...@polidea.com> > > wrote: > > > > > > TL;DR; I wanted to discuss the approach we are going to take for > > > backported > > > providers packages. This is important for PMCs to decide about how we are > > > going to make release process for it, but I wanted to make it public > > > discussion so that anyone else can chime-in and we can discuss it as a > > > community. > > > > > > *Context* > > > As explained in the other thread - we are close to have releasable/tested > > > backport packages for Airflow 1.10.* series for "providers" > > > operators/hooks/packages. The main purpose of those backport packages is > > > to > > > let users migrate to the new operators before they migrate to 2.0.* > > > version > > > of Airflow. > > > > > > The 2.0 version is still some time in the future, and we have a number of > > > operators/hooks/sensors implemented that are not actively used/tests > > > because they are in master version. There are a number of changes and > > > fixes > > > only implemented in master/2.0 so it would be great to use them in 1.10 - > > > to use the new features but also to test the master versions as early as > > > possible. > > > > > > Another great property of the backport packages is that they can be used > > > to > > > ease migration process - users can install the "apache-airflow-providers" > > > package and start using the new operators without migrating to a new > > > Airflow. They can incrementally move all their DAGs to use the new > > > "providers" package and only after all that is migrated they can migrate > > > Airflow to 2.0 when they are ready. That allows to have a smooth migration > > > path for those users. > > > > > > *Testing* > > > The issue we have with those packages is that we are not 100% sure if the > > > "providers" operators will work with any 1.10.* airflow version. There > > > were > > > no fundamental changes and they SHOULD work - but we never know until we > > > test. > > > > > > Some preliminary tests with subset of GCP operators show that the > > > operators > > > work out-of-the box. We have a big set of "system" tests for "GCP" > > > operators that we will run semi-automatically and make sure that all GCP > > > operators are working fine. This is already a great compatibility test > > > (GCP > > > operators are about 1/3 of all operators for Airflow). But also the > > > approach used in GCP system tests can be applied to other operators. > > > > > > I plan to have a matrix of "compatibilities" in > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Backported+providers+packages+for+Airflow+1.10.*+series > > > and > > > ask community to add/run tests with other packages as well. It should be > > > rather easy to add system tests for other systems - following the way it > > > is > > > implemented for GCP. > > > > > > *Releases* > > > I think the most important decision is how we are going to release the > > > packages. This is where PMCs have to decide I think as we have legal > > > responsibility for releasing Apache Airflow official software. > > > > > > What we have now (after the PRs get merged) - wheel and source packages > > > build automatically in Travis CI and uploaded to file.io ephemeral > > > storage. > > > The builds upload all the packages there - one big "providers" package and > > > separate packages for each "provider". > > > > > > It would be great if we can officially publish packages for backporting in > > > pypi however and here where we have to agree on the > > > process/versioning/cadence. > > > > > > We can follow the same process/keys etc as for releasing the main airflow > > > package, but I think it can be a bit more relaxed in terms of testing - > > > and > > > we can release it more often (as long as there will be new changes in > > > providers). Those packages might be released on "as-is" basis - without > > > guarantee that they work for all operators/hooks/sensors - and without > > > guarantee that they will work for all 1.10.* versions. We can have the > > > "compatibility" statement/matrix in our wiki where people who tested some > > > package might simply state that it works for them. At Polidea we can > > > assume > > > stewardship on the GCP packages and test them using our automated system > > > tests for every release for example - maybe others can assume > > > stewardship for other providers. > > > > > > For that - we will need some versioning/release policy. I would say a > > > CalVer > > > <https://calver.org/> approach might work best (YYYY.MM.DD). And to make > > > it > > > simple we should release one "big" providers package with all providers > > > in. > > > We can have roughly monthly cadence for it. > > > > > > But I am also open to any suggestions here. > > > Please let me know what you think. > > > J. > > > > > > > > > > > > > > > > > > > > > -- > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > -- > > Tomasz Urbaszek > > Polidea | Software Engineer > > > > M: +48 505 628 493 > > E: tomasz.urbas...@polidea.com > > > > Unique Tech > > Check out our projects! > > > -- Tomasz Urbaszek Polidea | Software Engineer M: +48 505 628 493 E: tomasz.urbas...@polidea.com Unique Tech Check out our projects!
