Yes, I understood this. But I m personally more paranoid about releasing. So formally vote (and corresponding testing) was done on rc. If we rebuild and resign, wouldn't that mean we also need to revote?
If I understand correctly, there will be some changed version string in distributed sources (setup.py?). So there is some binary difference. And just talking about me, doing that repackaging I would certainly mess it up and package some unwanted changes. On Thu, May 2, 2019 at 5:43 PM Robert Bradshaw <rober...@google.com> wrote: > On Thu, May 2, 2019 at 5:24 PM Michael Luckey <adude3...@gmail.com> wrote: > > > > Thanks Ahmet for calling out to the airflow folks. I believe, I am able > to follow their argument. So from my point of view I do not have an issue > with apache policy. But honestly still trying to wrap my head around > Roberts concern with rebuilding/resigning. Currently, our actual release is > only a tag on source repo and promoting artefacts. Do not yet understand > how that needs to change to get PyPi included. > > It's not a big change, but let me clarify. > > Currently our release preparation goes something like this: > > 1) Check out the repo, update the versions to 2.x, build and sign the > artifacts. > 2) Announce these artifacts as rcN > 2a) Push the artifacts to SVN dev/... > 2b) Push artifacts to the apache maven repository. > 3) Depending on vote, go back to step (1) or forward to step (4). > 4) Copy these artifacts as the actual release. > > Now if we just try to add (2c) Push these artifacts to Pypi, it will > be treated (by pypi's tooling, anyone who downloads the tarball, ...) > as an actual release. You also can't re-push a tarball with the same > name and different contents (the idea being that named releases should > never change). So we'd need to change step (1) to update the version > to 2.x.rcN *and* add a step in (4) to update the version to 2.x (no rc > suffix), rebuild, resign before publishing. > > As mentioned, possibly the rcN suffix could be part of the building > step for Python. > > > On Wed, May 1, 2019 at 1:33 AM Ahmet Altay <al...@google.com> wrote: > >> > >> Michael, Max and other folks who are concerned about the compatibility > with the apache release policy. Does the information in this thread > sufficiently address your concerns? Especially the part where, the rc > artifacts will be protected by a flag (i.e. --pre) from general consumption. > >> > >> On Tue, Apr 30, 2019 at 3:59 PM Robert Bradshaw <rober...@google.com> > wrote: > >>> > >>> On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay <al...@google.com> wrote: > >>> > > >>> > This conversation get quite Python centric. Is there a similar need > for Java? > >>> > >>> I think Java is already covered. Go is a different story (but the even > >>> versioning and releasing is being worked out). > >>> > >>> > On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw <rober...@google.com> > wrote: > >>> >> > >>> >> If we can, by the apache guidelines, post RCs to pypy that is > >>> >> definitely the way to go. (Note that test.pypi is for developing > >>> >> against the pypi interface, not for pushing anything real.) The > caveat > >>> >> about naming these with rcN in the version number still applies > >>> >> (that's how pypi guards them against non-explicit installs). > >>> > > >>> > Related to the caveat, I believe this can be easily scripted or even > made part of the travis/wheels pipeline to take the release branch, edit > the version string in place to add rc, and build the necessary files. > >>> > >>> Yes. But the resulting artifacts would have to be rebuilt (and > >>> re-signed) without the version edit for the actual release. (Well, we > >>> could possibly edit the artifacts rather than rebuild them.) And > >>> pushing un-edited ones early would be really bad. (It's the classic > >>> tension of whether a pre-release should be marked internally or > >>> externally, re-publishing a new set of bits for the actual release or > >>> re-using version numbers for different sets of bits. Pypi does one, > >>> apache does another...) > >>> > >>> >> The advantage is that a user can do "pip install --pre apache-beam" > to > >>> >> get the latest rc rather than "pip install > >>> >> > https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path"; > >>> >> > >>> >> On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada <pabl...@google.com> > wrote: > >>> >> > > >>> >> > Aw that's interesting! > >>> >> > > >>> >> > I think, with these considerations, I am only marginally more > inclined towards publishing to test.pypi. That would make me a +0.9 on > publishing RCs to the main pip repo then. > >>> >> > > >>> >> > Thanks for doing the research Ahmet. :) > >>> >> > Best > >>> >> > -P > >>> >> > > >>> >> > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay <al...@google.com> > wrote: > >>> >> >> > >>> >> >> I asked to Airflow folks about this. See [1] for the full > response and a link to one of their RC emails. To summarize their position > (specifically for pypi) is: Unless a user does something explicit (such as > using a flag, or explicitly requesting an rc release), pip install will not > serve RC binaries. And that is compatible with RC section of > http://www.apache.org/legal/release-policy.html#release-types > >>> >> >> > >>> >> >> Ahmet > >>> >> >> > >>> >> >> [1] > https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E > >>> >> >> > >>> >> >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay <al...@google.com> > wrote: > >>> >> >>> > >>> >> >>> The incremental value of publishing python artifacts to a > separate place but not to actual pypi listing will be low. Users can > already download RC artifacts, or even pip install from http location > directly. I think the incremental value will be low, because for a user or > a downstream library to test with Beam RCs using their usual ways will > still require them to get other dependencies from the regular pypi listing. > That would mean they need to change their setup to test with beam rcs, > which is the same state as today. There will be some incremental value of > putting them in more obvious places (e.g. pypi test repository). I would > rather not complicate the release process for doing this. > >>> >> >>> > >>> >> >>> > >>> >> >>> > >>> >> >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles < > k...@apache.org> wrote: > >>> >> >>>> > >>> >> >>>> Pip is also able to be pointed at any raw hosted directory for > the install, right? So we could publish RCs or snapshots somewhere with > more obvious caveats and not interfere with the pypi list of actual > releases. Much like the Java snapshots are stored in a separate opt-in > repository. > >>> >> >>>> > >>> >> >>>> Kenn > >>> >> >>>> > >>> >> >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels < > m...@apache.org> wrote: > >>> >> >>>>> > >>> >> >>>>> > wouldn't that be in conflict with Apache release policy [1] > ? > >>> >> >>>>> > [1] http://www.apache.org/legal/release-policy.html > >>> >> >>>>> > >>> >> >>>>> Indeed, advertising pre-release artifacts is against ASF > rules. For > >>> >> >>>>> example, Flink was asked to remove a link to the Maven > snapshot > >>> >> >>>>> repository from their download page. > >>> >> >>>>> > >>> >> >>>>> However, that does not mean we cannot publish Python > artifacts. We just > >>> >> >>>>> have to clearly mark them for developers only and not > advertise them > >>> >> >>>>> alongside with the official releases. > >>> >> >>>>> > >>> >> >>>>> -Max > >>> >> >>>>> > >>> >> >>>>> On 25.04.19 10:23, Robert Bradshaw wrote: > >>> >> >>>>> > Don't we push java artifacts to maven repositories as part > of the RC > >>> >> >>>>> > process? And completely unvetted snapshots? (Or is this OK > because > >>> >> >>>>> > they are special opt-in apache-only ones?) > >>> >> >>>>> > > >>> >> >>>>> > I am generally in favor of the idea, but would like to > avoid increased > >>> >> >>>>> > toil on the release manager. > >>> >> >>>>> > > >>> >> >>>>> > One potential hitch I see is that current release process > updates the > >>> >> >>>>> > versions to x.y.z (no RC or other pre-release indicator in > the version > >>> >> >>>>> > number) whereas pypi (and other systems) typically expect > distinct > >>> >> >>>>> > (recognizable) version numbers for each attempt, and only > the actual > >>> >> >>>>> > final result has the actual final release version. > >>> >> >>>>> > > >>> >> >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay < > al...@google.com> wrote: > >>> >> >>>>> >> > >>> >> >>>>> >> I do not know the answer.I believe this will be similar to > sharing the RC artifacts for validation purposes and would not be a formal > release by itself. But I am not an expert and I hope others will share > their opinions. > >>> >> >>>>> >> > >>> >> >>>>> >> I quickly searched pypi for apache projects and found at > least airflow [1] and libcloud [2] are publishing rc artifacts to pypi. We > can reach out to those communities and learn about their processes. > >>> >> >>>>> >> > >>> >> >>>>> >> Ahmet > >>> >> >>>>> >> > >>> >> >>>>> >> [1] https://pypi.org/project/apache-airflow/#history > >>> >> >>>>> >> [2] https://pypi.org/project/apache-libcloud/#history > >>> >> >>>>> >> > >>> >> >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey < > adude3...@gmail.com> wrote: > >>> >> >>>>> >>> > >>> >> >>>>> >>> Hi, > >>> >> >>>>> >>> > >>> >> >>>>> >>> wouldn't that be in conflict with Apache release policy > [1] ? > >>> >> >>>>> >>> > >>> >> >>>>> >>> [1] http://www.apache.org/legal/release-policy.html > >>> >> >>>>> >>> > >>> >> >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold < > amyrv...@google.com> wrote: > >>> >> >>>>> >>>> > >>> >> >>>>> >>>> Great idea. I like the RC candidates to follow as much > as the release artifact process as possible. > >>> >> >>>>> >>>> > >>> >> >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay < > al...@google.com> wrote: > >>> >> >>>>> >>>>> > >>> >> >>>>> >>>>> To clarify my proposal, I am proposing publishing to > the production pypi repository with an rc tag in the version. And in turn > allow users to depend on beam's rc version + all the other regular > dependencies users would have directly from pypi. > >>> >> >>>>> >>>>> > >>> >> >>>>> >>>>> Publishing to test pypi repo would also be helpful if > test pypi repo also mirrors other packages that exist in the production > pypi repository. > >>> >> >>>>> >>>>> > >>> >> >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada < > pabl...@google.com> wrote: > >>> >> >>>>> >>>>>> > >>> >> >>>>> >>>>>> I think this is a great idea. A way of doing it for > python would be by using the test repository for PyPi[1], and that way we > would not have to do an official PyPi release, but still would be able to > install it with pip (by passing an extra flag), and test. > >>> >> >>>>> >>>>>> > >>> >> >>>>> >>>>>> In fact, there are some Beam artifacts already in > there[2]. At some point I looked into this, but couldn't figure out who has > access/the password for it. > >>> >> >>>>> >>>>> > >>> >> >>>>> >>>>> > >>> >> >>>>> >>>>> I also don't know who owns beam package in test pypi > repo. Does anybody know? > >>> >> >>>>> >>>>> > >>> >> >>>>> >>>>>> > >>> >> >>>>> >>>>>> > >>> >> >>>>> >>>>>> In short: +1, and I would suggest using the test PyPi > repo to avoid publishing to the main PyPi repo. > >>> >> >>>>> >>>>>> Best > >>> >> >>>>> >>>>>> -P. > >>> >> >>>>> >>>>>> > >>> >> >>>>> >>>>>> [1] https://test.pypi.org/ > >>> >> >>>>> >>>>>> [2] https://test.pypi.org/project/apache-beam/ > >>> >> >>>>> >>>>>> > >>> >> >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay < > al...@google.com> wrote: > >>> >> >>>>> >>>>>>> > >>> >> >>>>> >>>>>>> Hi all, > >>> >> >>>>> >>>>>>> > >>> >> >>>>> >>>>>>> What do you think about the idea of publishing > pre-release artifacts as part of the RC emails? > >>> >> >>>>> >>>>>>> > >>> >> >>>>> >>>>>>> For Python this would translate into publishing the > same artifacts from RC email with a version like "2.X.0rcY" to pypi. I do > not know, but I am guessing we can do a similar thing with Maven central > for Java artifacts as well. > >>> >> >>>>> >>>>>>> > >>> >> >>>>> >>>>>>> Advantages would be: > >>> >> >>>>> >>>>>>> - Allow end users to validate RCs for their own > purposes using the same exact process they will normally use. > >>> >> >>>>> >>>>>>> - Enable early-adaptors to start using RC releases > early on in the release cycle if that is what they would like to do. This > will in turn reduce time pressure on some releases. Especially for cases > like someone needs a release to be finalized for an upcoming event. > >>> >> >>>>> >>>>>>> > >>> >> >>>>> >>>>>>> There will also be disadvantages, some I could think > of: > >>> >> >>>>> >>>>>>> - Users could request support for RC artifacts. > Hopefully in the form of feedback for us to improve the release. But it > could also be in the form of folks using RC artifacts for production for a > long time. > >>> >> >>>>> >>>>>>> - It will add toil to the current release process, > there will be one more step for each RC. I think for python this will be a > small step but nevertheless it will be additional work. > >>> >> >>>>> >>>>>>> > >>> >> >>>>> >>>>>>> For an example of this, you can take a look at > tensorflow releases. For 1.13 there were 3 pre-releases [1]. > >>> >> >>>>> >>>>>>> > >>> >> >>>>> >>>>>>> Ahmet > >>> >> >>>>> >>>>>>> > >>> >> >>>>> >>>>>>> [1] https://pypi.org/project/tensorflow/#history >
