Michael, Max and other folks who are concerned about the compatibility with the apache release policy. Does the information in this thread sufficiently address your concerns? Especially the part where, the rc artifacts will be protected by a flag (i.e. --pre) from general consumption.
On Tue, Apr 30, 2019 at 3:59 PM Robert Bradshaw <rober...@google.com> wrote: > On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay <al...@google.com> wrote: > > > > This conversation get quite Python centric. Is there a similar need for > Java? > > I think Java is already covered. Go is a different story (but the even > versioning and releasing is being worked out). > > > On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw <rober...@google.com> > wrote: > >> > >> If we can, by the apache guidelines, post RCs to pypy that is > >> definitely the way to go. (Note that test.pypi is for developing > >> against the pypi interface, not for pushing anything real.) The caveat > >> about naming these with rcN in the version number still applies > >> (that's how pypi guards them against non-explicit installs). > > > > Related to the caveat, I believe this can be easily scripted or even > made part of the travis/wheels pipeline to take the release branch, edit > the version string in place to add rc, and build the necessary files. > > Yes. But the resulting artifacts would have to be rebuilt (and > re-signed) without the version edit for the actual release. (Well, we > could possibly edit the artifacts rather than rebuild them.) And > pushing un-edited ones early would be really bad. (It's the classic > tension of whether a pre-release should be marked internally or > externally, re-publishing a new set of bits for the actual release or > re-using version numbers for different sets of bits. Pypi does one, > apache does another...) > > >> The advantage is that a user can do "pip install --pre apache-beam" to > >> get the latest rc rather than "pip install > >> https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path > " > >> > >> On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada <pabl...@google.com> > wrote: > >> > > >> > Aw that's interesting! > >> > > >> > I think, with these considerations, I am only marginally more > inclined towards publishing to test.pypi. That would make me a +0.9 on > publishing RCs to the main pip repo then. > >> > > >> > Thanks for doing the research Ahmet. :) > >> > Best > >> > -P > >> > > >> > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay <al...@google.com> wrote: > >> >> > >> >> I asked to Airflow folks about this. See [1] for the full response > and a link to one of their RC emails. To summarize their position > (specifically for pypi) is: Unless a user does something explicit (such as > using a flag, or explicitly requesting an rc release), pip install will not > serve RC binaries. And that is compatible with RC section of > http://www.apache.org/legal/release-policy.html#release-types > >> >> > >> >> Ahmet > >> >> > >> >> [1] > https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E > >> >> > >> >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay <al...@google.com> > wrote: > >> >>> > >> >>> The incremental value of publishing python artifacts to a separate > place but not to actual pypi listing will be low. Users can already > download RC artifacts, or even pip install from http location directly. I > think the incremental value will be low, because for a user or a downstream > library to test with Beam RCs using their usual ways will still require > them to get other dependencies from the regular pypi listing. That would > mean they need to change their setup to test with beam rcs, which is the > same state as today. There will be some incremental value of putting them > in more obvious places (e.g. pypi test repository). I would rather not > complicate the release process for doing this. > >> >>> > >> >>> > >> >>> > >> >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles <k...@apache.org> > wrote: > >> >>>> > >> >>>> Pip is also able to be pointed at any raw hosted directory for the > install, right? So we could publish RCs or snapshots somewhere with more > obvious caveats and not interfere with the pypi list of actual releases. > Much like the Java snapshots are stored in a separate opt-in repository. > >> >>>> > >> >>>> Kenn > >> >>>> > >> >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels <m...@apache.org> > wrote: > >> >>>>> > >> >>>>> > wouldn't that be in conflict with Apache release policy [1] ? > >> >>>>> > [1] http://www.apache.org/legal/release-policy.html > >> >>>>> > >> >>>>> Indeed, advertising pre-release artifacts is against ASF rules. > For > >> >>>>> example, Flink was asked to remove a link to the Maven snapshot > >> >>>>> repository from their download page. > >> >>>>> > >> >>>>> However, that does not mean we cannot publish Python artifacts. > We just > >> >>>>> have to clearly mark them for developers only and not advertise > them > >> >>>>> alongside with the official releases. > >> >>>>> > >> >>>>> -Max > >> >>>>> > >> >>>>> On 25.04.19 10:23, Robert Bradshaw wrote: > >> >>>>> > Don't we push java artifacts to maven repositories as part of > the RC > >> >>>>> > process? And completely unvetted snapshots? (Or is this OK > because > >> >>>>> > they are special opt-in apache-only ones?) > >> >>>>> > > >> >>>>> > I am generally in favor of the idea, but would like to avoid > increased > >> >>>>> > toil on the release manager. > >> >>>>> > > >> >>>>> > One potential hitch I see is that current release process > updates the > >> >>>>> > versions to x.y.z (no RC or other pre-release indicator in the > version > >> >>>>> > number) whereas pypi (and other systems) typically expect > distinct > >> >>>>> > (recognizable) version numbers for each attempt, and only the > actual > >> >>>>> > final result has the actual final release version. > >> >>>>> > > >> >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay <al...@google.com> > wrote: > >> >>>>> >> > >> >>>>> >> I do not know the answer.I believe this will be similar to > sharing the RC artifacts for validation purposes and would not be a formal > release by itself. But I am not an expert and I hope others will share > their opinions. > >> >>>>> >> > >> >>>>> >> I quickly searched pypi for apache projects and found at least > airflow [1] and libcloud [2] are publishing rc artifacts to pypi. We can > reach out to those communities and learn about their processes. > >> >>>>> >> > >> >>>>> >> Ahmet > >> >>>>> >> > >> >>>>> >> [1] https://pypi.org/project/apache-airflow/#history > >> >>>>> >> [2] https://pypi.org/project/apache-libcloud/#history > >> >>>>> >> > >> >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey < > adude3...@gmail.com> wrote: > >> >>>>> >>> > >> >>>>> >>> Hi, > >> >>>>> >>> > >> >>>>> >>> wouldn't that be in conflict with Apache release policy [1] ? > >> >>>>> >>> > >> >>>>> >>> [1] http://www.apache.org/legal/release-policy.html > >> >>>>> >>> > >> >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold < > amyrv...@google.com> wrote: > >> >>>>> >>>> > >> >>>>> >>>> Great idea. I like the RC candidates to follow as much as > the release artifact process as possible. > >> >>>>> >>>> > >> >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay < > al...@google.com> wrote: > >> >>>>> >>>>> > >> >>>>> >>>>> To clarify my proposal, I am proposing publishing to the > production pypi repository with an rc tag in the version. And in turn allow > users to depend on beam's rc version + all the other regular dependencies > users would have directly from pypi. > >> >>>>> >>>>> > >> >>>>> >>>>> Publishing to test pypi repo would also be helpful if test > pypi repo also mirrors other packages that exist in the production pypi > repository. > >> >>>>> >>>>> > >> >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada < > pabl...@google.com> wrote: > >> >>>>> >>>>>> > >> >>>>> >>>>>> I think this is a great idea. A way of doing it for python > would be by using the test repository for PyPi[1], and that way we would > not have to do an official PyPi release, but still would be able to install > it with pip (by passing an extra flag), and test. > >> >>>>> >>>>>> > >> >>>>> >>>>>> In fact, there are some Beam artifacts already in > there[2]. At some point I looked into this, but couldn't figure out who has > access/the password for it. > >> >>>>> >>>>> > >> >>>>> >>>>> > >> >>>>> >>>>> I also don't know who owns beam package in test pypi repo. > Does anybody know? > >> >>>>> >>>>> > >> >>>>> >>>>>> > >> >>>>> >>>>>> > >> >>>>> >>>>>> In short: +1, and I would suggest using the test PyPi repo > to avoid publishing to the main PyPi repo. > >> >>>>> >>>>>> Best > >> >>>>> >>>>>> -P. > >> >>>>> >>>>>> > >> >>>>> >>>>>> [1] https://test.pypi.org/ > >> >>>>> >>>>>> [2] https://test.pypi.org/project/apache-beam/ > >> >>>>> >>>>>> > >> >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay < > al...@google.com> wrote: > >> >>>>> >>>>>>> > >> >>>>> >>>>>>> Hi all, > >> >>>>> >>>>>>> > >> >>>>> >>>>>>> What do you think about the idea of publishing > pre-release artifacts as part of the RC emails? > >> >>>>> >>>>>>> > >> >>>>> >>>>>>> For Python this would translate into publishing the same > artifacts from RC email with a version like "2.X.0rcY" to pypi. I do not > know, but I am guessing we can do a similar thing with Maven central for > Java artifacts as well. > >> >>>>> >>>>>>> > >> >>>>> >>>>>>> Advantages would be: > >> >>>>> >>>>>>> - Allow end users to validate RCs for their own purposes > using the same exact process they will normally use. > >> >>>>> >>>>>>> - Enable early-adaptors to start using RC releases > early on in the release cycle if that is what they would like to do. This > will in turn reduce time pressure on some releases. Especially for cases > like someone needs a release to be finalized for an upcoming event. > >> >>>>> >>>>>>> > >> >>>>> >>>>>>> There will also be disadvantages, some I could think of: > >> >>>>> >>>>>>> - Users could request support for RC artifacts. Hopefully > in the form of feedback for us to improve the release. But it could also be > in the form of folks using RC artifacts for production for a long time. > >> >>>>> >>>>>>> - It will add toil to the current release process, there > will be one more step for each RC. I think for python this will be a small > step but nevertheless it will be additional work. > >> >>>>> >>>>>>> > >> >>>>> >>>>>>> For an example of this, you can take a look at tensorflow > releases. For 1.13 there were 3 pre-releases [1]. > >> >>>>> >>>>>>> > >> >>>>> >>>>>>> Ahmet > >> >>>>> >>>>>>> > >> >>>>> >>>>>>> [1] https://pypi.org/project/tensorflow/#history >
