This conversation get quite Python centric. Is there a similar need for Java?
On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw <rober...@google.com> wrote: > If we can, by the apache guidelines, post RCs to pypy that is > definitely the way to go. (Note that test.pypi is for developing > against the pypi interface, not for pushing anything real.) The caveat > about naming these with rcN in the version number still applies > (that's how pypi guards them against non-explicit installs). > Related to the caveat, I believe this can be easily scripted or even made part of the travis/wheels pipeline to take the release branch, edit the version string in place to add rc, and build the necessary files. > > The advantage is that a user can do "pip install --pre apache-beam" to > get the latest rc rather than "pip install > https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path"; > > On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada <pabl...@google.com> wrote: > > > > Aw that's interesting! > > > > I think, with these considerations, I am only marginally more inclined > towards publishing to test.pypi. That would make me a +0.9 on publishing > RCs to the main pip repo then. > > > > Thanks for doing the research Ahmet. :) > > Best > > -P > > > > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay <al...@google.com> wrote: > >> > >> I asked to Airflow folks about this. See [1] for the full response and > a link to one of their RC emails. To summarize their position (specifically > for pypi) is: Unless a user does something explicit (such as using a flag, > or explicitly requesting an rc release), pip install will not serve RC > binaries. And that is compatible with RC section of > http://www.apache.org/legal/release-policy.html#release-types > >> > >> Ahmet > >> > >> [1] > https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E > >> > >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay <al...@google.com> wrote: > >>> > >>> The incremental value of publishing python artifacts to a separate > place but not to actual pypi listing will be low. Users can already > download RC artifacts, or even pip install from http location directly. I > think the incremental value will be low, because for a user or a downstream > library to test with Beam RCs using their usual ways will still require > them to get other dependencies from the regular pypi listing. That would > mean they need to change their setup to test with beam rcs, which is the > same state as today. There will be some incremental value of putting them > in more obvious places (e.g. pypi test repository). I would rather not > complicate the release process for doing this. > >>> > >>> > >>> > >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles <k...@apache.org> > wrote: > >>>> > >>>> Pip is also able to be pointed at any raw hosted directory for the > install, right? So we could publish RCs or snapshots somewhere with more > obvious caveats and not interfere with the pypi list of actual releases. > Much like the Java snapshots are stored in a separate opt-in repository. > >>>> > >>>> Kenn > >>>> > >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels <m...@apache.org> > wrote: > >>>>> > >>>>> > wouldn't that be in conflict with Apache release policy [1] ? > >>>>> > [1] http://www.apache.org/legal/release-policy.html > >>>>> > >>>>> Indeed, advertising pre-release artifacts is against ASF rules. For > >>>>> example, Flink was asked to remove a link to the Maven snapshot > >>>>> repository from their download page. > >>>>> > >>>>> However, that does not mean we cannot publish Python artifacts. We > just > >>>>> have to clearly mark them for developers only and not advertise them > >>>>> alongside with the official releases. > >>>>> > >>>>> -Max > >>>>> > >>>>> On 25.04.19 10:23, Robert Bradshaw wrote: > >>>>> > Don't we push java artifacts to maven repositories as part of the > RC > >>>>> > process? And completely unvetted snapshots? (Or is this OK because > >>>>> > they are special opt-in apache-only ones?) > >>>>> > > >>>>> > I am generally in favor of the idea, but would like to avoid > increased > >>>>> > toil on the release manager. > >>>>> > > >>>>> > One potential hitch I see is that current release process updates > the > >>>>> > versions to x.y.z (no RC or other pre-release indicator in the > version > >>>>> > number) whereas pypi (and other systems) typically expect distinct > >>>>> > (recognizable) version numbers for each attempt, and only the > actual > >>>>> > final result has the actual final release version. > >>>>> > > >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay <al...@google.com> > wrote: > >>>>> >> > >>>>> >> I do not know the answer.I believe this will be similar to > sharing the RC artifacts for validation purposes and would not be a formal > release by itself. But I am not an expert and I hope others will share > their opinions. > >>>>> >> > >>>>> >> I quickly searched pypi for apache projects and found at least > airflow [1] and libcloud [2] are publishing rc artifacts to pypi. We can > reach out to those communities and learn about their processes. > >>>>> >> > >>>>> >> Ahmet > >>>>> >> > >>>>> >> [1] https://pypi.org/project/apache-airflow/#history > >>>>> >> [2] https://pypi.org/project/apache-libcloud/#history > >>>>> >> > >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey < > adude3...@gmail.com> wrote: > >>>>> >>> > >>>>> >>> Hi, > >>>>> >>> > >>>>> >>> wouldn't that be in conflict with Apache release policy [1] ? > >>>>> >>> > >>>>> >>> [1] http://www.apache.org/legal/release-policy.html > >>>>> >>> > >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold < > amyrv...@google.com> wrote: > >>>>> >>>> > >>>>> >>>> Great idea. I like the RC candidates to follow as much as the > release artifact process as possible. > >>>>> >>>> > >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay <al...@google.com> > wrote: > >>>>> >>>>> > >>>>> >>>>> To clarify my proposal, I am proposing publishing to the > production pypi repository with an rc tag in the version. And in turn allow > users to depend on beam's rc version + all the other regular dependencies > users would have directly from pypi. > >>>>> >>>>> > >>>>> >>>>> Publishing to test pypi repo would also be helpful if test > pypi repo also mirrors other packages that exist in the production pypi > repository. > >>>>> >>>>> > >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada < > pabl...@google.com> wrote: > >>>>> >>>>>> > >>>>> >>>>>> I think this is a great idea. A way of doing it for python > would be by using the test repository for PyPi[1], and that way we would > not have to do an official PyPi release, but still would be able to install > it with pip (by passing an extra flag), and test. > >>>>> >>>>>> > >>>>> >>>>>> In fact, there are some Beam artifacts already in there[2]. > At some point I looked into this, but couldn't figure out who has > access/the password for it. > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> I also don't know who owns beam package in test pypi repo. > Does anybody know? > >>>>> >>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> In short: +1, and I would suggest using the test PyPi repo to > avoid publishing to the main PyPi repo. > >>>>> >>>>>> Best > >>>>> >>>>>> -P. > >>>>> >>>>>> > >>>>> >>>>>> [1] https://test.pypi.org/ > >>>>> >>>>>> [2] https://test.pypi.org/project/apache-beam/ > >>>>> >>>>>> > >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay <al...@google.com> > wrote: > >>>>> >>>>>>> > >>>>> >>>>>>> Hi all, > >>>>> >>>>>>> > >>>>> >>>>>>> What do you think about the idea of publishing pre-release > artifacts as part of the RC emails? > >>>>> >>>>>>> > >>>>> >>>>>>> For Python this would translate into publishing the same > artifacts from RC email with a version like "2.X.0rcY" to pypi. I do not > know, but I am guessing we can do a similar thing with Maven central for > Java artifacts as well. > >>>>> >>>>>>> > >>>>> >>>>>>> Advantages would be: > >>>>> >>>>>>> - Allow end users to validate RCs for their own purposes > using the same exact process they will normally use. > >>>>> >>>>>>> - Enable early-adaptors to start using RC releases early > on in the release cycle if that is what they would like to do. This will in > turn reduce time pressure on some releases. Especially for cases like > someone needs a release to be finalized for an upcoming event. > >>>>> >>>>>>> > >>>>> >>>>>>> There will also be disadvantages, some I could think of: > >>>>> >>>>>>> - Users could request support for RC artifacts. Hopefully in > the form of feedback for us to improve the release. But it could also be in > the form of folks using RC artifacts for production for a long time. > >>>>> >>>>>>> - It will add toil to the current release process, there > will be one more step for each RC. I think for python this will be a small > step but nevertheless it will be additional work. > >>>>> >>>>>>> > >>>>> >>>>>>> For an example of this, you can take a look at tensorflow > releases. For 1.13 there were 3 pre-releases [1]. > >>>>> >>>>>>> > >>>>> >>>>>>> Ahmet > >>>>> >>>>>>> > >>>>> >>>>>>> [1] https://pypi.org/project/tensorflow/#history >
