Aw that's interesting! I think, with these considerations, I am only marginally more inclined towards publishing to test.pypi. That would make me a +0.9 on publishing RCs to the main pip repo then.
Thanks for doing the research Ahmet. :) Best -P On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay <al...@google.com> wrote: > I asked to Airflow folks about this. See [1] for the full response and a > link to one of their RC emails. To summarize their position (specifically > for pypi) is: Unless a user does something explicit (such as using a flag, > or explicitly requesting an rc release), pip install will not serve RC > binaries. And that is compatible with RC section of > http://www.apache.org/legal/release-policy.html#release-types > > Ahmet > > [1] > https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E > > On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay <al...@google.com> wrote: > >> The incremental value of publishing python artifacts to a separate place >> but not to actual pypi listing will be low. Users can already download RC >> artifacts, or even pip install from http location directly. I think the >> incremental value will be low, because for a user or a downstream library >> to test with Beam RCs using their usual ways will still require them to get >> other dependencies from the regular pypi listing. That would mean they need >> to change their setup to test with beam rcs, which is the same state as >> today. There will be some incremental value of putting them in more obvious >> places (e.g. pypi test repository). I would rather not complicate the >> release process for doing this. >> >> >> >> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles <k...@apache.org> wrote: >> >>> Pip is also able to be pointed at any raw hosted directory for the >>> install, right? So we could publish RCs or snapshots somewhere with more >>> obvious caveats and not interfere with the pypi list of actual releases. >>> Much like the Java snapshots are stored in a separate opt-in repository. >>> >>> Kenn >>> >>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels <m...@apache.org> >>> wrote: >>> >>>> > wouldn't that be in conflict with Apache release policy [1] ? >>>> > [1] http://www.apache.org/legal/release-policy.html >>>> >>>> Indeed, advertising pre-release artifacts is against ASF rules. For >>>> example, Flink was asked to remove a link to the Maven snapshot >>>> repository from their download page. >>>> >>>> However, that does not mean we cannot publish Python artifacts. We just >>>> have to clearly mark them for developers only and not advertise them >>>> alongside with the official releases. >>>> >>>> -Max >>>> >>>> On 25.04.19 10:23, Robert Bradshaw wrote: >>>> > Don't we push java artifacts to maven repositories as part of the RC >>>> > process? And completely unvetted snapshots? (Or is this OK because >>>> > they are special opt-in apache-only ones?) >>>> > >>>> > I am generally in favor of the idea, but would like to avoid increased >>>> > toil on the release manager. >>>> > >>>> > One potential hitch I see is that current release process updates the >>>> > versions to x.y.z (no RC or other pre-release indicator in the version >>>> > number) whereas pypi (and other systems) typically expect distinct >>>> > (recognizable) version numbers for each attempt, and only the actual >>>> > final result has the actual final release version. >>>> > >>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay <al...@google.com> wrote: >>>> >> >>>> >> I do not know the answer.I believe this will be similar to sharing >>>> the RC artifacts for validation purposes and would not be a formal release >>>> by itself. But I am not an expert and I hope others will share their >>>> opinions. >>>> >> >>>> >> I quickly searched pypi for apache projects and found at least >>>> airflow [1] and libcloud [2] are publishing rc artifacts to pypi. We can >>>> reach out to those communities and learn about their processes. >>>> >> >>>> >> Ahmet >>>> >> >>>> >> [1] https://pypi.org/project/apache-airflow/#history >>>> >> [2] https://pypi.org/project/apache-libcloud/#history >>>> >> >>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey <adude3...@gmail.com> >>>> wrote: >>>> >>> >>>> >>> Hi, >>>> >>> >>>> >>> wouldn't that be in conflict with Apache release policy [1] ? >>>> >>> >>>> >>> [1] http://www.apache.org/legal/release-policy.html >>>> >>> >>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold <amyrv...@google.com> >>>> wrote: >>>> >>>> >>>> >>>> Great idea. I like the RC candidates to follow as much as the >>>> release artifact process as possible. >>>> >>>> >>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay <al...@google.com> >>>> wrote: >>>> >>>>> >>>> >>>>> To clarify my proposal, I am proposing publishing to the >>>> production pypi repository with an rc tag in the version. And in turn allow >>>> users to depend on beam's rc version + all the other regular dependencies >>>> users would have directly from pypi. >>>> >>>>> >>>> >>>>> Publishing to test pypi repo would also be helpful if test pypi >>>> repo also mirrors other packages that exist in the production pypi >>>> repository. >>>> >>>>> >>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada <pabl...@google.com> >>>> wrote: >>>> >>>>>> >>>> >>>>>> I think this is a great idea. A way of doing it for python would >>>> be by using the test repository for PyPi[1], and that way we would not have >>>> to do an official PyPi release, but still would be able to install it with >>>> pip (by passing an extra flag), and test. >>>> >>>>>> >>>> >>>>>> In fact, there are some Beam artifacts already in there[2]. At >>>> some point I looked into this, but couldn't figure out who has access/the >>>> password for it. >>>> >>>>> >>>> >>>>> >>>> >>>>> I also don't know who owns beam package in test pypi repo. Does >>>> anybody know? >>>> >>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> In short: +1, and I would suggest using the test PyPi repo to >>>> avoid publishing to the main PyPi repo. >>>> >>>>>> Best >>>> >>>>>> -P. >>>> >>>>>> >>>> >>>>>> [1] https://test.pypi.org/ >>>> >>>>>> [2] https://test.pypi.org/project/apache-beam/ >>>> >>>>>> >>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay <al...@google.com> >>>> wrote: >>>> >>>>>>> >>>> >>>>>>> Hi all, >>>> >>>>>>> >>>> >>>>>>> What do you think about the idea of publishing pre-release >>>> artifacts as part of the RC emails? >>>> >>>>>>> >>>> >>>>>>> For Python this would translate into publishing the same >>>> artifacts from RC email with a version like "2.X.0rcY" to pypi. I do not >>>> know, but I am guessing we can do a similar thing with Maven central for >>>> Java artifacts as well. >>>> >>>>>>> >>>> >>>>>>> Advantages would be: >>>> >>>>>>> - Allow end users to validate RCs for their own purposes using >>>> the same exact process they will normally use. >>>> >>>>>>> - Enable early-adaptors to start using RC releases early on >>>> in the release cycle if that is what they would like to do. This will in >>>> turn reduce time pressure on some releases. Especially for cases like >>>> someone needs a release to be finalized for an upcoming event. >>>> >>>>>>> >>>> >>>>>>> There will also be disadvantages, some I could think of: >>>> >>>>>>> - Users could request support for RC artifacts. Hopefully in >>>> the form of feedback for us to improve the release. But it could also be in >>>> the form of folks using RC artifacts for production for a long time. >>>> >>>>>>> - It will add toil to the current release process, there will >>>> be one more step for each RC. I think for python this will be a small step >>>> but nevertheless it will be additional work. >>>> >>>>>>> >>>> >>>>>>> For an example of this, you can take a look at tensorflow >>>> releases. For 1.13 there were 3 pre-releases [1]. >>>> >>>>>>> >>>> >>>>>>> Ahmet >>>> >>>>>>> >>>> >>>>>>> [1] https://pypi.org/project/tensorflow/#history >>>> >>>
