On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay <al...@google.com> wrote: > > This conversation get quite Python centric. Is there a similar need for Java?
I think Java is already covered. Go is a different story (but the even versioning and releasing is being worked out). > On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw <rober...@google.com> wrote: >> >> If we can, by the apache guidelines, post RCs to pypy that is >> definitely the way to go. (Note that test.pypi is for developing >> against the pypi interface, not for pushing anything real.) The caveat >> about naming these with rcN in the version number still applies >> (that's how pypi guards them against non-explicit installs). > > Related to the caveat, I believe this can be easily scripted or even made > part of the travis/wheels pipeline to take the release branch, edit the > version string in place to add rc, and build the necessary files. Yes. But the resulting artifacts would have to be rebuilt (and re-signed) without the version edit for the actual release. (Well, we could possibly edit the artifacts rather than rebuild them.) And pushing un-edited ones early would be really bad. (It's the classic tension of whether a pre-release should be marked internally or externally, re-publishing a new set of bits for the actual release or re-using version numbers for different sets of bits. Pypi does one, apache does another...) >> The advantage is that a user can do "pip install --pre apache-beam" to >> get the latest rc rather than "pip install >> https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path"; >> >> On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada <pabl...@google.com> wrote: >> > >> > Aw that's interesting! >> > >> > I think, with these considerations, I am only marginally more inclined >> > towards publishing to test.pypi. That would make me a +0.9 on publishing >> > RCs to the main pip repo then. >> > >> > Thanks for doing the research Ahmet. :) >> > Best >> > -P >> > >> > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay <al...@google.com> wrote: >> >> >> >> I asked to Airflow folks about this. See [1] for the full response and a >> >> link to one of their RC emails. To summarize their position (specifically >> >> for pypi) is: Unless a user does something explicit (such as using a >> >> flag, or explicitly requesting an rc release), pip install will not serve >> >> RC binaries. And that is compatible with RC section of >> >> http://www.apache.org/legal/release-policy.html#release-types >> >> >> >> Ahmet >> >> >> >> [1] >> >> https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E >> >> >> >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay <al...@google.com> wrote: >> >>> >> >>> The incremental value of publishing python artifacts to a separate place >> >>> but not to actual pypi listing will be low. Users can already download >> >>> RC artifacts, or even pip install from http location directly. I think >> >>> the incremental value will be low, because for a user or a downstream >> >>> library to test with Beam RCs using their usual ways will still require >> >>> them to get other dependencies from the regular pypi listing. That would >> >>> mean they need to change their setup to test with beam rcs, which is the >> >>> same state as today. There will be some incremental value of putting >> >>> them in more obvious places (e.g. pypi test repository). I would rather >> >>> not complicate the release process for doing this. >> >>> >> >>> >> >>> >> >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles <k...@apache.org> wrote: >> >>>> >> >>>> Pip is also able to be pointed at any raw hosted directory for the >> >>>> install, right? So we could publish RCs or snapshots somewhere with >> >>>> more obvious caveats and not interfere with the pypi list of actual >> >>>> releases. Much like the Java snapshots are stored in a separate opt-in >> >>>> repository. >> >>>> >> >>>> Kenn >> >>>> >> >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels <m...@apache.org> >> >>>> wrote: >> >>>>> >> >>>>> > wouldn't that be in conflict with Apache release policy [1] ? >> >>>>> > [1] http://www.apache.org/legal/release-policy.html >> >>>>> >> >>>>> Indeed, advertising pre-release artifacts is against ASF rules. For >> >>>>> example, Flink was asked to remove a link to the Maven snapshot >> >>>>> repository from their download page. >> >>>>> >> >>>>> However, that does not mean we cannot publish Python artifacts. We just >> >>>>> have to clearly mark them for developers only and not advertise them >> >>>>> alongside with the official releases. >> >>>>> >> >>>>> -Max >> >>>>> >> >>>>> On 25.04.19 10:23, Robert Bradshaw wrote: >> >>>>> > Don't we push java artifacts to maven repositories as part of the RC >> >>>>> > process? And completely unvetted snapshots? (Or is this OK because >> >>>>> > they are special opt-in apache-only ones?) >> >>>>> > >> >>>>> > I am generally in favor of the idea, but would like to avoid >> >>>>> > increased >> >>>>> > toil on the release manager. >> >>>>> > >> >>>>> > One potential hitch I see is that current release process updates the >> >>>>> > versions to x.y.z (no RC or other pre-release indicator in the >> >>>>> > version >> >>>>> > number) whereas pypi (and other systems) typically expect distinct >> >>>>> > (recognizable) version numbers for each attempt, and only the actual >> >>>>> > final result has the actual final release version. >> >>>>> > >> >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay <al...@google.com> wrote: >> >>>>> >> >> >>>>> >> I do not know the answer.I believe this will be similar to sharing >> >>>>> >> the RC artifacts for validation purposes and would not be a formal >> >>>>> >> release by itself. But I am not an expert and I hope others will >> >>>>> >> share their opinions. >> >>>>> >> >> >>>>> >> I quickly searched pypi for apache projects and found at least >> >>>>> >> airflow [1] and libcloud [2] are publishing rc artifacts to pypi. >> >>>>> >> We can reach out to those communities and learn about their >> >>>>> >> processes. >> >>>>> >> >> >>>>> >> Ahmet >> >>>>> >> >> >>>>> >> [1] https://pypi.org/project/apache-airflow/#history >> >>>>> >> [2] https://pypi.org/project/apache-libcloud/#history >> >>>>> >> >> >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey >> >>>>> >> <adude3...@gmail.com> wrote: >> >>>>> >>> >> >>>>> >>> Hi, >> >>>>> >>> >> >>>>> >>> wouldn't that be in conflict with Apache release policy [1] ? >> >>>>> >>> >> >>>>> >>> [1] http://www.apache.org/legal/release-policy.html >> >>>>> >>> >> >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold <amyrv...@google.com> >> >>>>> >>> wrote: >> >>>>> >>>> >> >>>>> >>>> Great idea. I like the RC candidates to follow as much as the >> >>>>> >>>> release artifact process as possible. >> >>>>> >>>> >> >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay <al...@google.com> >> >>>>> >>>> wrote: >> >>>>> >>>>> >> >>>>> >>>>> To clarify my proposal, I am proposing publishing to the >> >>>>> >>>>> production pypi repository with an rc tag in the version. And in >> >>>>> >>>>> turn allow users to depend on beam's rc version + all the other >> >>>>> >>>>> regular dependencies users would have directly from pypi. >> >>>>> >>>>> >> >>>>> >>>>> Publishing to test pypi repo would also be helpful if test pypi >> >>>>> >>>>> repo also mirrors other packages that exist in the production >> >>>>> >>>>> pypi repository. >> >>>>> >>>>> >> >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada >> >>>>> >>>>> <pabl...@google.com> wrote: >> >>>>> >>>>>> >> >>>>> >>>>>> I think this is a great idea. A way of doing it for python >> >>>>> >>>>>> would be by using the test repository for PyPi[1], and that way >> >>>>> >>>>>> we would not have to do an official PyPi release, but still >> >>>>> >>>>>> would be able to install it with pip (by passing an extra >> >>>>> >>>>>> flag), and test. >> >>>>> >>>>>> >> >>>>> >>>>>> In fact, there are some Beam artifacts already in there[2]. At >> >>>>> >>>>>> some point I looked into this, but couldn't figure out who has >> >>>>> >>>>>> access/the password for it. >> >>>>> >>>>> >> >>>>> >>>>> >> >>>>> >>>>> I also don't know who owns beam package in test pypi repo. Does >> >>>>> >>>>> anybody know? >> >>>>> >>>>> >> >>>>> >>>>>> >> >>>>> >>>>>> >> >>>>> >>>>>> In short: +1, and I would suggest using the test PyPi repo to >> >>>>> >>>>>> avoid publishing to the main PyPi repo. >> >>>>> >>>>>> Best >> >>>>> >>>>>> -P. >> >>>>> >>>>>> >> >>>>> >>>>>> [1] https://test.pypi.org/ >> >>>>> >>>>>> [2] https://test.pypi.org/project/apache-beam/ >> >>>>> >>>>>> >> >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay <al...@google.com> >> >>>>> >>>>>> wrote: >> >>>>> >>>>>>> >> >>>>> >>>>>>> Hi all, >> >>>>> >>>>>>> >> >>>>> >>>>>>> What do you think about the idea of publishing pre-release >> >>>>> >>>>>>> artifacts as part of the RC emails? >> >>>>> >>>>>>> >> >>>>> >>>>>>> For Python this would translate into publishing the same >> >>>>> >>>>>>> artifacts from RC email with a version like "2.X.0rcY" to >> >>>>> >>>>>>> pypi. I do not know, but I am guessing we can do a similar >> >>>>> >>>>>>> thing with Maven central for Java artifacts as well. >> >>>>> >>>>>>> >> >>>>> >>>>>>> Advantages would be: >> >>>>> >>>>>>> - Allow end users to validate RCs for their own purposes using >> >>>>> >>>>>>> the same exact process they will normally use. >> >>>>> >>>>>>> - Enable early-adaptors to start using RC releases early on >> >>>>> >>>>>>> in the release cycle if that is what they would like to do. >> >>>>> >>>>>>> This will in turn reduce time pressure on some releases. >> >>>>> >>>>>>> Especially for cases like someone needs a release to be >> >>>>> >>>>>>> finalized for an upcoming event. >> >>>>> >>>>>>> >> >>>>> >>>>>>> There will also be disadvantages, some I could think of: >> >>>>> >>>>>>> - Users could request support for RC artifacts. Hopefully in >> >>>>> >>>>>>> the form of feedback for us to improve the release. But it >> >>>>> >>>>>>> could also be in the form of folks using RC artifacts for >> >>>>> >>>>>>> production for a long time. >> >>>>> >>>>>>> - It will add toil to the current release process, there will >> >>>>> >>>>>>> be one more step for each RC. I think for python this will be >> >>>>> >>>>>>> a small step but nevertheless it will be additional work. >> >>>>> >>>>>>> >> >>>>> >>>>>>> For an example of this, you can take a look at tensorflow >> >>>>> >>>>>>> releases. For 1.13 there were 3 pre-releases [1]. >> >>>>> >>>>>>> >> >>>>> >>>>>>> Ahmet >> >>>>> >>>>>>> >> >>>>> >>>>>>> [1] https://pypi.org/project/tensorflow/#history
