One low-effort thing you can do if you want to freeze at a particular date rather than use -SNAPSHOT is to just mirror the directory yourself at that time. You don't need fancy infrastructure to treat a directory hierarchy as a maven repo. TBH I would guess most organizations would benefit from having their own internal maven repo anyhow for sharing draft builds of their own if they decompose their software into modules?
Kenn On Mon, Jul 25, 2022 at 11:17 AM Daniel Thevessen <dant...@google.com> wrote: > Thanks everyone for giving this a try, it's much appreciated. I was still > seeing the Jul 14 snapshot on Friday, but it looks like it's gotten cleared > out over the weekend. > We will switch to 2.41.0-SNAPSHOT where necessary, at least the 2.41.0 > release isn't too much further out. > > We'll set up our own Maven repo next time this happens. If anything like > this is on the roadmap though, it'd be really nice if Beam had a defined > process for releasing hotfix versions in the future, that would > have probably made things easier here. > > Best, > Daniel > > > On Mon, Jul 25, 2022 at 8:04 AM Byron Ellis <byronel...@google.com> wrote: > >> Thanks for looking into this Danny >> >> Daniel, it seems like your best bet would be to host snapshots somewhere >> on your own infrastructure? I feel like that's probably the safest and most >> reliable solution in case you need to do this again in the future. >> >> Best, >> B >> >> On Fri, Jul 22, 2022 at 1:35 PM Danny McCormick < >> dannymccorm...@google.com> wrote: >> >>> I reached out to the Apache Infra team in their slack channel and have >>> yet to hear back. Given that it's getting pretty late on a Friday here (I'm >>> located in East US), it is unlikely that I will have a workaround by the >>> time the snapshot is removed by the retention policy; to ensure continued >>> functionality over the weekend and the start of next week, I recommend >>> pinning to a later snapshot. >>> >>> Thanks, >>> Danny >>> >>> On Fri, Jul 22, 2022 at 2:17 PM Daniel Thevessen <dant...@google.com> >>> wrote: >>> >>>> I don't know if this helps, but I found one directory that has retained >>>> an old build from 2017 even, the one for 0.7.0-SNAPSHOT >>>> <https://repository.apache.org/content/groups/snapshots/org/apache/beam/beam-sdks-java-io-google-cloud-platform/0.7.0-SNAPSHOT/>. >>>> Every other snapshots directory is empty. >>>> Is it possible they don't get deleted if they're in use? The sonatype >>>> documentation you linked mentions a "last downloaded" criterion, I'm not >>>> sure if it is enabled though. >>>> I haven't been able to find a matching cleanup script in the Beam repo >>>> or a config file in the snapshots directory, it must be configured >>>> somewhere else. >>>> >>>> If we could somehow get (1) or (2) to work that would be extremely >>>> helpful, changing the dependency back to 2.41.0-SNAPSHOT comes with risks >>>> and difficulties for us. >>>> >>>> >>>> >>>> On Fri, Jul 22, 2022 at 8:10 AM Danny McCormick < >>>> dannymccorm...@google.com> wrote: >>>> >>>>> > I think you could change the TTL on the Jenkins side (That sound >>>>> right to you Danny?) but I'm not sure we could preserve a specific >>>>> snapshot >>>>> without keeping all of them... >>>>> >>>>> I'm admittedly not particularly well versed in this area of our infra, >>>>> but after some digging I don't see a way to do this. I'm still piecing >>>>> together how our release infra works, but at a high level we use the Maven >>>>> publish plugin [1] to publish >>>>> snapshots to the ASF managed Nexus snapshots repo (which is shared >>>>> across projects). As best I can tell, there isn't a way to configure >>>>> retention from the repo side, and it is likely managed by a repo-wide >>>>> cleanup policy [3]. I don't have the right permission set to actually view >>>>> cleanup policies, so I can't say that with 100% confidence, but I do see >>>>> that other projects seem to be limited to a smallish number of snapshot >>>>> versions as well [4] [5] [6] which points to a repo-wide policy, and I >>>>> think that's a common way to handle things like this. >>>>> >>>>> If someone with more experience with our releases wants to chime in >>>>> and confirm that understanding or say I'm wrong, I'd appreciate it. Or, if >>>>> I'm right and someone actually has access to modify the cleanup policy >>>>> (maybe @Kenneth Knowles <k...@google.com>) and can just change it, >>>>> that would be awesome! >>>>> >>>>> Options for moving forward: >>>>> 1) If someone has permissions, change the retention policy for the >>>>> time being. >>>>> 2) Engage infra and see if (a) they can confirm the setup is indeed >>>>> what I described and (b) if they can help temporarily change the retention >>>>> policy for Beam. >>>>> 3) Daniel, you could change how you're consuming the artifacts. The >>>>> shortest term version of this (which would buy more time to figure out a >>>>> solution) is just updating to a more recent snapshot, but you could also >>>>> just pin to the latest snapshot version (obviously has its risks), or >>>>> mirror the dependencies into your own maven repo. You could also just pin >>>>> to 2.40 if this is a regression. >>>>> >>>>> Obviously any solution that isn't pinning to 2.40 is operating in an >>>>> unsafe mode since we haven't performed the usual level of validation that >>>>> accompanies a release. >>>>> >>>>> Thanks, >>>>> Danny >>>>> >>>>> [1] https://docs.gradle.org/current/userguide/publishing_maven.html >>>>> [2] https://infra.apache.org/publishing-maven-artifacts.html >>>>> [3] >>>>> https://help.sonatype.com/repomanager3/nexus-repository-administration/repository-management/cleanup-policies >>>>> [4] >>>>> https://repository.apache.org/content/groups/snapshots/org/apache/avro/avro-compiler/1.12.0-SNAPSHOT/ >>>>> [5] >>>>> https://repository.apache.org/content/groups/snapshots/org/apache/flink/flink-architecture-tests-base/1.15-SNAPSHOT/ >>>>> [6] >>>>> https://repository.apache.org/content/groups/snapshots/org/apache/iotdb/client-cpp-example/0.14.0-SNAPSHOT/ >>>>> >>>>> On Thu, Jul 21, 2022 at 8:54 PM Evan Galpin <egal...@apache.org> >>>>> wrote: >>>>> >>>>>> Admittedly this is potentially self-serving, but I feel there could >>>>>> be mutual benefit. >>>>>> >>>>>> I have a similar situation where I want to use pre-release version of >>>>>> beam-sdks-java-io-google-cloud-platform. Though I’ve been having >>>>>> trouble doing so, a possible alternative solution to using the nightly >>>>>> snapshots might be building beam-sdks-java-io-google-cloud-platform >>>>>> from source and including the resulting jar as part of the pipeline >>>>>> deployment. I’ve successfully done this for direct runner, but not >>>>>> Dataflow >>>>>> runner. >>>>>> >>>>>> Perhaps some others on the thread might be able to shed light on this >>>>>> technique (only if applicable to solving the original problem, as I don’t >>>>>> intend to thread-hijack). >>>>>> >>>>>> - Evan >>>>>> >>>>>> On Thu, Jul 21, 2022 at 19:45 Byron Ellis via dev < >>>>>> dev@beam.apache.org> wrote: >>>>>> >>>>>>> I think you could change the TTL on the Jenkins side (That sound >>>>>>> right to you Danny?) but I'm not sure we could preserve a specific >>>>>>> snapshot >>>>>>> without keeping all of them... >>>>>>> >>>>>>> On Thu, Jul 21, 2022 at 4:16 PM Ahmet Altay <al...@google.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Thank you for the email Daniel. >>>>>>>> >>>>>>>> Adding people who could help: @Kenneth Knowles <k...@google.com> @Danny >>>>>>>> McCormick <dannymccorm...@google.com> @Chamikara Jayalath >>>>>>>> <chamik...@google.com> @John Casey <johnjca...@google.com> @Byron >>>>>>>> Ellis <byronel...@google.com> >>>>>>>> >>>>>>>> On Thu, Jul 21, 2022, 4:14 PM Daniel Thevessen via dev < >>>>>>>> dev@beam.apache.org> wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> The Java Firestore connector had a bug >>>>>>>>> <https://github.com/apache/beam/issues/22089> recently that >>>>>>>>> needed to be fixed. Until the fix can be released as part of 2.41.0, >>>>>>>>> we >>>>>>>>> need one of the daily snapshot builds >>>>>>>>> <https://repository.apache.org/content/groups/snapshots/org/apache/beam/beam-sdks-java-io-google-cloud-platform/2.41.0-SNAPSHOT/> >>>>>>>>> as a safe version to use. This is the one from Jul 15 >>>>>>>>> (2.41.0-20220715.201105-31), which we have checked to be working >>>>>>>>> correctly >>>>>>>>> and some have already switched to. >>>>>>>>> Unfortunately it looks like these builds get cleared out after a >>>>>>>>> while for storage reasons. Would it be possible to extend the TTL on >>>>>>>>> just >>>>>>>>> that build, at least until 2.41.0 is released? I'm guessing this >>>>>>>>> would just >>>>>>>>> be a settings change for whoever owns the Snapshots repository. >>>>>>>>> The change is in beam-sdks-java-io-google-cloud-platform, but I'm >>>>>>>>> fairly certain the other Java packages need to be kept as well for >>>>>>>>> compatibility. >>>>>>>>> >>>>>>>>> This is relatively urgent, it looks like the TTL might be weekly >>>>>>>>> so that build will be deleted on Saturday. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Daniel Thevessen >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>> >>>> -- >>>> >>>> Daniel Thevessen | Site Reliability Engineer >>>> >>>> Firestore SRE >>>> San Francisco, CA, USA | +1 (415) 373-7762 <(415)%20373-7762> >>>> >>> > > -- > > Daniel Thevessen | Site Reliability Engineer > > Firestore SRE > San Francisco, CA, USA | +1 (415) 373-7762 <(415)%20373-7762> >
