I don't know if this helps, but I found one directory that has retained an old build from 2017 even, the one for 0.7.0-SNAPSHOT <https://repository.apache.org/content/groups/snapshots/org/apache/beam/beam-sdks-java-io-google-cloud-platform/0.7.0-SNAPSHOT/>. Every other snapshots directory is empty. Is it possible they don't get deleted if they're in use? The sonatype documentation you linked mentions a "last downloaded" criterion, I'm not sure if it is enabled though. I haven't been able to find a matching cleanup script in the Beam repo or a config file in the snapshots directory, it must be configured somewhere else.
If we could somehow get (1) or (2) to work that would be extremely helpful, changing the dependency back to 2.41.0-SNAPSHOT comes with risks and difficulties for us. On Fri, Jul 22, 2022 at 8:10 AM Danny McCormick <dannymccorm...@google.com> wrote: > > I think you could change the TTL on the Jenkins side (That sound right > to you Danny?) but I'm not sure we could preserve a specific snapshot > without keeping all of them... > > I'm admittedly not particularly well versed in this area of our infra, but > after some digging I don't see a way to do this. I'm still piecing together > how our release infra works, but at a high level we use the Maven publish > plugin [1] to publish > snapshots to the ASF managed Nexus snapshots repo (which is shared across > projects). As best I can tell, there isn't a way to configure retention > from the repo side, and it is likely managed by a repo-wide cleanup policy > [3]. I don't have the right permission set to actually view cleanup > policies, so I can't say that with 100% confidence, but I do see that > other projects seem to be limited to a smallish number of snapshot versions > as well [4] [5] [6] which points to a repo-wide policy, and I think that's > a common way to handle things like this. > > If someone with more experience with our releases wants to chime in and > confirm that understanding or say I'm wrong, I'd appreciate it. Or, if I'm > right and someone actually has access to modify the cleanup policy (maybe > @Kenneth > Knowles <k...@google.com>) and can just change it, that would be awesome! > > Options for moving forward: > 1) If someone has permissions, change the retention policy for the time > being. > 2) Engage infra and see if (a) they can confirm the setup is indeed what I > described and (b) if they can help temporarily change the retention policy > for Beam. > 3) Daniel, you could change how you're consuming the artifacts. The > shortest term version of this (which would buy more time to figure out a > solution) is just updating to a more recent snapshot, but you could also > just pin to the latest snapshot version (obviously has its risks), or > mirror the dependencies into your own maven repo. You could also just pin > to 2.40 if this is a regression. > > Obviously any solution that isn't pinning to 2.40 is operating in an > unsafe mode since we haven't performed the usual level of validation that > accompanies a release. > > Thanks, > Danny > > [1] https://docs.gradle.org/current/userguide/publishing_maven.html > [2] https://infra.apache.org/publishing-maven-artifacts.html > [3] > https://help.sonatype.com/repomanager3/nexus-repository-administration/repository-management/cleanup-policies > [4] > https://repository.apache.org/content/groups/snapshots/org/apache/avro/avro-compiler/1.12.0-SNAPSHOT/ > [5] > https://repository.apache.org/content/groups/snapshots/org/apache/flink/flink-architecture-tests-base/1.15-SNAPSHOT/ > [6] > https://repository.apache.org/content/groups/snapshots/org/apache/iotdb/client-cpp-example/0.14.0-SNAPSHOT/ > > On Thu, Jul 21, 2022 at 8:54 PM Evan Galpin <egal...@apache.org> wrote: > >> Admittedly this is potentially self-serving, but I feel there could be >> mutual benefit. >> >> I have a similar situation where I want to use pre-release version of >> beam-sdks-java-io-google-cloud-platform. Though I’ve been having trouble >> doing so, a possible alternative solution to using the nightly snapshots >> might be building beam-sdks-java-io-google-cloud-platform from source >> and including the resulting jar as part of the pipeline deployment. I’ve >> successfully done this for direct runner, but not Dataflow runner. >> >> Perhaps some others on the thread might be able to shed light on this >> technique (only if applicable to solving the original problem, as I don’t >> intend to thread-hijack). >> >> - Evan >> >> On Thu, Jul 21, 2022 at 19:45 Byron Ellis via dev <dev@beam.apache.org> >> wrote: >> >>> I think you could change the TTL on the Jenkins side (That sound right >>> to you Danny?) but I'm not sure we could preserve a specific snapshot >>> without keeping all of them... >>> >>> On Thu, Jul 21, 2022 at 4:16 PM Ahmet Altay <al...@google.com> wrote: >>> >>>> Thank you for the email Daniel. >>>> >>>> Adding people who could help: @Kenneth Knowles <k...@google.com> @Danny >>>> McCormick <dannymccorm...@google.com> @Chamikara Jayalath >>>> <chamik...@google.com> @John Casey <johnjca...@google.com> @Byron Ellis >>>> <byronel...@google.com> >>>> >>>> On Thu, Jul 21, 2022, 4:14 PM Daniel Thevessen via dev < >>>> dev@beam.apache.org> wrote: >>>> >>>>> Hi all, >>>>> >>>>> The Java Firestore connector had a bug >>>>> <https://github.com/apache/beam/issues/22089> recently that needed to >>>>> be fixed. Until the fix can be released as part of 2.41.0, we need one of >>>>> the daily snapshot builds >>>>> <https://repository.apache.org/content/groups/snapshots/org/apache/beam/beam-sdks-java-io-google-cloud-platform/2.41.0-SNAPSHOT/> >>>>> as a safe version to use. This is the one from Jul 15 >>>>> (2.41.0-20220715.201105-31), which we have checked to be working correctly >>>>> and some have already switched to. >>>>> Unfortunately it looks like these builds get cleared out after a while >>>>> for storage reasons. Would it be possible to extend the TTL on just that >>>>> build, at least until 2.41.0 is released? I'm guessing this would just be >>>>> a >>>>> settings change for whoever owns the Snapshots repository. >>>>> The change is in beam-sdks-java-io-google-cloud-platform, but I'm >>>>> fairly certain the other Java packages need to be kept as well for >>>>> compatibility. >>>>> >>>>> This is relatively urgent, it looks like the TTL might be weekly so >>>>> that build will be deleted on Saturday. >>>>> >>>>> Thanks, >>>>> Daniel Thevessen >>>>> >>>>> >>>>> -- Daniel Thevessen | Site Reliability Engineer Firestore SRE San Francisco, CA, USA | +1 (415) 373-7762 <(415)%20373-7762>
