Thanks for looking into this Danny Daniel, it seems like your best bet would be to host snapshots somewhere on your own infrastructure? I feel like that's probably the safest and most reliable solution in case you need to do this again in the future.
Best, B On Fri, Jul 22, 2022 at 1:35 PM Danny McCormick <dannymccorm...@google.com> wrote: > I reached out to the Apache Infra team in their slack channel and have yet > to hear back. Given that it's getting pretty late on a Friday here (I'm > located in East US), it is unlikely that I will have a workaround by the > time the snapshot is removed by the retention policy; to ensure continued > functionality over the weekend and the start of next week, I recommend > pinning to a later snapshot. > > Thanks, > Danny > > On Fri, Jul 22, 2022 at 2:17 PM Daniel Thevessen <dant...@google.com> > wrote: > >> I don't know if this helps, but I found one directory that has retained >> an old build from 2017 even, the one for 0.7.0-SNAPSHOT >> <https://repository.apache.org/content/groups/snapshots/org/apache/beam/beam-sdks-java-io-google-cloud-platform/0.7.0-SNAPSHOT/>. >> Every other snapshots directory is empty. >> Is it possible they don't get deleted if they're in use? The sonatype >> documentation you linked mentions a "last downloaded" criterion, I'm not >> sure if it is enabled though. >> I haven't been able to find a matching cleanup script in the Beam repo or >> a config file in the snapshots directory, it must be configured somewhere >> else. >> >> If we could somehow get (1) or (2) to work that would be extremely >> helpful, changing the dependency back to 2.41.0-SNAPSHOT comes with risks >> and difficulties for us. >> >> >> >> On Fri, Jul 22, 2022 at 8:10 AM Danny McCormick < >> dannymccorm...@google.com> wrote: >> >>> > I think you could change the TTL on the Jenkins side (That sound right >>> to you Danny?) but I'm not sure we could preserve a specific snapshot >>> without keeping all of them... >>> >>> I'm admittedly not particularly well versed in this area of our infra, >>> but after some digging I don't see a way to do this. I'm still piecing >>> together how our release infra works, but at a high level we use the Maven >>> publish plugin [1] to publish >>> snapshots to the ASF managed Nexus snapshots repo (which is shared >>> across projects). As best I can tell, there isn't a way to configure >>> retention from the repo side, and it is likely managed by a repo-wide >>> cleanup policy [3]. I don't have the right permission set to actually view >>> cleanup policies, so I can't say that with 100% confidence, but I do see >>> that other projects seem to be limited to a smallish number of snapshot >>> versions as well [4] [5] [6] which points to a repo-wide policy, and I >>> think that's a common way to handle things like this. >>> >>> If someone with more experience with our releases wants to chime in and >>> confirm that understanding or say I'm wrong, I'd appreciate it. Or, if I'm >>> right and someone actually has access to modify the cleanup policy (maybe >>> @Kenneth >>> Knowles <k...@google.com>) and can just change it, that would be awesome! >>> >>> Options for moving forward: >>> 1) If someone has permissions, change the retention policy for the time >>> being. >>> 2) Engage infra and see if (a) they can confirm the setup is indeed what >>> I described and (b) if they can help temporarily change the retention >>> policy for Beam. >>> 3) Daniel, you could change how you're consuming the artifacts. The >>> shortest term version of this (which would buy more time to figure out a >>> solution) is just updating to a more recent snapshot, but you could also >>> just pin to the latest snapshot version (obviously has its risks), or >>> mirror the dependencies into your own maven repo. You could also just pin >>> to 2.40 if this is a regression. >>> >>> Obviously any solution that isn't pinning to 2.40 is operating in an >>> unsafe mode since we haven't performed the usual level of validation that >>> accompanies a release. >>> >>> Thanks, >>> Danny >>> >>> [1] https://docs.gradle.org/current/userguide/publishing_maven.html >>> [2] https://infra.apache.org/publishing-maven-artifacts.html >>> [3] >>> https://help.sonatype.com/repomanager3/nexus-repository-administration/repository-management/cleanup-policies >>> [4] >>> https://repository.apache.org/content/groups/snapshots/org/apache/avro/avro-compiler/1.12.0-SNAPSHOT/ >>> [5] >>> https://repository.apache.org/content/groups/snapshots/org/apache/flink/flink-architecture-tests-base/1.15-SNAPSHOT/ >>> [6] >>> https://repository.apache.org/content/groups/snapshots/org/apache/iotdb/client-cpp-example/0.14.0-SNAPSHOT/ >>> >>> On Thu, Jul 21, 2022 at 8:54 PM Evan Galpin <egal...@apache.org> wrote: >>> >>>> Admittedly this is potentially self-serving, but I feel there could be >>>> mutual benefit. >>>> >>>> I have a similar situation where I want to use pre-release version of >>>> beam-sdks-java-io-google-cloud-platform. Though I’ve been having >>>> trouble doing so, a possible alternative solution to using the nightly >>>> snapshots might be building beam-sdks-java-io-google-cloud-platform >>>> from source and including the resulting jar as part of the pipeline >>>> deployment. I’ve successfully done this for direct runner, but not Dataflow >>>> runner. >>>> >>>> Perhaps some others on the thread might be able to shed light on this >>>> technique (only if applicable to solving the original problem, as I don’t >>>> intend to thread-hijack). >>>> >>>> - Evan >>>> >>>> On Thu, Jul 21, 2022 at 19:45 Byron Ellis via dev <dev@beam.apache.org> >>>> wrote: >>>> >>>>> I think you could change the TTL on the Jenkins side (That sound right >>>>> to you Danny?) but I'm not sure we could preserve a specific snapshot >>>>> without keeping all of them... >>>>> >>>>> On Thu, Jul 21, 2022 at 4:16 PM Ahmet Altay <al...@google.com> wrote: >>>>> >>>>>> Thank you for the email Daniel. >>>>>> >>>>>> Adding people who could help: @Kenneth Knowles <k...@google.com> @Danny >>>>>> McCormick <dannymccorm...@google.com> @Chamikara Jayalath >>>>>> <chamik...@google.com> @John Casey <johnjca...@google.com> @Byron >>>>>> Ellis <byronel...@google.com> >>>>>> >>>>>> On Thu, Jul 21, 2022, 4:14 PM Daniel Thevessen via dev < >>>>>> dev@beam.apache.org> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> The Java Firestore connector had a bug >>>>>>> <https://github.com/apache/beam/issues/22089> recently that needed >>>>>>> to be fixed. Until the fix can be released as part of 2.41.0, we need >>>>>>> one >>>>>>> of the daily snapshot builds >>>>>>> <https://repository.apache.org/content/groups/snapshots/org/apache/beam/beam-sdks-java-io-google-cloud-platform/2.41.0-SNAPSHOT/> >>>>>>> as a safe version to use. This is the one from Jul 15 >>>>>>> (2.41.0-20220715.201105-31), which we have checked to be working >>>>>>> correctly >>>>>>> and some have already switched to. >>>>>>> Unfortunately it looks like these builds get cleared out after a >>>>>>> while for storage reasons. Would it be possible to extend the TTL on >>>>>>> just >>>>>>> that build, at least until 2.41.0 is released? I'm guessing this would >>>>>>> just >>>>>>> be a settings change for whoever owns the Snapshots repository. >>>>>>> The change is in beam-sdks-java-io-google-cloud-platform, but I'm >>>>>>> fairly certain the other Java packages need to be kept as well for >>>>>>> compatibility. >>>>>>> >>>>>>> This is relatively urgent, it looks like the TTL might be weekly so >>>>>>> that build will be deleted on Saturday. >>>>>>> >>>>>>> Thanks, >>>>>>> Daniel Thevessen >>>>>>> >>>>>>> >>>>>>> >> >> -- >> >> Daniel Thevessen | Site Reliability Engineer >> >> Firestore SRE >> San Francisco, CA, USA | +1 (415) 373-7762 <(415)%20373-7762> >> >
