On Sun, 7 Nov 2021 at 21:21, Jarek Potiuk <ja...@potiuk.com> wrote: > > I (and Craig I guess) were discussing the whole solution - which also > includes cleaning download artifacts.
Purging old releases needs at least two data sources: - the list of files on downloads.apache.org (or the dist.a.o source). This is trivial to obtain. - the list of current versions for each release. This is the most difficult part. Not only does it vary between PMCs, there are different release strategies for different products. For example Commons DBCP has multiple releases for different versions of JDBC, whereas most other Commons components have one active release. Furthermore, the number of active releases for a product may change from time to time. Also products sometimes get retired. I think you are going to need more than just a download page link to enable accurate reporting of unpurged releases. By the way, how are you going to let people know about the changes to the reporter tool? > J. > > On Sun, Nov 7, 2021 at 9:32 PM sebb <seb...@gmail.com> wrote: > > > > AFAICT there is only one case to handle. > > > > The reporter tool lists recent releases. > > Each such release must have an associated download page. > > > > Whether there are multiple download pages or a shared one does not > > matter in this context. > > > > On Sun, 7 Nov 2021 at 20:20, Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > Yep. Those are all valid concerns and I will look into how we could > > > handle those complexities. Hopefully there will be only a handful of > > > various cases :D > > > > > > On Sun, Nov 7, 2021 at 9:17 PM Craig Russell <apache....@gmail.com> wrote: > > > > > > > > Hi Jarek, > > > > > > > > I think that updating the reporter tool to check for a conformant > > > > download(s) page is a great idea. I encourage you to see how to > > > > incorporate that check into the tool. > > > > > > > > I agree with you that sending out mass mailings is probably not going > > > > to result in changes in behavior. Only targeted messages are likely to > > > > work. > > > > > > > > One challenge will be to handle projects with multiple "independent" > > > > sub-projects like db that has jdo and derby with not much in common. > > > > They have completely independent web sites. So the tool will have to be > > > > told about all of the projects. The tool also needs to handle projects > > > > like Airflow which IIUC has multiple sub-projects that share a few > > > > common download pages and scripts. > > > > > > > > Warm regards, > > > > Craig > > > > > > > > > On Nov 4, 2021, at 5:57 AM, Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > > > > > Hello Everyone @devcom, > > > > > > > > > > I am following up after the discussion at users@infra.a.o: > > > > > https://lists.apache.org/thread/skswflvf0q8lbgkmxo18v19s62hkqf3j. > > > > > This is > > > > > not a public mailing list and maybe not everyone here is subscribed > > > > > so let > > > > > me summarise the context and the proposal. BTW. That's my view on the > > > > > state > > > > > of the discussion we had and conclusions/proposals I drew from that. > > > > > > > > > > Context: > > > > > > > > > > We seem to have some of the projects that do not follow all the > > > > > important > > > > > requirements when it comes to publishing GA releases for their > > > > > projects. > > > > > The requirements are pretty firm and documented here: > > > > > https://www.apache.org/legal/release-policy.html#host-GA and in the > > > > > following paragraphs. They boil down to : > > > > > > > > > > a) have an installation page where you can verify releases with: > > > > > * signatures and checksums (links to those should be directly do > > > > > downloads.apache.org > > > > > * packages - links to those should use closer.lua scripts (which are > > > > > then > > > > > automatically mapped to CDN currently - previously to mirroring sites) > > > > > > > > > > b) Have they releases archived appropriately (i.e. only last version > > > > > from > > > > > the development "branch" should remain in downloads@a.o > > > > > > > > > > Proposal: > > > > > > > > > > I think the best way to get this solved permanently (rather than > > > > > sending > > > > > reminder emails to everyone) is to add a check in the reporter tool ( > > > > > https://reporter.apache.org/about.html) to verify those two aspects, > > > > > so > > > > > that they could be verified by projects themselves and the board - > > > > > seems > > > > > that this is a very important aspect that ASF is very concerned > > > > > about, so > > > > > making it part of the board review makes sense and gives the > > > > > opportunity of > > > > > individual flagging when the project does not follow it, It has > > > > > happened > > > > > with our project - Airflow at the last board review, and we fixed all > > > > > that, > > > > > but the idea is that the ASF has an easy to use tool that will add > > > > > such > > > > > checks and flags > > > > > > > > > > Point b) is quite easy it seems. Airflow already has simple Python > > > > > script > > > > > that does that can be adjusted or rewritten to handle multiple > > > > > "lines" of > > > > > development (with some per-project specifics likely) : > > > > > https://github.com/apache/airflow/blob/main/dev/provider_packages/remove_old_releases.py. > > > > > It can also be used to actually help with clean-up (we use it for that > > > > > purpose as we have ~40/50 packages to release every month). > > > > > > > > > > Point a) also should not be difficult as long as each project will > > > > > submit > > > > > their installation page links, we can scrape them and see if the pages > > > > > contain signature/ascii direct links and closer.lua links for > > > > > packages. > > > > > > > > > > Do you think it is a good idea? I just wanted to run it by the > > > > > community to > > > > > see what you think of it, whether you see any potential > > > > > problems/limitations/concerns? > > > > > > > > > > Just one request please - without yet getting into technical details > > > > > and > > > > > discussing HOWs. We will have time for that. > > > > > > > > > > If there is a community (and board :) buy-in, I am happy to > > > > > implement/lead/introduce it. It would dramatically help if others - > > > > > especially people from various projects that might have some special > > > > > cases > > > > > - could help with testing/piloting it. > > > > > > > > > > J. > > > > > > > > Craig L Russell > > > > c...@apache.org > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org > > > > For additional commands, e-mail: dev-h...@community.apache.org > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org > > > For additional commands, e-mail: dev-h...@community.apache.org > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org > > For additional commands, e-mail: dev-h...@community.apache.org > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org > For additional commands, e-mail: dev-h...@community.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@community.apache.org For additional commands, e-mail: dev-h...@community.apache.org
