Hi, actually forwarding this to the bug.
And adding a small note that since August we now have buildinfo.debian.net, so maybe for a start it would be sufficient if dak would submit these .buildinfo files via curl/https to buildinfo.d.n!?! ----- Forwarded message from Ximin Luo <infini...@debian.org> ----- Date: Wed, 24 Aug 2016 13:16:00 +0000 From: Ximin Luo <infini...@debian.org> To: ftpmas...@debian.org Cc: Reproducible Builds discussion list <reproducible-bui...@lists.alioth.debian.org> Subject: [Reproducible-builds] Clarification regarding FTP resource constraints for buildinfo files Reply-To: Reproducible Builds discussion list <reproducible-bui...@lists.alioth.debian.org> Hi, I'm emailing to follow-up regarding #763822. I know we have not yet come up with a concrete proposal on that, and that is largely because we were waiting for comments regarding the resource constraints of ftp-master and mirrors. There is broad understanding across the R-B team that you'd prefer a design that does not involve "lots of small files", but there's a lot of breadth in this statement, and none of us know the precise details involved. Originally Lunar proposed a design with 1 large file, but there are issues with this as well, such as the performance of updates. Here are our current main requirements as stated by dkg in message #10, and I confirm they're still accurate as of today: 1. We want an archive user to be able to find and fetch all .buildinfo files that produced a given binary package 2. We want the eventual possibility of multiple .buildinfo files per <srcpkg,version,arch> 3. We understsand that mirror operators don't like small files because rsync gets fussy with them. 4. We want both buildds and debian developers to be able to upload .buildinfo files. (4) by itself is easy; people have already written code to allow dak to accept such files and discard them. So we need to figure out how to reconcile (1,2,3). For this, it would be good if you could tell me in more detail what the restriction (3) consists of. We would never be uploading 10,000k buildinfo files at once, but Mattia tells me that 1k might happen during medium binNMU transitions, growing up to 4k for large transitions (but this would be over several days, i.e. split across multiple runs of dinstall). Each buildinfo file is about 5.4k (median), with 7.7k as the 75% percentile, though the largest is 148k. [1] There is also the distinction between uploading vs mirroring. Just because we might upload 1k files over a short time, does not mean that we have to transfer these to mirrors as 1k files. We could tar some of them up and compress them. So could you clarify some details regarding upload resource limits, as well as mirroring resource limits? For example, is one extra file per source-package OK or "too much"? Or one extra file per binary upload? How about one extra file-update, of the same file, per binary upload? (I assume that rsync means we are free to update any files that we store in pool/, if we need to?) More clarifications to the above, regarding what we *don't* need: N1. It's not essential to store 1 uploaded-buildinfo-file per file-in-the-archive, as long as we can still do (1). N2. We don't care particularly about being able to get *a specific buildinfo-file*, as long as we can still do (1). N3. It's OK to over-satisfy (1) with extra irrelevant data, then the user can just filter this out locally. We have more ideas, but I think it's best to keep this email short for now. Also I don't know what is feasible until I hear more details about the constraints, and it would be pointless to skip further ahead to potentially unfeasible things. X [1] (use wget, too big for browser) https://tests.reproducible-builds.org/debian/buildinfo/unstable/amd64/?C=S;O=A -- GPG: ed25519/56034877E1F87C35 GPG: rsa4096/1318EFAC5FBBDBCE https://github.com/infinity0/pubkeys.git _______________________________________________ Reproducible-builds mailing list reproducible-bui...@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds ----- End forwarded message ----- -- cheers, Holger
signature.asc
Description: Digital signature
