On Sat, Aug 15, 2009 at 02:10:16AM -0400, Daniel Richard G. wrote: > ----8<---- > Hit http://midnight jaunty-security Release.gpg > Ign http://midnight jaunty-security/main Translation-en_US > Err http://midnight jaunty-security/restricted Translation-en_US > Error reading from server - read (104 Connection reset by peer) > Ign http://midnight jaunty-security/universe Translation-en_US > Err http://midnight jaunty-security/multiverse Translation-en_US > Error reading from server. Remote end closed connection > Hit http://midnight jaunty Release.gpg > Ign http://midnight jaunty/free Translation-en_US > ---->8---- > > Whoops! Spoke too soon. There appears to be a completely separate > failure mode that also leads to "Connection reset by peer" errors for > nonexistent Translation* files. This one has nothing to do with > pipelining (setting Pipeline-Depth to "0" doesn't help), and it looks a > lot hairier to figure out, so I'll post my observations so far: > > 1. It's very easy to determine if this problem is affecting you: you get > a big fat warning in your error.log file, even with debugging > switched off: > > Sat Aug 15 01:34:17 2009|info [10743]: ALARM! > /srv/apt-cacher/packages/ubuntu-security_dists_jaunty-security_main_i18n_Translation-en_US.bz2 > file is larger than expected (343). Renaming to > /srv/apt-cacher/packages/ubuntu-security_dists_jaunty-security_main_i18n_Translation-en_US.bz2.corrupted. > > 2. The expected size in the above message is 343, as returned by the > upstream server in the Content-Length: field. This reflects the size > of the 404 error page. > > 3. The actual (unexpected) size here is 502 bytes, which reflects the > combined size of the HTTP header AND body. apt-cacher is reading from > a file that it expects to contain only the 404 body, but for some > reason it has both header and body. > > (Incidentally, while apt-cacher claims to be reading the 404-body > file from $cfg->{cache_dir}/packages/, I've never managed to actually > *see* such a file in there. Is this deleted immediately after being > sent to the client or something?) > > 4. This size anomaly only occurs when you attempt to download a > (nonexistent) Translation* file from a path_map location that has > *multiple servers behind it*. (Note that because the Translation* > file always returns 404, apt-cacher always cycles through each > server, searching for the file in vain.) I had only one server > behind my apt-cacher's /ubuntu path, but two behind /ubuntu- > security, and I was only getting errors from Translation-en_US files > in /ubuntu-security. > > 5. The server-failover in itself appears to be triggering the improper > download of the 404 body file as described in #3. If I edit my > path_map to have only one server behind /ubuntu-security, the problem > goes away. > > 6. Server failover is handled in the libcurl-related code, which is > still a rat's nest to me :-(
Yes, it is complex. The issue was to use a single process for all requests so that DNS lookups were cached. Many mirrors use round robin DNS for load balancing and if you just do multiple single requests you can get files from different servers which are not quite in sync. The code becomes a bit convoluted as it has to separate libcurl errors, client errors and server errors. All communication with the libcurl process goes through a unix socket. The process exits itself after being idle for 2 minutes. Mark -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
