Re: [announce] yum: parallel downloading

2012-05-22 Thread James Antill
On Tue, 2012-05-22 at 11:24 +0930, Glen Turner wrote: > On 21/05/2012, at 5:12 PM, Zdenek Pavlas wrote: > > > > Three connections limit is used when the above is not available > > (e.g. a baseurl setup with just one mirror). I don't mind lowering > > it to just two, as that should work good enoug

Re: [announce] yum: parallel downloading

2012-05-22 Thread Zdenek Pavlas
> I'd be happy if yum/urlgrabber/libcurl finally used http keepalives. It does, indeed. Parallel downloader tries to use keepalives, too. (we cache and reuse the last idle process instead of closing it) > Last time I looked (and it has been a while), it didn't, so you > always paid the TCP slow s

Re: [announce] yum: parallel downloading

2012-05-22 Thread Matt Domsch
On Mon, May 21, 2012 at 05:33:51AM -0500, Zdenek Pavlas wrote: > > The number of concurrent users is now lower because, well, each of > > them now completes a "yum update" in one third of the time. > > I think Glen's concerns were that the consumed resources > (flow caches, TCP hash entries, socke

Re: [announce] yum: parallel downloading

2012-05-21 Thread Glen Turner
On 21/05/2012, at 5:12 PM, Zdenek Pavlas wrote: > > Three connections limit is used when the above is not available > (e.g. a baseurl setup with just one mirror). I don't mind lowering > it to just two, as that should work good enough in most cases. Yes please. Two is better than three from the

Re: [announce] yum: parallel downloading

2012-05-21 Thread David Malcolm
On Wed, 2012-05-16 at 12:07 -0400, Zdenek Pavlas wrote: > Hi, > > A new yum and urlgrabber packages have just hit Rawhide. These releases > include some new features, including parallel downloading of packages and > metadata, and a new mirror selection code. As we plan to include these > feature

Re: [announce] yum: parallel downloading

2012-05-21 Thread Zdenek Pavlas
> The number of concurrent users is now lower because, well, each of > them now completes a "yum update" in one third of the time. I think Glen's concerns were that the consumed resources (flow caches, TCP hash entries, sockets) may scale faster than the aggregated downloading speed. I am aware o

Re: [announce] yum: parallel downloading

2012-05-21 Thread Zdenek Pavlas
Hi Glen, > Why is the default three connections rather than one? Is a tripling > of the number of connections to a mirror on a Fedora release day > desirable? $ grep maxconnections /var/cache/yum/*/metalink.xml /var/cache/yum/fedora/metalink.xml: /var/cache/yum/updates/metalink.xml:

Re: [announce] yum: parallel downloading

2012-05-20 Thread Reindl Harald
Am 20.05.2012 07:02, schrieb Rob K: > On Sat, May 19, 2012 at 1:34 AM, José Matos wrote: > >> The total number of connections should be the same, as far as I >> understand only the number of connections from a single host will be three. >> Since it should be safe to assume that the downloads ar

Re: [announce] yum: parallel downloading

2012-05-20 Thread Roberto Ragusa
On 05/20/2012 06:10 AM, Glen Turner wrote: > On 19/05/12 01:04, José Matos wrote: > >> The total number of connections should be the same, as far as I >> understand only the number of connections from a single host will be three. > > The risk is the rise in the maximum number of concurrent connec

Re: [announce] yum: parallel downloading

2012-05-19 Thread Rob K
On Sat, May 19, 2012 at 1:34 AM, José Matos wrote: > The total number of connections should be the same, as far as I > understand only the number of connections from a single host will be three. > Since it should be safe to assume that the downloads are independent > events then there should not

Re: [announce] yum: parallel downloading

2012-05-19 Thread Glen Turner
On 19/05/12 01:04, José Matos wrote: > The total number of connections should be the same, as far as I > understand only the number of connections from a single host will be three. The risk is the rise in the maximum number of concurrent connections. A server happily supplying 50,000 concurrent c

Re: [announce] yum: parallel downloading

2012-05-19 Thread Frank Murphy
On 16/05/12 17:36, Antonio Trande wrote: > Both packages are compatible with older versions. Can we use them in Fedora 17 too ? Currently using the rawhide pks in F17 -- Regards, Frank "Jack of all, fubars" -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/

Re: [announce] yum: parallel downloading

2012-05-18 Thread Reindl Harald
Am 18.05.2012 20:31, schrieb John Reiser: > On 05/18/2012 08:59 AM, Reindl Harald wrote: >> why are making the connections to the SAME mirror at all? >> it would make much more sense to download packages >> parallel and each one from a different mirror > > I find that two simultaneous threads to

Re: [announce] yum: parallel downloading

2012-05-18 Thread John Reiser
On 05/18/2012 08:59 AM, Reindl Harald wrote: > why are making the connections to the SAME mirror at all? > it would make much more sense to download packages > parallel and each one from a different mirror I find that two simultaneous threads to the same one mirror gives shortest time to completio

Re: [announce] yum: parallel downloading

2012-05-18 Thread Reindl Harald
Am 18.05.2012 17:34, schrieb José Matos: > On 2012-05-18 15:23, Glen Turner wrote: >> Hi Zdenek, >> >> Why is the default three connections rather than one? Is a tripling of >> the number of connections to a mirror on a Fedora release day desirable? >> Consider that a large mirror site already se

Re: [announce] yum: parallel downloading

2012-05-18 Thread Remi Collet
Le 17/05/2012 18:25, Frank Murphy a écrit : Traceback: https://bugzilla.redhat.com/show_bug.cgi?id=822632 Another taceback with latest python-urlgrabber-3.9.1-13.fc18.noarch File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 2308, in update speed = (k1 * speed + k2 * d

Re: [announce] yum: parallel downloading

2012-05-18 Thread José Matos
On 2012-05-18 15:23, Glen Turner wrote: > Hi Zdenek, > > Why is the default three connections rather than one? Is a tripling of > the number of connections to a mirror on a Fedora release day desirable? > Consider that a large mirror site already sees concurrent connections in > the multiple 10,000

Re: [announce] yum: parallel downloading

2012-05-18 Thread Glen Turner
On 17/05/12 01:37, Zdenek Pavlas wrote: > - mirror limits are honored, too. > > Making many connections to the same mirror usually does not help much, it just > consumes more resources. That's why Yum also uses mirror limits from > metalink.xml. If no such limit is available, at most 3 simultane

Re: [announce] yum: parallel downloading

2012-05-17 Thread elison.ni...@gmail.com
On Wed, May 16, 2012 at 9:37 PM, Zdenek Pavlas wrote: > Hi, > > A new yum and urlgrabber packages have just hit Rawhide.  These releases > include some new features, including parallel downloading of packages and > metadata, and a new mirror selection code.  As we plan to include these > features

Re: [announce] yum: parallel downloading

2012-05-17 Thread Frank Murphy
On 17/05/12 09:33, Zdenek Pavlas wrote: So disable fastestmirror plugin before testing this, would be the way to go? The fastestmirror plugin does some initial mirror sorting. We mostly ignore this, so disabling fastestmirror makes sense but is not strictly necessary. -- Zdeněk Pavlas Traceb

Re: [announce] yum: parallel downloading

2012-05-17 Thread Zdenek Pavlas
> So disable fastestmirror plugin before testing this, > would be the way to go? The fastestmirror plugin does some initial mirror sorting. We mostly ignore this, so disabling fastestmirror makes sense but is not strictly necessary. -- Zdeněk Pavlas -- devel mailing list devel@lists.fedoraproj

Re: [announce] yum: parallel downloading

2012-05-17 Thread Zdenek Pavlas
> > Both packages are compatible with older versions. > > Can we use them in Fedora 17 too ? Yes, I've used it in F14 for some time. -- Zdeněk Pavlas -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel

Re: [announce] yum: parallel downloading

2012-05-16 Thread Frank Murphy
On 16/05/12 17:07, Zdenek Pavlas wrote: - new mirror selection algorithm This should be more accurate than measuring latencies in fastestmirror plugin, but slow mirrors now have to be tried from time to time, and the statistics need some time to build up. So disable fastestmirror plugin be

Re: [announce] yum: parallel downloading

2012-05-16 Thread Antonio Trande
> Both packages are compatible with older versions. Can we use them in Fedora 17 too ? 2012/5/16 Zdenek Pavlas > Hi, > > A new yum and urlgrabber packages have just hit Rawhide. These releases > include some new features, including parallel downloading of packages and > metadata, and a new mir

[announce] yum: parallel downloading

2012-05-16 Thread Zdenek Pavlas
Hi, A new yum and urlgrabber packages have just hit Rawhide. These releases include some new features, including parallel downloading of packages and metadata, and a new mirror selection code. As we plan to include these features in RHEL7, I welcome any feedback or bug reports! python-urlgrabbe