Re: upload-pack is slow with lots of refs

2012-10-09 Thread Johannes Sixt
Am 09.10.2012 22:30, schrieb Johannes Sixt: > Am 09.10.2012 08:46, schrieb Shawn Pearce: >> As it turns out we don't really have this problem with git://. Clients >> can bury a v2 request in the extended headers where the host line >> appears today. > > I tried, but it seems that todays git-daemon

Re: upload-pack is slow with lots of refs

2012-10-09 Thread Johannes Sixt
Am 09.10.2012 08:46, schrieb Shawn Pearce: > On Mon, Oct 8, 2012 at 8:05 AM, Johannes Sixt wrote: >> Am 05.10.2012 18:57, schrieb Shawn Pearce: >>> Smart HTTP is not bidirectional. The client can't cut off the server. >> >> Smart HTTP does not need it: you already posted a better solution (I'm >>

Re: upload-pack is slow with lots of refs

2012-10-08 Thread Shawn Pearce
On Mon, Oct 8, 2012 at 8:05 AM, Johannes Sixt wrote: > Am 05.10.2012 18:57, schrieb Shawn Pearce: >> On Thu, Oct 4, 2012 at 11:24 PM, Johannes Sixt wrote: >>> Upload-pack can just start >>> advertising refs in the "v1" way and announce a "v2" capability and listen >>> for response in parallel. A

Re: upload-pack is slow with lots of refs

2012-10-08 Thread Johannes Sixt
Am 05.10.2012 18:57, schrieb Shawn Pearce: > On Thu, Oct 4, 2012 at 11:24 PM, Johannes Sixt wrote: >> Upload-pack can just start >> advertising refs in the "v1" way and announce a "v2" capability and listen >> for response in parallel. A v2 capable client can start sending "wants" or >> some other

Re: upload-pack is slow with lots of refs

2012-10-05 Thread Shawn Pearce
On Thu, Oct 4, 2012 at 11:24 PM, Johannes Sixt wrote: > Am 10/3/2012 21:41, schrieb Shawn Pearce: >> On Wed, Oct 3, 2012 at 11:55 AM, Jeff King wrote: >>> On Wed, Oct 03, 2012 at 11:53:35AM -0700, Junio C Hamano wrote: Jeff King writes: >> Has there been any work on extending the p

Re: upload-pack is slow with lots of refs

2012-10-04 Thread Johannes Sixt
Am 10/3/2012 21:41, schrieb Shawn Pearce: > On Wed, Oct 3, 2012 at 11:55 AM, Jeff King wrote: >> On Wed, Oct 03, 2012 at 11:53:35AM -0700, Junio C Hamano wrote: >>> Jeff King writes: >>> > Has there been any work on extending the protocol so that the client > tells the server what refs it

Re: upload-pack is slow with lots of refs

2012-10-04 Thread Jeff King
On Thu, Oct 04, 2012 at 11:52:13PM +0200, Sascha Cunz wrote: > Would it be possible to use this workflow: > > - Every client connects per default to v1 > > - If server is capable of v2, it sends a flag along with the usual response > (A v1 server will obviously not send that flag) That is mor

Re: upload-pack is slow with lots of refs

2012-10-04 Thread Sascha Cunz
Am Mittwoch, 3. Oktober 2012, 16:13:16 schrieb Jeff King: > On Wed, Oct 03, 2012 at 12:41:38PM -0700, Shawn O. Pearce wrote: > > > Out of curiosity, how are you thinking about triggering such a new > > > behavior in a backwards-compatible way? Invoke git-upload-pack2, and > > > fall back to reconne

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Ævar Arnfjörð Bjarmason
On Thu, Oct 4, 2012 at 1:15 AM, Jeff King wrote: > On Thu, Oct 04, 2012 at 12:15:47AM +0200, Ævar Arnfjörð Bjarmason wrote: > >> I think he was wrong, I tested this on git.git by first creating a lot >> of tags: >> >> parallel --eta "git tag -a -m"{}" test-again-{}" ::: $(git rev-list >> HEA

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Ævar Arnfjörð Bjarmason
On Thu, Oct 4, 2012 at 1:21 AM, Jeff King wrote: > On Thu, Oct 04, 2012 at 12:32:35AM +0200, Ævar Arnfjörð Bjarmason wrote: > >> On Wed, Oct 3, 2012 at 8:03 PM, Jeff King wrote: >> > What version of git are you using? In the past year or so, I've made >> > several tweaks to speed up large number

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Jeff King
On Thu, Oct 04, 2012 at 12:32:35AM +0200, Ævar Arnfjörð Bjarmason wrote: > On Wed, Oct 3, 2012 at 8:03 PM, Jeff King wrote: > > What version of git are you using? In the past year or so, I've made > > several tweaks to speed up large numbers of refs, including: > > > > - cff38a5 (receive-pack:

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Jeff King
On Thu, Oct 04, 2012 at 12:15:47AM +0200, Ævar Arnfjörð Bjarmason wrote: > I think he was wrong, I tested this on git.git by first creating a lot > of tags: > > parallel --eta "git tag -a -m"{}" test-again-{}" ::: $(git rev-list HEAD) > > Then doing: > > git pack-refs --all > git r

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Ævar Arnfjörð Bjarmason
On Wed, Oct 3, 2012 at 8:03 PM, Jeff King wrote: > What version of git are you using? In the past year or so, I've made > several tweaks to speed up large numbers of refs, including: > > - cff38a5 (receive-pack: eliminate duplicate .have refs, v1.7.6); note > that this only helps if they ar

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Ævar Arnfjörð Bjarmason
On Wed, Oct 3, 2012 at 11:20 PM, Jeff King wrote: Thanks for all that info, it's really useful. >> * A co-worker who was working on this today tried it on 1.7.12 and >>claimed that it had the same performance characteristics. > > That's surprising to me. Can you try to verify those numbers?

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Jeff King
On Wed, Oct 03, 2012 at 10:16:56PM +0200, Ævar Arnfjörð Bjarmason wrote: > I can't provide all the details now (not with access to that machine > now), but briefly: > > * The git client/server version is 1.7.8 > > * The repository has around 50k refs, they're "real" refs, almost all >of th

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Ævar Arnfjörð Bjarmason
On Wed, Oct 3, 2012 at 8:03 PM, Jeff King wrote: > On Wed, Oct 03, 2012 at 02:36:00PM +0200, Ævar Arnfjörð Bjarmason wrote: > >> I'm creating a system where a lot of remotes constantly fetch from a >> central repository for deployment purposes, but I've noticed that even >> with a remote.$name.fet

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Jeff King
On Wed, Oct 03, 2012 at 12:41:38PM -0700, Shawn O. Pearce wrote: > > Out of curiosity, how are you thinking about triggering such a new > > behavior in a backwards-compatible way? Invoke git-upload-pack2, and > > fall back to reconnecting to start git-upload-pack if it fails? > > Basically, yes.

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Shawn Pearce
On Wed, Oct 3, 2012 at 11:55 AM, Jeff King wrote: > On Wed, Oct 03, 2012 at 11:53:35AM -0700, Junio C Hamano wrote: >> Jeff King writes: >> >> >> Has there been any work on extending the protocol so that the client >> >> tells the server what refs it's interested in? >> > >> > I don't think so. I

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Junio C Hamano
Ævar Arnfjörð Bjarmason writes: > I'm creating a system where a lot of remotes constantly fetch from a > central repository for deployment purposes, but I've noticed that even > with a remote.$name.fetch configuration to only get certain refs a > "git fetch" will still call git-upload pack which

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Jeff King
On Wed, Oct 03, 2012 at 11:53:35AM -0700, Junio C Hamano wrote: > Jeff King writes: > > >> Has there been any work on extending the protocol so that the client > >> tells the server what refs it's interested in? > > > > I don't think so. It would be hard to do in a backwards-compatible way, > >

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Junio C Hamano
Jeff King writes: >> Has there been any work on extending the protocol so that the client >> tells the server what refs it's interested in? > > I don't think so. It would be hard to do in a backwards-compatible way, > because the advertisement is the first thing the server says, before it > has n

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Jeff King
On Wed, Oct 03, 2012 at 02:36:00PM +0200, Ævar Arnfjörð Bjarmason wrote: > I'm creating a system where a lot of remotes constantly fetch from a > central repository for deployment purposes, but I've noticed that even > with a remote.$name.fetch configuration to only get certain refs a > "git fetch

Re: upload-pack is slow with lots of refs

2012-10-03 Thread Nguyen Thai Ngoc Duy
On Wed, Oct 3, 2012 at 7:36 PM, Ævar Arnfjörð Bjarmason wrote: > I'm creating a system where a lot of remotes constantly fetch from a > central repository for deployment purposes, but I've noticed that even > with a remote.$name.fetch configuration to only get certain refs a > "git fetch" will sti