Re: 2.18.0 Regression: packing performance and effectiveness

2018-07-20 Thread Elijah Newren
On Thu, Jul 19, 2018 at 10:28 PM, Jeff King wrote: > On Thu, Jul 19, 2018 at 04:11:01PM -0700, Elijah Newren wrote: > >> Looking at the output from Peff's instrumentation elsewhere in this >> thread, I see a lot of lines like >>mismatched get: 32889efd307c7be376da9e3d45a78305f14ba73a = (, 28)

Re: 2.18.0 Regression: packing performance and effectiveness

2018-07-19 Thread Duy Nguyen
On Fri, Jul 20, 2018 at 7:28 AM Jeff King wrote: > > On Thu, Jul 19, 2018 at 04:11:01PM -0700, Elijah Newren wrote: > > > Looking at the output from Peff's instrumentation elsewhere in this > > thread, I see a lot of lines like > >mismatched get: 32889efd307c7be376da9e3d45a78305f14ba73a = (, 2

Re: 2.18.0 Regression: packing performance and effectiveness

2018-07-19 Thread Jeff King
On Fri, Jul 20, 2018 at 01:28:29AM -0400, Jeff King wrote: > @@ -162,8 +166,10 @@ struct object_entry *packlist_alloc(struct packing_data > *pdata, > > if (!pdata->in_pack_by_idx) > REALLOC_ARRAY(pdata->in_pack, pdata->nr_alloc); > + pack_delta_lo

Re: 2.18.0 Regression: packing performance and effectiveness

2018-07-19 Thread Jeff King
On Thu, Jul 19, 2018 at 04:11:01PM -0700, Elijah Newren wrote: > Looking at the output from Peff's instrumentation elsewhere in this > thread, I see a lot of lines like >mismatched get: 32889efd307c7be376da9e3d45a78305f14ba73a = (, 28) > Does that mean it was reading the array when it wasn't r

Re: 2.18.0 Regression: packing performance and effectiveness

2018-07-19 Thread Elijah Newren
On Thu, Jul 19, 2018 at 11:24 AM, Duy Nguyen wrote: > On Thu, Jul 19, 2018 at 07:31:35PM +0200, Duy Nguyen wrote: >> On Thu, Jul 19, 2018 at 01:23:58PM -0400, Jeff King wrote: >> > On Thu, Jul 19, 2018 at 09:42:00AM -0700, Elijah Newren wrote: >> > >> > > Thanks for the quick turnaround. Unfortun

Re: 2.18.0 Regression: packing performance and effectiveness

2018-07-19 Thread Junio C Hamano
Junio C Hamano writes: > Imagine that you create a delta and set its size (which happens to > fit in the entry) and then create another whose size does not fit in > the entry. How does oe_delta_size() know not to look at > pack->delta_size[] and instead look at e->delta_size_ when it wants > to

Re: 2.18.0 Regression: packing performance and effectiveness

2018-07-19 Thread Junio C Hamano
Duy Nguyen writes: > @@ -330,20 +330,27 @@ static inline void oe_set_size(struct packing_data > *pack, > static inline unsigned long oe_delta_size(struct packing_data *pack, > const struct object_entry *e) > { > - if (e->delta_size_valid) > + if

Re: 2.18.0 Regression: packing performance and effectiveness

2018-07-19 Thread Jeff King
On Thu, Jul 19, 2018 at 08:24:42PM +0200, Duy Nguyen wrote: > > > Looking at that output, my _guess_ is that we somehow end up with a > > > bogus delta_size value and write out a truncated entry. But I couldn't > > > reproduce the issue with smaller test cases. > > > > Could it be a race conditio

Re: 2.18.0 Regression: packing performance and effectiveness

2018-07-19 Thread Duy Nguyen
On Thu, Jul 19, 2018 at 07:31:35PM +0200, Duy Nguyen wrote: > On Thu, Jul 19, 2018 at 01:23:58PM -0400, Jeff King wrote: > > On Thu, Jul 19, 2018 at 09:42:00AM -0700, Elijah Newren wrote: > > > > > Thanks for the quick turnaround. Unfortunately, I have some bad news. > > > With this patch, I get

Re: 2.18.0 Regression: packing performance and effectiveness

2018-07-19 Thread Duy Nguyen
On Thu, Jul 19, 2018 at 01:23:58PM -0400, Jeff King wrote: > On Thu, Jul 19, 2018 at 09:42:00AM -0700, Elijah Newren wrote: > > > Thanks for the quick turnaround. Unfortunately, I have some bad news. > > With this patch, I get the following: > > > > $ /usr/bin/time -f 'MaxRSS:%M Time:%e' git gc

Re: 2.18.0 Regression: packing performance and effectiveness

2018-07-19 Thread Jeff King
On Thu, Jul 19, 2018 at 09:42:00AM -0700, Elijah Newren wrote: > Thanks for the quick turnaround. Unfortunately, I have some bad news. > With this patch, I get the following: > > $ /usr/bin/time -f 'MaxRSS:%M Time:%e' git gc --aggressive > Enumerating objects: 4460703, done. > Counting objects:

Re: 2.18.0 Regression: packing performance and effectiveness

2018-07-19 Thread Jeff King
On Thu, Jul 19, 2018 at 05:16:42PM +0200, Duy Nguyen wrote: > On Thu, Jul 19, 2018 at 07:57:37AM +0200, Duy Nguyen wrote: > > I thought 2M was generous but I was obviously wrong. I'd like to see > > the biggest delta size in this pack and whether it's still reasonable > > to increase OE_DELTA_SIZE

Re: 2.18.0 Regression: packing performance and effectiveness

2018-07-19 Thread Elijah Newren
On Thu, Jul 19, 2018 at 8:16 AM, Duy Nguyen wrote: > On Thu, Jul 19, 2018 at 07:57:37AM +0200, Duy Nguyen wrote: >> I thought 2M was generous but I was obviously wrong. I'd like to see >> the biggest delta size in this pack and whether it's still reasonable >> to increase OE_DELTA_SIZE_BITS. But i

Re: 2.18.0 Regression: packing performance and effectiveness

2018-07-19 Thread Duy Nguyen
On Thu, Jul 19, 2018 at 5:27 PM Elijah Newren wrote: > > On Wed, Jul 18, 2018 at 10:41 PM, Duy Nguyen wrote: > > On Thu, Jul 19, 2018 at 12:51 AM Elijah Newren wrote: > >> > >> I had a user report some poor behavior of 'git gc --aggressive' on a > >> certain repo (which I sadly cannot share). T

Re: 2.18.0 Regression: packing performance and effectiveness

2018-07-19 Thread Elijah Newren
On Wed, Jul 18, 2018 at 10:41 PM, Duy Nguyen wrote: > On Thu, Jul 19, 2018 at 12:51 AM Elijah Newren wrote: >> >> I had a user report some poor behavior of 'git gc --aggressive' on a >> certain repo (which I sadly cannot share). Turns out that on this >> repo, this operation takes about 60% long

Re: 2.18.0 Regression: packing performance and effectiveness

2018-07-19 Thread Duy Nguyen
On Thu, Jul 19, 2018 at 07:57:37AM +0200, Duy Nguyen wrote: > I thought 2M was generous but I was obviously wrong. I'd like to see > the biggest delta size in this pack and whether it's still reasonable > to increase OE_DELTA_SIZE_BITS. But if it's very close to 4GB limit > then we could just store

Re: 2.18.0 Regression: packing performance and effectiveness

2018-07-18 Thread Duy Nguyen
On Thu, Jul 19, 2018 at 7:44 AM Jeff King wrote: > > On Wed, Jul 18, 2018 at 03:51:08PM -0700, Elijah Newren wrote: > > > I had a user report some poor behavior of 'git gc --aggressive' on a > > certain repo (which I sadly cannot share). Turns out that on this > > repo, this operation takes about

Re: 2.18.0 Regression: packing performance and effectiveness

2018-07-18 Thread Jeff King
On Thu, Jul 19, 2018 at 07:41:03AM +0200, Duy Nguyen wrote: > On Thu, Jul 19, 2018 at 12:51 AM Elijah Newren wrote: > > > > I had a user report some poor behavior of 'git gc --aggressive' on a > > certain repo (which I sadly cannot share). Turns out that on this > > repo, this operation takes ab

Re: 2.18.0 Regression: packing performance and effectiveness

2018-07-18 Thread Jeff King
On Wed, Jul 18, 2018 at 03:51:08PM -0700, Elijah Newren wrote: > I had a user report some poor behavior of 'git gc --aggressive' on a > certain repo (which I sadly cannot share). Turns out that on this > repo, this operation takes about 60% longer and produces a pack > roughly twice the expected

Re: 2.18.0 Regression: packing performance and effectiveness

2018-07-18 Thread Duy Nguyen
On Thu, Jul 19, 2018 at 12:51 AM Elijah Newren wrote: > > I had a user report some poor behavior of 'git gc --aggressive' on a > certain repo (which I sadly cannot share). Turns out that on this > repo, this operation takes about 60% longer and produces a pack > roughly twice the expected size.