Re: [PATCH 2/2] index-pack: prefetch missing REF_DELTA bases

2019-05-20 Thread Nicolas Pitre
On Sat, 18 May 2019, Duy Nguyen wrote: > On Fri, May 17, 2019 at 3:55 PM Jeff King wrote: > > > > On Fri, May 17, 2019 at 02:20:42PM +0700, Duy Nguyen wrote: > > > > > On Fri, May 17, 2019 at 12:35 PM Jeff King wrote: > > > > As it turns out, index-pack does not handle these complicated cases at

Re: [PATCH 0/5] handle corruption in patch-delta

2018-08-30 Thread Nicolas Pitre
uption > > Your related trailing-garbage fix. I split this into two in order to > better demonstrate the cases this part covers. > > [5/5]: patch-delta: handle truncated copy parameters > > My fix for the cp_size read. > > I hope you don't mind me hacking up your patches a bit. Thanks again for > your original report and patch. Looks good to me (feels like traveling back in time). Reviewed-by: Nicolas Pitre > > -Peff >

Re: [PATCH v4] Refactor recv_sideband()

2016-07-06 Thread Nicolas Pitre
On Wed, 6 Jul 2016, Junio C Hamano wrote: > Thanks; will apply with a miniscule fix. > > > - >8 > > Subject: sideband.c: small optimization of strbuf usage > > > > Signed-off-by: Nicolas Pitre > > ... > > @@ -97,7 +97,7 @@ int recv_si

Re: [PATCH v4] Refactor recv_sideband()

2016-07-05 Thread Nicolas Pitre
On Wed, 29 Jun 2016, Junio C Hamano wrote: > Nicolas Pitre writes: > > > To make it clearer, here's a patch on top of pu that fixes all the > > issues I think are remaining. All tests pass now. > > Lukas, can you see what is in 'pu' after I push out

Re: [PATCH v4] Refactor recv_sideband()

2016-06-28 Thread Nicolas Pitre
On Tue, 28 Jun 2016, Junio C Hamano wrote: > Junio C Hamano writes: > > > It's just that if you take the latter, then the conditional after > > the loop exits (i.e. the last transmission was an incomplete line) > > cannot be "is outbuf empty?", as your base state is "has PREFIX and > > can never

Re: [PATCH v4] Refactor recv_sideband()

2016-06-28 Thread Nicolas Pitre
On Tue, 28 Jun 2016, Junio C Hamano wrote: > Nicolas Pitre writes: > > >> The basic structure of the code (without the "SQUASH" we discussed) > >> looks like this: > >> > >>strbuf_addf(&outbuf, "%s", PREFIX); > >>

Re: [PATCH v4] Refactor recv_sideband()

2016-06-28 Thread Nicolas Pitre
On Tue, 28 Jun 2016, Junio C Hamano wrote: > Nicolas Pitre writes: > > >> There is something else going on. I cannot quite explain why I am > >> getting this failure from t5401-update-hooks.sh, for example: > >> > >> --- expect 2016-06-28 19

Re: [PATCH v4] Refactor recv_sideband()

2016-06-28 Thread Nicolas Pitre
On Tue, 28 Jun 2016, Junio C Hamano wrote: > Nicolas Pitre writes: > > > Without this, the error and remaining buffer would be reversed as > > mentioned previously. With this, the order is restored, but a newline > > is added to unterminated lines whereas the erro

Re: [PATCH v4] Refactor recv_sideband()

2016-06-28 Thread Nicolas Pitre
On Tue, 28 Jun 2016, Junio C Hamano wrote: > Nicolas Pitre writes: > > >> When we exit the loop because we set retval to a non-zero value, > >> should we still drain the outbuf? > > > > I would think so. Anything that the remote sent before any error shoul

Re: [PATCH v4] Refactor recv_sideband()

2016-06-28 Thread Nicolas Pitre
On Tue, 28 Jun 2016, Junio C Hamano wrote: > And then that made me stare at the patch even more. We still write > some error messages to stderr in the updated code (without my crap > SQUASH) inside "while (!retval)" loop: > > while (retval == 0) { > int band, len; >

Re: [PATCH v2] Refactor recv_sideband()

2016-06-27 Thread Nicolas Pitre
On Mon, 27 Jun 2016, Lukas Fleischer wrote: > On Mon, 27 Jun 2016 at 19:50:13, Junio C Hamano wrote: > > Jeff King writes: > > > > > On Mon, Jun 27, 2016 at 08:54:22AM -0700, Junio C Hamano wrote: > > > > > >> It's just you used xwrite() there that introduced a different issue. > > >> Wouldn't r

Re: [PATCH v3] Refactor recv_sideband()

2016-06-23 Thread Nicolas Pitre
On Thu, 23 Jun 2016, Lukas Fleischer wrote: > On Thu, 23 Jun 2016 at 00:47:39, Nicolas Pitre wrote: > > On Wed, 22 Jun 2016, Nicolas Pitre wrote: > > [...] > > > if (*b) { > > > xwrite(STDE

Re: [PATCH v3] Refactor recv_sideband()

2016-06-22 Thread Nicolas Pitre
On Wed, 22 Jun 2016, Nicolas Pitre wrote: > On Wed, 22 Jun 2016, Lukas Fleischer wrote: > > > Before this patch, we used character buffer manipulations to split > > messages from the sideband at line breaks and insert "remote: " at the > > beginning of e

Re: [PATCH v3] Refactor recv_sideband()

2016-06-22 Thread Nicolas Pitre
utput which is then > printed using a single write() call, such that the atomicity of the > output is preserved. See 9ac13ec (atomic write for sideband remote > messages, 2006-10-11) for details. > > Helped-by: Nicolas Pitre > Signed-off-by: Lukas Fleischer The patch is buggy.

Re: [PATCH] Refactor recv_sideband()

2016-06-14 Thread Nicolas Pitre
On Tue, 14 Jun 2016, Lukas Fleischer wrote: > On Tue, 14 Jun 2016 at 19:55:06, Nicolas Pitre wrote: > > It is not buffered as it writes to stderr. And some C libs do separate > > calls to write() for every string format specifier. So "%s%s%c" may end > > up calli

Re: [PATCH] Refactor recv_sideband()

2016-06-14 Thread Nicolas Pitre
On Tue, 14 Jun 2016, Lukas Fleischer wrote: > Hi Nicolas, > > On Tue, 14 Jun 2016 at 19:09:15, Nicolas Pitre wrote: > > I just looked again at all the contraptions _I_ wrote (not Junio's) for > > a reason why I went to such extremes in making this code co complicated

Re: [PATCH] Refactor recv_sideband()

2016-06-14 Thread Nicolas Pitre
On Mon, 13 Jun 2016, Nicolas Pitre wrote: > On Mon, 13 Jun 2016, Lukas Fleischer wrote: > > > Improve the readability of recv_sideband() significantly by replacing > > fragile buffer manipulations with more sophisticated format strings. > > Also, reorganize the overall c

Re: [PATCH] Refactor recv_sideband()

2016-06-14 Thread Nicolas Pitre
On Tue, 14 Jun 2016, Johannes Schindelin wrote: > Hi Nico, > > On Tue, 14 Jun 2016, Nicolas Pitre wrote: > > > On Tue, 14 Jun 2016, Johannes Schindelin wrote: > > > > > On Mon, 13 Jun 2016, Nicolas Pitre wrote: > > > > >

Re: [PATCH] Refactor recv_sideband()

2016-06-14 Thread Nicolas Pitre
On Tue, 14 Jun 2016, Johannes Schindelin wrote: > Hi, > > On Mon, 13 Jun 2016, Nicolas Pitre wrote: > > > On Mon, 13 Jun 2016, Lukas Fleischer wrote: > > > > > Improve the readability of recv_sideband() significantly by replacing > > > fragile bu

Re: [PATCH] Refactor recv_sideband()

2016-06-13 Thread Nicolas Pitre
ace a custom implementation of strpbrk() with a call > to the standard C library function. > > Signed-off-by: Lukas Fleischer The previous code was a total abomination, even if I happen to know who wrote it. Acked-by: Nicolas Pitre > I had a really hard time reading and understanding thi

Re: [PATCH v2] reduce progress updates in background

2015-04-14 Thread Nicolas Pitre
On Tue, 14 Apr 2015, Luke Mewburn wrote: > Updated patch where is_foreground_fd() is only called in display() > just before the output is to be displayed. Acked-by: Nicolas Pitre > -- To unsubscribe from this list: send the line "unsubscribe git" in the body of

Re: [PATCH] reduce progress updates in background

2015-04-13 Thread Nicolas Pitre
On Tue, 14 Apr 2015, Luke Mewburn wrote: > On Mon, Apr 13, 2015 at 10:11:09AM -0400, Nicolas Pitre wrote: > | What if you suspend the task and push it into the background? Would be > | nice to inhibit progress display in that case, and resume it if the task > | returns to th

Re: [PATCH] reduce progress updates in background

2015-04-13 Thread Nicolas Pitre
On Mon, 13 Apr 2015, Luke Mewburn wrote: > Hi, > > I've noticed that when a long-running git operation that generates > progress output is suspended and converted to a background process, > the terminal still gets spammed with progress updates (to stderr). > > Many years ago I fixed a similar is

Re: Pack v4 again..

2015-02-15 Thread Nicolas Pitre
On Fri, 13 Feb 2015, Duy Nguyen wrote: > After taking 1.5 years "vacation" from pack v4, I plan to do something > about it again. Will post more when I have some patches to discuss. > Only one question for now (forgive me if I asked already, it's been > quite some time) Yeah. I had to re-study m

Re: [PATCH v2 2/2] Document some functions defined in object.c

2014-02-28 Thread Nicolas Pitre
On Fri, 28 Feb 2014, Michael Haggerty wrote: > Signed-off-by: Michael Haggerty Acked-by: Nicolas Pitre > --- > object.c | 29 - > object.h | 7 +++ > 2 files changed, 35 insertions(+), 1 deletion(-) > > diff --git a/object.c b/obj

Re: [PATCH 6/6] Document some functions defined in object.c

2014-02-21 Thread Nicolas Pitre
- /* Assumes power-of-2 hash sizes in grow_object_hash */ > return hash & (n - 1); > } Other than that... Reviewed-by: Nicolas Pitre > > +/* > + * Insert obj into the hash table hash, which has length size (which > + * must be a power of 2). On collisions, sim

Re: [PATCH 5/6] Document a bunch of functions defined in sha1_file.c

2014-02-21 Thread Nicolas Pitre
On Fri, 21 Feb 2014, Michael Haggerty wrote: > Signed-off-by: Michael Haggerty Acked-by: Nicolas Pitre > --- > cache.h | 66 > ++--- > sha1_file.c | 26 +--- > 2 files changed, 78 insertion

Re: [PATCH 3/6] find_pack_entry(): document last_found_pack

2014-02-21 Thread Nicolas Pitre
; > Signed-off-by: Michael Haggerty REviewed-by: Nicolas Pitre > --- > sha1_file.c | 16 > 1 file changed, 12 insertions(+), 4 deletions(-) > > diff --git a/sha1_file.c b/sha1_file.c > index 6e8c05d..0910939 100644 > --- a/sha1_file.c > +++ b/sha1

Re: pack corruption post-mortem

2013-10-20 Thread Nicolas Pitre
On Sun, 20 Oct 2013, Duy Nguyen wrote: > On Sat, Oct 19, 2013 at 9:41 PM, Nicolas Pitre wrote: > > On Sat, 19 Oct 2013, Duy Nguyen wrote: > > The SHA1 used in the name of the pack file is actually the SHA1 checksum > > of the SHA1 table. > > > > The path and iden

Re: pack corruption post-mortem

2013-10-20 Thread Nicolas Pitre
On Sat, 19 Oct 2013, Shawn Pearce wrote: > On Sat, Oct 19, 2013 at 7:41 AM, Nicolas Pitre wrote: > > On Sat, 19 Oct 2013, Duy Nguyen wrote: > > > >> On Wed, Oct 16, 2013 at 3:34 PM, Jeff King wrote: > >> > I was recently presented with a repository with

Re: pack corruption post-mortem

2013-10-19 Thread Nicolas Pitre
On Sat, 19 Oct 2013, Duy Nguyen wrote: > On Wed, Oct 16, 2013 at 3:34 PM, Jeff King wrote: > > I was recently presented with a repository with a corrupted packfile, > > and was asked if the data was recoverable. This post-mortem describes > > the steps I took to investigate and fix the problem. I

Re: pack-object's try_delta fast path for v2 trees?

2013-10-14 Thread Nicolas Pitre
On Mon, 14 Oct 2013, Jeff King wrote: > By the way, I'm sorry I haven't looked more carefully at the packv4 > patches yet. I am excited about it, but I've just got a long queue of > other things (and because it's big and challenging, it's easy to put > off). ;-) While I consider the format prett

Re: pack-object's try_delta fast path for v2 trees?

2013-10-14 Thread Nicolas Pitre
On Mon, 14 Oct 2013, Jeff King wrote: > On Sat, Oct 12, 2013 at 10:42:17AM +0700, Nguyen Thai Ngoc Duy wrote: > > > Just wondering if this has been considered and dropped before. > > Currently we use try_delta() for every object including trees. But > > trees are special. All tree entries must be

Re: pack-object's try_delta fast path for v2 trees?

2013-10-12 Thread Nicolas Pitre
On Sat, 12 Oct 2013, Duy Nguyen wrote: > Hi, > > Just wondering if this has been considered and dropped before. > Currently we use try_delta() for every object including trees. But > trees are special. All tree entries must be unique and sorted. That > helps simplify diff algorithm, as demonstrat

Re: [BAD PATCH 0/9] v4-aware tree walker API

2013-10-12 Thread Nicolas Pitre
On Fri, 11 Oct 2013, Duy Nguyen wrote: > On Fri, Oct 11, 2013 at 07:22:59PM +0700, Duy Nguyen wrote: > > > > Maybe we could make an exception and allow the tree walker to pass > > > > pv4_tree_cache* directly to decode_entries so it does not need to do > > > > the first lookup every time.. > > > >

Re: [BAD PATCH 0/9] v4-aware tree walker API

2013-10-09 Thread Nicolas Pitre
On Wed, 9 Oct 2013, Nguyễn Thái Ngọc Duy wrote: > I know I still have a lot of holes to plug, but this was more > interesting because we could see some encouraging numbers. > Unfortunately the result is disappointing. Maybe I did it in a stupid > way and need to restart with a totally different wa

Re: Local tag killer

2013-09-30 Thread Nicolas Pitre
On Mon, 30 Sep 2013, Marc Branchaud wrote: > On 13-09-30 06:44 PM, Nicolas Pitre wrote: > > On Mon, 30 Sep 2013, Marc Branchaud wrote: > > > > > On 13-09-30 04:08 PM, Nicolas Pitre wrote: > > > > Again, in the cases where there is actually a SHA1 conflict be

Re: Local tag killer

2013-09-30 Thread Nicolas Pitre
On Mon, 30 Sep 2013, Marc Branchaud wrote: > On 13-09-30 04:08 PM, Nicolas Pitre wrote: > > Again, in the cases where there is actually a SHA1 conflict between all > > possible tags that match a tag short-end then listing them and asking the > > user to be more explicit is

Re: Local tag killer

2013-09-30 Thread Nicolas Pitre
On Mon, 30 Sep 2013, Marc Branchaud wrote: > On 13-09-30 11:52 AM, Nicolas Pitre wrote: > > Consider that I have in my Linux kernel tree: > > > > - a remote branch corresponding to Linus' master tree > > > > - multiple remote branches corresponding to Lin

Re: Local tag killer

2013-09-30 Thread Nicolas Pitre
On Mon, 30 Sep 2013, Marc Branchaud wrote: > Why would there be ambiguity warnings? The fetch command shouldn't issue any > warnings, since all the remotes' names get safely tucked away in distinct > namespaces. > > Are we talking about DWIM warnings? Aside from git-describe I don't see why > s

Re: [PATCH 00/10] pack v4 UI support

2013-09-26 Thread Nicolas Pitre
On Thu, 26 Sep 2013, Duy Nguyen wrote: > On Thu, Sep 26, 2013 at 11:51 AM, Nicolas Pitre wrote: > >> Multi-base tree support is not part of "packv4" capability. Let's see > >> if such support comes before the series is merged to master. If so we &

Re: [PATCH 00/10] pack v4 UI support

2013-09-25 Thread Nicolas Pitre
On Thu, 26 Sep 2013, Nguyễn Thái Ngọc Duy wrote: > About the "packv4" capability in git protocol. I considered a more > generic cap "packver=ver[,ver[,ver...]]" that represents all supported > versions, with the order of preference from the first ver (most preferred) > to the last (least preferred

Re: Local tag killer

2013-09-25 Thread Nicolas Pitre
On Tue, 24 Sep 2013, Jeff King wrote: > On Sat, Sep 21, 2013 at 08:42:26AM +0200, Michael Haggerty wrote: > > > I think it would be preferable if "--prune" would *not* affect tags, and > > if there were an extra option like "--prune-tags" that would have to be > > used explicitly to cause tags to

Re: [PATCH 15/17] t5300: avoid testing ofs-delta with --packv4

2013-09-21 Thread Nicolas Pitre
On Sun, 22 Sep 2013, Duy Nguyen wrote: > On Sat, Sep 21, 2013 at 11:46 PM, Nicolas Pitre wrote: > > On Sat, 21 Sep 2013, Nguy­n Thái Ng÷c Duy wrote: > > > >> > >> Signed-off-by: Nguy­n Thái Ng÷c Duy > >> --- > >> t/t5300-pack-object.sh | 39 +

Re: [PATCH 07/17] fixup! index-pack, pack-objects: allow creating .idx v2 with .pack v4

2013-09-21 Thread Nicolas Pitre
On Sat, 21 Sep 2013, Nguyễn Thái Ngọc Duy wrote: > Signed-off-by: Nguyễn Thái Ngọc Duy > --- > sha1_file.c | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/sha1_file.c b/sha1_file.c > index e98eb8b..ef6ecc8 100644 > --- a/sha1_file.c > +++ b/sha1_file.c > @@ -605,7 +6

Re: [PATCH 15/17] t5300: avoid testing ofs-delta with --packv4

2013-09-21 Thread Nicolas Pitre
On Sat, 21 Sep 2013, Nguyễn Thái Ngọc Duy wrote: > > Signed-off-by: Nguyễn Thái Ngọc Duy > --- > t/t5300-pack-object.sh | 39 ++- > 1 file changed, 22 insertions(+), 17 deletions(-) [...] This, in combination with patch 10/17, is making the test suite to tes

Re: [PATCH 00/17] np/pack-v4 updates

2013-09-21 Thread Nicolas Pitre
On Sat, 21 Sep 2013, Nguyễn Thái Ngọc Duy wrote: > This contains many bug fixes or cleanups. Also you can now run the > test suite with v4 by setting GIT_TEST_OPTS=--packv4. The test suite > passes now. pack size limit is not officially not supported with v4. > index-pack also learns to convert ap

Git pack v4: next step, help required

2013-09-18 Thread Nicolas Pitre
I think the pack v4 format and compatibility code is pretty stable now, and probably as optimal as it can reasonably be. @Junio: might be a good idea to refresh your pu branch again. Now the biggest problem to solve is the actual tree "deltification". I don't have the time to spend on this ot

Re: [PATCH np/pack-v4] index-pack: tighten object type check based on pack version

2013-09-18 Thread Nicolas Pitre
On Wed, 18 Sep 2013, Nguyễn Thái Ngọc Duy wrote: > In pack version 4, ref-delta technically could be used to compress any > objects including commits and trees (both canonical and v4). But it > does not make sense to do so. It can only lead to performance > degradation. Catch those packers. > > S

Re: [PATCH 2/4] pack v4: add v4_size to struct delta_base_cache_entry

2013-09-15 Thread Nicolas Pitre
On Sun, 15 Sep 2013, Duy Nguyen wrote: > On Sat, Sep 14, 2013 at 11:22 AM, Nicolas Pitre wrote: > > The cache is currently updated by the caller. The caller may ask for a > > copy of 2 entries from a base object, but that base object may itself > > copy those objects fro

Re: [PATCH 2/4] pack v4: add v4_size to struct delta_base_cache_entry

2013-09-13 Thread Nicolas Pitre
On Fri, 13 Sep 2013, Nicolas Pitre wrote: > On Fri, 13 Sep 2013, Duy Nguyen wrote: > > > On Fri, Sep 13, 2013 at 8:27 PM, Nicolas Pitre wrote: > > > However I can see that, as you say, the same base object is repeatedly > > > referenced. This means that we need t

Re: [PATCH] pack-objects: no crc check when the cached version is used

2013-09-13 Thread Nicolas Pitre
On Sat, 14 Sep 2013, Duy Nguyen wrote: > On Sat, Sep 14, 2013 at 4:26 AM, Thomas Rast wrote: > > I tried the perf script below, but at least for the git repo the only > > thing I can see is noise. > > --stdout does not set do_check_packed_object_crc, you need to run > pack-objects without --stdo

Re: [PATCH 2/4] pack v4: add v4_size to struct delta_base_cache_entry

2013-09-13 Thread Nicolas Pitre
On Fri, 13 Sep 2013, Duy Nguyen wrote: > On Fri, Sep 13, 2013 at 8:27 PM, Nicolas Pitre wrote: > > On Thu, 12 Sep 2013, Nguyễn Thái Ngọc Duy wrote: > > > >> The intention is to store flat v4 trees in delta base cache to avoid > >> repeatedly expanding copy seq

Re: [PATCH 2/4] pack v4: add v4_size to struct delta_base_cache_entry

2013-09-13 Thread Nicolas Pitre
On Thu, 12 Sep 2013, Nguyễn Thái Ngọc Duy wrote: > The intention is to store flat v4 trees in delta base cache to avoid > repeatedly expanding copy sequences in v4 trees. When the user needs > to unpack a v4 tree and the tree is found in the cache, the tree will > be converted back to canonical fo

Re: [PATCH] lookup_object: remove hashtable_index() and optimize hash_obj()

2013-09-12 Thread Nicolas Pitre
On Wed, 11 Sep 2013, Jeff King wrote: > On Tue, Sep 10, 2013 at 06:17:12PM -0400, Nicolas Pitre wrote: > > > Also remove the modulus as this is an expansive operation. > > The size argument is always a power of 2 anyway, so a simple > > mask operation provides the same re

Re: [PATCH 00/21] np/pack-v4 updates

2013-09-12 Thread Nicolas Pitre
On Thu, 12 Sep 2013, Duy Nguyen wrote: > On Wed, Sep 11, 2013 at 11:25 PM, Nicolas Pitre wrote: > > On Wed, 11 Sep 2013, Duy Nguyen wrote: > > > >> Nico, if you have time you may want to look into this. The result v4 > >> pack from pack-objects on git.git

Re: [PATCH v2 15/16] index-pack: use nr_objects_final as sha1_table size

2013-09-12 Thread Nicolas Pitre
On Tue, 10 Sep 2013, Duy Nguyen wrote: > On Mon, Sep 9, 2013 at 10:01 PM, Nicolas Pitre wrote: > > However this means that the progress meter will now be wrong and that's > > terrible ! Users *will* complain that the meter doesn't reach 100% and > > they&

Re: [PATCH 00/21] np/pack-v4 updates

2013-09-11 Thread Nicolas Pitre
On Wed, 11 Sep 2013, Nguyễn Thái Ngọc Duy wrote: > This contains fixups for some of my patches, some of Nico's, adds v4 > support to unpack-objects because the test suite needs it. With these, > when force generating pack v4 unconditionally, the remaining failed > tests are: [...] @junio: I've fo

Re: [PATCH 00/21] np/pack-v4 updates

2013-09-11 Thread Nicolas Pitre
On Wed, 11 Sep 2013, Duy Nguyen wrote: > Nico, if you have time you may want to look into this. The result v4 > pack from pack-objects on git.git for me is 35MB (one branch) while > packv4-create produces 30MB (v2 is 40MB). I don't know why there is > such a big difference in size. I compared. Ide

Re: [PATCH 20/21] show-index: acknowledge that it does not read .idx v3

2013-09-11 Thread Nicolas Pitre
On Wed, 11 Sep 2013, Nguyễn Thái Ngọc Duy wrote: > show-index takes .idx from stdin while v3 requires the .pack. It's > used for testing purposes only. Let those test scripts force .idx v2 > with index-pack. Since I have a patch adding (partial) index v3 support to show-index in my tree, I've dr

Re: [PATCH 19/21] index-pack, pack-objects: allow creating .idx v2 with .pack v4

2013-09-11 Thread Nicolas Pitre
On Wed, 11 Sep 2013, Nguyễn Thái Ngọc Duy wrote: > While .idx v3 is recommended because it's smaller, there is no reason > why .idx v2 can't use with .pack v4. Enable it, at least for the test > suite as some tests need to this kind of information from show-index > and show-index does not support

Re: [PULL REQUEST] initial pack v4 support

2013-09-10 Thread Nicolas Pitre
On Tue, 10 Sep 2013, Junio C Hamano wrote: > > packv4-parse.c: allow tree entry copying from a canonical tree object > > This one needed a small fix-up to make it compile. > > I do not particularly like reusing that "size" variable, but it > seemed to be dead at that point, so... > > pac

[PATCH] lookup_object: remove hashtable_index() and optimize hash_obj()

2013-09-10 Thread Nicolas Pitre
t rev-list --all --objects' run this decreased the time spent in lookup_object from 27.5% to 24.1%. Signed-off-by: Nicolas Pitre --- I discovered this patch in my git work tree dating from 2 years ago. diff --git a/object.c b/object.c index d8a4b1f..e2dae22 100644 --- a/object.c +++

Re: [PULL REQUEST] initial pack v4 support

2013-09-10 Thread Nicolas Pitre
On Tue, 10 Sep 2013, Junio C Hamano wrote: > Nicolas Pitre writes: > > > Junio, would you please pull the following into pu: > > > > git://git.linaro.org/people/nico/git > > > > This is the pack v4 work to date which is somewhat getting usable. It &g

Re: pack v4 trees with a canonical base

2013-09-10 Thread Nicolas Pitre
On Tue, 10 Sep 2013, Junio C Hamano wrote: > Nicolas Pitre writes: > > > On Tue, 10 Sep 2013, Junio C Hamano wrote: > > > >> There may be trees in the wild that record 100775 or 100777 in the > >> mode field for executable blobs, which also need to be spec

Re: pack v4 trees with a canonical base

2013-09-10 Thread Nicolas Pitre
On Tue, 10 Sep 2013, Junio C Hamano wrote: > Duy Nguyen writes: > > > On Tue, Sep 10, 2013 at 2:25 AM, Nicolas Pitre wrote: > >> An eventual optimization to index-pack when completing a pack would be > >> to attempt the encoding of appended tree objects into th

[PULL REQUEST] initial pack v4 support

2013-09-09 Thread Nicolas Pitre
ack-objects: prepare SHA-1 table in v4 pack-objects: support writing pack v4 pack v4: support "end-of-pack" indicator in index-pack and pack-objects index-pack: use nr_objects_final as sha1_table size index-pack: support completing thin packs v4 Nicolas Pitre (41

Re: [PATCH v2 15/16] index-pack: use nr_objects_final as sha1_table size

2013-09-09 Thread Nicolas Pitre
On Mon, 9 Sep 2013, Junio C Hamano wrote: > Nicolas Pitre writes: > > > Do we know the actual number of objects to send during the capability > > negociation? > > No, and that is not what I meant. We know upfront after capability > negotiation (by seeing a reque

packv4: current performance

2013-09-09 Thread Nicolas Pitre
Let's replace that with specialized string generation functions. This reduced the total runtime by a wopping 40% on a Core2 processor. Signed-off-by: Nicolas Pitre diff --git a/packv4-parse.c b/packv4-parse.c index 63bba03..c62c4ae 100644 --- a/packv4-parse.c +++ b/packv4-

pack v4 trees with a canonical base

2013-09-09 Thread Nicolas Pitre
base tree object to be in the canonical representation. This may happen if the encoder detected an irregularity preventing a loss free encoding to the pack v4 format, or if a thin pack was completed with such tree objects. Signed-off-by: Nicolas Pitre diff --git a/packv4-parse.

Re: [PATCH v2 15/16] index-pack: use nr_objects_final as sha1_table size

2013-09-09 Thread Nicolas Pitre
On Mon, 9 Sep 2013, Junio C Hamano wrote: > Nicolas Pitre writes: > > >> > ... I was > >> > wondering if some kind of prefix to the pack stream could be inserted > >> > onto the wire when sending a pack v4. Something like: > >> > > &

Re: [PATCH v2 15/16] index-pack: use nr_objects_final as sha1_table size

2013-09-09 Thread Nicolas Pitre
On Mon, 9 Sep 2013, Junio C Hamano wrote: > Nicolas Pitre writes: > > > On Mon, 9 Sep 2013, Nguyễn Thái Ngọc Duy wrote: > > > >> nr_objects in the next patch is used to reflect the number of actual > >> objects in the stream, which may be smaller than the

Re: [PATCH v2 15/16] index-pack: use nr_objects_final as sha1_table size

2013-09-09 Thread Nicolas Pitre
On Mon, 9 Sep 2013, Nguyễn Thái Ngọc Duy wrote: > nr_objects in the next patch is used to reflect the number of actual > objects in the stream, which may be smaller than the number recorded > in pack header. This highlights an issue that has been nagging me for a while. We decided to send the fi

Re: [PATCH 08/11] pack-objects: create pack v4 tables

2013-09-09 Thread Nicolas Pitre
On Mon, 9 Sep 2013, Duy Nguyen wrote: > On Sun, Sep 8, 2013 at 10:04 PM, Nguyễn Thái Ngọc Duy > wrote: > > +static void prepare_sha1_table(void) > > +{ > > + unsigned i; > > + /* > > +* This table includes SHA-1s that may not be present in the > > +* pack. One of the

Re: [PATCH 03/11] pack v4: move packv4-create.c to libgit.a

2013-09-08 Thread Nicolas Pitre
On Sun, 8 Sep 2013, Nguyễn Thái Ngọc Duy wrote: > git-packv4-create now becomes test-packv4. Code that will not be used > by pack-objects.c is moved to test-packv4.c. It may be removed when > the code transition to pack-objects completes. > > Signed-off-by: Nguyễn Thái Ngọc Duy > --- > Makefile

Re: [PATCH 05/11] pack-write.c: add pv4_encode_in_pack_object_header

2013-09-08 Thread Nicolas Pitre
On Sun, 8 Sep 2013, Nguyễn Thái Ngọc Duy wrote: > > Signed-off-by: Nguyễn Thái Ngọc Duy > --- > pack-write.c | 29 + > pack.h | 1 + > 2 files changed, 30 insertions(+) > > diff --git a/pack-write.c b/pack-write.c > index 88e4788..6f11104 100644 > --- a/pack-

Re: [PATCH 12/12] index-pack: resolve v4 one-base trees

2013-09-07 Thread Nicolas Pitre
On Sat, 7 Sep 2013, Nguyễn Thái Ngọc Duy wrote: > This is the most common case for delta trees. In fact it's the only > kind that's produced by packv4-create. It fits well in the way > index-pack resolves deltas and benefits from threading (the set of > objects depending on this base does not over

Re: [PATCH 08/12] index-pack: parse v4 tree format

2013-09-07 Thread Nicolas Pitre
On Sat, 7 Sep 2013, Nguyễn Thái Ngọc Duy wrote: > > Signed-off-by: Nguyễn Thái Ngọc Duy > --- > builtin/index-pack.c | 104 > +-- > 1 file changed, 100 insertions(+), 4 deletions(-) > > diff --git a/builtin/index-pack.c b/builtin/index-pack.c >

Re: [PATCH 05/12] index-pack: parse v4 header and dictionaries

2013-09-07 Thread Nicolas Pitre
On Sat, 7 Sep 2013, Nguyễn Thái Ngọc Duy wrote: > > Signed-off-by: Nguyễn Thái Ngọc Duy > --- [...] > @@ -1677,6 +1727,15 @@ int cmd_index_pack(int argc, const char **argv, const > char *prefix) > free(objects); > free(index_name_buf); > free(keep_name_buf); > + free(sha

Re: [PATCH v3] Document pack v4 format

2013-09-06 Thread Nicolas Pitre
On Fri, 6 Sep 2013, Nicolas Pitre wrote: > On Fri, 6 Sep 2013, Duy Nguyen wrote: > > > In your code you reject sha1ref starting with zero too > > (sha1_file.c::get_base_delta and packv4-parse.c::decode_entries) > > Yeah... because I wrote the minimum code to make it

Re: [PATCH v3] Document pack v4 format

2013-09-06 Thread Nicolas Pitre
v2. Therefore the new types are used for the pack v4 optimized commit and tree representations. I've therefore added this patch to my tree (with the needed changes to the documentation patch): commit 98d4b75aff266015b5dff0a324a2984c2a8f7fa2 Author: Nicolas Pitre Date: Fri Sep 6 23:45

Re: [PATCH 10/38] pack v4: commit object encoding

2013-09-06 Thread Nicolas Pitre
On Fri, 6 Sep 2013, Junio C Hamano wrote: > Nicolas Pitre writes: > > > OK. If I understand correctly, the committer time stamp is more > > important than the author's, right? > > Yeah, it matters a lot more when doing timestamp based traversal >

Re: [PATCH 10/38] pack v4: commit object encoding

2013-09-06 Thread Nicolas Pitre
On Thu, 5 Sep 2013, Junio C Hamano wrote: > Nicolas Pitre writes: > > > This goes as follows: > > > > - Tree reference: either variable length encoding of the index > > into the SHA1 table or the literal SHA1 prefixed by 0 (see > > encode_sha1ref()). >

Re: [PATCH v3] Document pack v4 format

2013-09-06 Thread Nicolas Pitre
On Fri, 6 Sep 2013, Duy Nguyen wrote: > On Fri, Sep 6, 2013 at 8:25 PM, Nicolas Pitre wrote: > > On Fri, 6 Sep 2013, Duy Nguyen wrote: > > > >> On Fri, Sep 6, 2013 at 10:23 AM, Nicolas Pitre wrote: > >> > On Fri, 6 Sep 2013, Nguy­n Thái Ng÷c Duy wrote:

Re: [PATCH v3] Document pack v4 format

2013-09-06 Thread Nicolas Pitre
On Fri, 6 Sep 2013, Duy Nguyen wrote: > On Fri, Sep 6, 2013 at 10:23 AM, Nicolas Pitre wrote: > > On Fri, 6 Sep 2013, Nguy­n Thái Ng÷c Duy wrote: > > > >> > >> Signed-off-by: Nguy­n Thái Ng÷c Duy > >> --- > >> Should be up to da

Re: [PATCH v2] Document pack v4 format

2013-09-06 Thread Nicolas Pitre
On Fri, 6 Sep 2013, Duy Nguyen wrote: > On Thu, Sep 5, 2013 at 11:52 PM, Duy Nguyen wrote: > > On Thu, Sep 5, 2013 at 12:39 PM, Nicolas Pitre wrote: > >> Now the pack index v3 probably needs to be improved a little, again to > >> accommodate completion of thin packs

Re: [PATCH v3] Document pack v4 format

2013-09-05 Thread Nicolas Pitre
On Fri, 6 Sep 2013, Nguyễn Thái Ngọc Duy wrote: > > Signed-off-by: Nguyễn Thái Ngọc Duy > --- > Should be up to date with Nico's latest implementation and also cover > additions to the format that everybody seems to agree on: > > - new types for canonical trees and commits > - sha-1 table

Re: [PATCH 37/38] pack v4: introduce "escape hatches" in the name and path indexes

2013-09-05 Thread Nicolas Pitre
On Thu, 5 Sep 2013, Nicolas Pitre wrote: > On Thu, 5 Sep 2013, Nicolas Pitre wrote: > > > If the path or name index is zero, this means the entry data is to be > > found inline rather than being located in the dictionary table. This is > > there to allow easy completi

Re: [PATCH v2] Document pack v4 format

2013-09-05 Thread Nicolas Pitre
On Thu, 5 Sep 2013, Junio C Hamano wrote: > Nicolas Pitre writes: > > > On Thu, 5 Sep 2013, Duy Nguyen wrote: > > > >> On Thu, Sep 5, 2013 at 12:39 PM, Nicolas Pitre wrote: > >> > Now the pack index v3 probably needs to be improved a little, again to >

Re: [PATCH 37/38] pack v4: introduce "escape hatches" in the name and path indexes

2013-09-05 Thread Nicolas Pitre
On Thu, 5 Sep 2013, Nicolas Pitre wrote: > If the path or name index is zero, this means the entry data is to be > found inline rather than being located in the dictionary table. This is > there to allow easy completion of thin packs without having to add new > table entries whic

Re: [PATCH 05/38] pack v4: add commit object parsing

2013-09-05 Thread Nicolas Pitre
On Thu, 5 Sep 2013, SZEDER Gábor wrote: > Hi, > > > On Thu, Sep 05, 2013 at 02:19:28AM -0400, Nicolas Pitre wrote: > > Let's create another dictionary table to hold the author and committer > > entries. We use the same table format used for tree entries where

Re: Zero padded file modes...

2013-09-05 Thread Nicolas Pitre
On Thu, 5 Sep 2013, Jeff King wrote: > On Thu, Sep 05, 2013 at 11:18:24PM +0700, Nguyen Thai Ngoc Duy wrote: > > > > There are basically two solutions: > > > > > > 1. Add a single-bit flag for "I am 0-padded in the real data". We > > > could probably even squeeze it into the same integer.

Re: Zero padded file modes...

2013-09-05 Thread Nicolas Pitre
On Thu, 5 Sep 2013, Jeff King wrote: > There are basically two solutions: > > 1. Add a single-bit flag for "I am 0-padded in the real data". We > could probably even squeeze it into the same integer. > > 2. Have a "classic" section of the pack that stores the raw object > bytes. Fo

Re: [PATCH v2] Document pack v4 format

2013-09-05 Thread Nicolas Pitre
On Thu, 5 Sep 2013, Duy Nguyen wrote: > On Thu, Sep 5, 2013 at 12:39 PM, Nicolas Pitre wrote: > > Now the pack index v3 probably needs to be improved a little, again to > > accommodate completion of thin packs. Given that the main SHA1 table is > > now in the main pa

[PATCH 27/38] pack v4: code to obtain a SHA1 from a sha1ref

2013-09-04 Thread Nicolas Pitre
Let's start actually parsing pack v4 data. Here's the first item. Signed-off-by: Nicolas Pitre --- Makefile | 1 + packv4-parse.c | 30 ++ 2 files changed, 31 insertions(+) create mode 100644 packv4-parse.c diff --git a/Makefile b/Makefile ind

[PATCH 18/38] pack v4: load delta candidate for encoding tree objects

2013-09-04 Thread Nicolas Pitre
investigated eventually. Signed-off-by: Nicolas Pitre --- packv4-create.c | 63 ++--- 1 file changed, 60 insertions(+), 3 deletions(-) diff --git a/packv4-create.c b/packv4-create.c index 6830a0a..15c5959 100644 --- a/packv4-create.c +++ b/packv4

[PATCH 05/38] pack v4: add commit object parsing

2013-09-04 Thread Nicolas Pitre
ed to get the string length as the provided string pointer is not always be null terminated. Signed-off-by: Nicolas Pitre --- packv4-create.c | 98 +++-- 1 file changed, 89 insertions(+), 9 deletions(-) diff --git a/packv4-create.c b/packv4-cre

[PATCH 15/38] pack v4: object data copy

2013-09-04 Thread Nicolas Pitre
not found in the table (see encode_sha1ref). This is true for both REF_DELTA as well as OFS_DELTA. Object payload is validated against the recorded CRC32 in the source pack index file when possible before being copied. Signed-off-by: Nicolas Pitre --- packv4-create.c | 60

[PATCH 25/38] pack v4: initial pack index v3 support on the read side

2013-09-04 Thread Nicolas Pitre
A bit crud but good enough for now. Signed-off-by: Nicolas Pitre --- cache.h | 1 + pack-check.c| 4 +++- pack-revindex.c | 7 --- sha1_file.c | 56 ++-- 4 files changed, 58 insertions(+), 10 deletions(-) diff --git a

[PATCH 19/38] packv4-create: optimize delta encoding

2013-09-04 Thread Nicolas Pitre
Make sure the copy sequence is smaller than the list of tree entries it is meant to replace. We do so by encoding tree entries in parallel with the delta entry comparison, and then comparing the length of both sequences. Signed-off-by: Nicolas Pitre --- packv4-create.c | 65

  1   2   >