[PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too

2016-07-07 Thread Kirill Smelkov
up extracts many packs on repositories restoration. That was my initial motivation for the patch. [1] https://lab.nexedi.com/nexedi/erp5 [2] https://lab.nexedi.com/kirr/git-backup Cc: Vicent Marti Cc: Jeff King Signed-off-by: Kirill Smelkov --- builtin/pack-objects.c | 7 +-- t/t5310-

Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too

2016-07-08 Thread Kirill Smelkov
Peff first of all thanks for feedback, On Thu, Jul 07, 2016 at 04:52:23PM -0400, Jeff King wrote: > On Thu, Jul 07, 2016 at 10:09:17PM +0300, Kirill Smelkov wrote: > > > Starting from 6b8fda2d (pack-objects: use bitmaps when packing objects) > > if a repository has bitmap index

Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too

2016-07-12 Thread Kirill Smelkov
On Fri, Jul 08, 2016 at 01:38:55PM +0300, Kirill Smelkov wrote: > Peff first of all thanks for feedback, > > On Thu, Jul 07, 2016 at 04:52:23PM -0400, Jeff King wrote: > > On Thu, Jul 07, 2016 at 10:09:17PM +0300, Kirill Smelkov wrote: > > > > > Starting from 6b8fd

Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too

2016-07-13 Thread Kirill Smelkov
On Wed, Jul 13, 2016 at 04:26:53AM -0400, Jeff King wrote: > On Fri, Jul 08, 2016 at 01:38:55PM +0300, Kirill Smelkov wrote: > > > > - we will not compute the same write order (which is based on > > > traversal order), leading to packs that h

Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too

2016-07-17 Thread Kirill Smelkov
On Wed, Jul 13, 2016 at 01:52:16PM +0300, Kirill Smelkov wrote: > On Wed, Jul 13, 2016 at 04:26:53AM -0400, Jeff King wrote: > > On Fri, Jul 08, 2016 at 01:38:55PM +0300, Kirill Smelkov wrote: > > > > > > - we will not compute the same write order (which is based on

Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too

2016-07-19 Thread Kirill Smelkov
On Tue, Jul 19, 2016 at 05:29:07AM -0600, Jeff King wrote: > On Sun, Jul 17, 2016 at 08:06:49PM +0300, Kirill Smelkov wrote: > > > > Anyway, please find below updated patch according to your suggestion. > > > Hope it is ok now. > > > > Ping. Is the patch

Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too

2016-07-27 Thread Kirill Smelkov
On Mon, Jul 25, 2016 at 02:40:25PM -0400, Jeff King wrote: > On Wed, Jul 13, 2016 at 01:52:17PM +0300, Kirill Smelkov wrote: > > > > So I think if you were to repeatedly "git repack -adb" over time, you > > > would get worse and worse ordering as objects

Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too

2016-07-28 Thread Kirill Smelkov
Junio, first of all thanks for feedback, On Wed, Jul 27, 2016 at 01:40:36PM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > > From: Kirill Smelkov > > Subject: [PATCH 1/2] pack-objects: Make sure use_bitmap_index is not active > > under > &

Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too

2016-07-29 Thread Kirill Smelkov
On Thu, Jul 28, 2016 at 02:18:29PM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > I'm waiting so long for main patch to be at least queued to pu, that I'm > > now a bit frustrated and ready to do something not related to main goal :) > > Perha

[PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental

2016-07-29 Thread Kirill Smelkov
Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there are two codepaths in pack-objects: with & without using bitmap reachability index. However add_object_entry_from_bitmap(), despite its non-bitmapped counterpart add_object_entry(), in no way does check for whether --local or --h

[PATCH v4 2/2] pack-objects: Teach it to use reachability bitmap index when generating non-stdout pack too

2016-07-29 Thread Kirill Smelkov
gmane.comp.version-control.git/299107 http://article.gmane.org/gmane.comp.version-control.git/299420 http://article.gmane.org/gmane.comp.version-control.git/300217 Cc: Vicent Marti Helped-by: Jeff King Signed-off-by: Kirill Smelkov --- Documentation/config.txt | 3 +++ builtin/pack-objects.c |

Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental

2016-08-08 Thread Kirill Smelkov
On Mon, Aug 01, 2016 at 11:17:30AM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there > > are two codepaths in pack-objects: with & without using bitmap > > reachab

Re: [PATCH v4 2/2] pack-objects: Teach it to use reachability bitmap index when generating non-stdout pack too

2016-08-08 Thread Kirill Smelkov
On Mon, Aug 08, 2016 at 09:56:00AM -0400, Jeff King wrote: > On Fri, Jul 29, 2016 at 10:47:46AM +0300, Kirill Smelkov wrote: > > > @@ -2527,7 +2528,7 @@ static int get_object_list_from_bitmap(struct > > rev_info *revs) > > if (prepare_bitmap_walk(revs) < 0

Re: [PATCH v4 2/2] pack-objects: Teach it to use reachability bitmap index when generating non-stdout pack too

2016-08-08 Thread Kirill Smelkov
On Mon, Aug 08, 2016 at 11:08:34AM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > Thanks for the info. I did not knew about show-index when I was starting > > to work on this and later it just came out of sight. Please find > > corrected patch below. >

Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental

2016-08-08 Thread Kirill Smelkov
( updated patch is in the end of this mail ) Jeff, first of all thanks for commenting, On Mon, Aug 08, 2016 at 09:50:20AM -0400, Jeff King wrote: > On Mon, Aug 08, 2016 at 03:37:35PM +0300, Kirill Smelkov wrote: > > > @@ -958,15 +958,30 @@ static int want_object_in_pack(const u

[PATCH v5] pack-objects: teach it to use reachability bitmap index when generating non-stdout pack too

2016-08-08 Thread Kirill Smelkov
tore` we want to teach `git pack-objects file.pack` to use bitmaps instead of using `git pack-objects --stdout >file.pack` + `git index-pack file.pack`. More context: http://marc.info/?t=14679210141&r=1&w=2 Cc: Vicent Marti Helped-by: Jeff King Signed-off-by: Kirill Smelkov ---

[PATCH v3] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental

2016-08-08 Thread Kirill Smelkov
hange with Junio C Hamano and Jeff King. Signed-off-by: Kirill Smelkov --- builtin/pack-objects.c | 94 ++-- t/t5310-pack-bitmaps.sh | 111 2 files changed, 172 insertions(+), 33 deletions(-) diff --git a/bui

Re: [PATCH v4 2/2] pack-objects: Teach it to use reachability bitmap index when generating non-stdout pack too

2016-08-08 Thread Kirill Smelkov
On Mon, Aug 08, 2016 at 11:28:02AM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > Another question: I'm preparing another version of "pack-objects: Teach > > --use-bitmap-index codepath to respect --local ..." and was going to > > put >

Re: [PATCH v5] pack-objects: teach it to use reachability bitmap index when generating non-stdout pack too

2016-08-09 Thread Kirill Smelkov
On Mon, Aug 08, 2016 at 01:53:20PM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > diff --git a/Documentation/config.txt b/Documentation/config.txt > > index bc1c433..4ba0c4a 100644 > > --- a/Documentation/config.txt > > +++ b/Documentation/con

Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental

2016-08-09 Thread Kirill Smelkov
Junio, first of all thanks for feedback, On Mon, Aug 08, 2016 at 12:26:33PM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: [...] > > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c > > index c4c2a3c..e06c1bf 100644 > > --- a/builtin/pack-objects.c &

[PATCH 1/2 v4] pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in use

2016-08-09 Thread Kirill Smelkov
hange with Junio C Hamano and Jeff King. Signed-off-by: Kirill Smelkov --- builtin/pack-objects.c | 93 +++-- t/t5310-pack-bitmaps.sh | 92 2 files changed, 152 insertions(+), 33 deletions(-) diff --git

[PATCH 2/2 v6] pack-objects: use reachability bitmap index when generating non-stdout pack

2016-08-09 Thread Kirill Smelkov
tore` we want to teach `git pack-objects file.pack` to use bitmaps instead of using `git pack-objects --stdout >file.pack` + `git index-pack file.pack`. More context: http://marc.info/?t=14679210141&r=1&w=2 Cc: Vicent Marti Helped-by: Jeff King Signed-off-by: Kirill Smelkov --- bui

Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental

2016-08-09 Thread Kirill Smelkov
On Tue, Aug 09, 2016 at 09:52:18AM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > Would you please explain why we should not use touch if we do not care > > about timestamps? Simply style? > > To help readers. > > "touch A" forcess the

[PATCH 1/2 v5] pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in use

2016-08-09 Thread Kirill Smelkov
hange with Junio C Hamano and Jeff King. Signed-off-by: Kirill Smelkov --- builtin/pack-objects.c | 93 +++-- t/t5310-pack-bitmaps.sh | 92 2 files changed, 152 insertions(+), 33 deletions(-) diff --git

[PATCH 2/2 v7] pack-objects: use reachability bitmap index when generating non-stdout pack

2016-08-09 Thread Kirill Smelkov
tore` we want to teach `git pack-objects file.pack` to use bitmaps instead of using `git pack-objects --stdout >file.pack` + `git index-pack file.pack`. More context: http://marc.info/?t=14679210141&r=1&w=2 Cc: Vicent Marti Helped-by: Jeff King Signed-off-by: Kirill Smelkov --- bui

Re: [PATCH 1/2 v5] pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in use

2016-09-10 Thread Kirill Smelkov
On Thu, Aug 18, 2016 at 01:52:22PM -0400, Jeff King wrote: > On Tue, Aug 09, 2016 at 10:31:43PM +0300, Kirill Smelkov wrote: > > > Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there > > are two codepaths in pack-objects: with & without using bitmap

Re: [PATCH 2/2 v7] pack-objects: use reachability bitmap index when generating non-stdout pack

2016-09-10 Thread Kirill Smelkov
On Thu, Aug 18, 2016 at 02:06:15PM -0400, Jeff King wrote: > On Tue, Aug 09, 2016 at 10:32:17PM +0300, Kirill Smelkov wrote: > > > Subject: Re: [PATCH 2/2 v7] pack-objects: use reachability bitmap index when > >generating non-stdout pack > > This is v7, but as I under

[PATCH 2/2 v8] pack-objects: use reachability bitmap index when generating non-stdout pack

2016-09-10 Thread Kirill Smelkov
bitmap) 6.86(6.58+0.30) 2.87(2.74+0.20) -58.2% More context: http://marc.info/?t=14679210141&r=1&w=2 http://public-inbox.org/git/20160707190917.20011-1-k...@nexedi.com/T/#t Cc: Vicent Marti Helped-by: Jeff King Signed-off-by: Kirill Smelkov Signed-off-by:

[PATCH 1/2 v8] pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in use

2016-09-10 Thread Kirill Smelkov
pack() into two parts: finding the object and seeing if it impacts our choice to include it in the pack. We may call the inexpensive want_found_object() twice, but we will never call find_pack_entry_one() if we do not need to. I appreciate help and discussing this change with Junio C Hamano and

[PATCH] t/perf/run: Don't forget to copy config.mak.autogen & friends to build area

2016-09-10 Thread Kirill Smelkov
...@sigill.intra.peff.net/ for example. NOTE config.status has to be copied because otherwise without it the build would want to run reconfigure this way loosing just copied config.mak.autogen. Signed-off-by: Kirill Smelkov --- ( Resending as separate patch-mail, just in case ) t/perf/run | 2 +- 1 file changed

Re: [PATCH] t/perf/run: Don't forget to copy config.mak.autogen & friends to build area

2016-09-12 Thread Kirill Smelkov
d_ is regular file. Some people might have config.mak as a symlink for example. We don't want to miss them too. Please find updated patch below: 8< From: Kirill Smelkov Subject: [PATCH] t/perf/run: Don't forget to copy config.mak.autogen & friends to build area Other

Re: [PATCH 1/2 v8] pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in use

2016-09-13 Thread Kirill Smelkov
On Mon, Sep 12, 2016 at 11:23:18PM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > +static int want_found_object(int exclude, struct packed_git *p) > > +{ > > + if (exclude) > > + return 1; > > + if (i

[ANNOUNCE] Git-backup - Backup set of Git repositories & just files; efficiently

2016-01-27 Thread Kirill Smelkov
https://lab.nexedi.com/kirr/git-backup.git This program backups files and set of bare Git repositories into one Git repository. Files are copied to blobs and then added to tree under certain place, and for Git repositories, all reachable objects are pulled in with maintaining index which remember

[PATCH] fetch-pack: demonstrate --all breakage when remote have tags to non-commit objects

2018-06-10 Thread Kirill Smelkov
A2 * [new tag] TAGB1 -> TAGB1 * [new tag] TAGB2 -> TAGB2 * [new tag] tag-to-blob -> tag-to-blob # <-- NOTE * [new tag] tag-to-tree -> tag-to-tree # &

Re: [PATCH v2] fetch-pack: don't try to fetch peel values with --all

2018-06-11 Thread Kirill Smelkov
gt; [...] > > > Original report and test from Kirill Smelkov. > > > > > > Signed-off-by: Kirill Smelkov > > > Signed-off-by: Jeff King > > > --- > > > diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh > > > @@ -506,30 +50

Re: [PATCH v2] fetch-pack: don't try to fetch peel values with --all

2018-06-12 Thread Kirill Smelkov
On Tue, Jun 12, 2018 at 05:48:49AM -0400, Jeff King wrote: > On Mon, Jun 11, 2018 at 09:43:02AM +0000, Kirill Smelkov wrote: > > > > Looking deeper, we do not need these trees and blobs at all. The problem > > > is really just a tag that peels to an object that is not o

[PATCH] fetch-pack: test explicitly that --all can fetch tag references pointing to non-commits

2018-06-13 Thread Kirill Smelkov
refs/tags/tag-to-tree^{} .../git/t/trash directory.t5500-fetch-pack/fetchall$ git fetch-pack --all .. fatal: A git upload-pack: not our ref 038f48ad0beaffbea71d186a05084b79e3870cbf fatal: The remote end hung up unexpectedly Signed-off-by: Kirill Smelkov --- t/t5500-fe

[PATCH] fetch-pack: demonstrate --all failure when remote is empty

2018-06-13 Thread Kirill Smelkov
( Junio, please pick up the patch provided in the end ) On Tue, Jun 12, 2018 at 06:54:17PM +, Kirill Smelkov wrote: > On Tue, Jun 12, 2018 at 05:48:49AM -0400, Jeff King wrote: > > On Mon, Jun 11, 2018 at 09:43:02AM +0000, Kirill Smelkov wrote: [...] > > > I'm not sur

Re: [PATCH] fetch-pack: demonstrate --all failure when remote is empty

2018-06-13 Thread Kirill Smelkov
On Wed, Jun 13, 2018 at 10:13:07AM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > ( Junio, please pick up the patch provided in the end ) > > > > On Tue, Jun 12, 2018 at 06:54:17PM +, Kirill Smelkov wrote: > >> On Tue, Jun 12, 2018 at 05:48:49AM

Re: [PATCH] fetch-pack: test explicitly that --all can fetch tag references pointing to non-commits

2018-06-13 Thread Kirill Smelkov
On Wed, Jun 13, 2018 at 10:42:33AM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > Fetch-pack --all became broken with respect to unusual tags in > > 5f0fc64513 (fetch-pack: eliminate spurious error messages, 2012-09-09), > > and was fixed only recently i

Re: [PATCH] fetch-pack: test explicitly that --all can fetch tag references pointing to non-commits

2018-06-13 Thread Kirill Smelkov
On Wed, Jun 13, 2018 at 07:11:47PM -0400, Jeff King wrote: > On Wed, Jun 13, 2018 at 05:05:09PM -0400, Jeff King wrote: > > > > In order to be sure fetching funky tags will never break, let's > > > explicitly test all relevant cases with 4 tag objects pointing to 1) a > > > blob, 2) a tree, 3) a c

Re: [PATCH v2] fetch-pack: don't try to fetch peel values with --all

2018-06-13 Thread Kirill Smelkov
On Wed, Jun 13, 2018 at 05:13:02PM -0400, Jeff King wrote: > On Tue, Jun 12, 2018 at 06:54:17PM +0000, Kirill Smelkov wrote: > > > > If an extra connection isn't a problem, you might be better off with > > > "git ls-remote", and then picking through the

Re: [PATCH] fetch-pack: test explicitly that --all can fetch tag references pointing to non-commits

2018-06-14 Thread Kirill Smelkov
On Thu, Jun 14, 2018 at 09:07:26AM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > Jeff, thanks for corrections. I originally tried to look into invoking > > "git tag" two times, but since git tag always creates a reference it > > would not be s

Re: [PATCH] t5500: prettify non-commit tag tests

2018-07-03 Thread Kirill Smelkov
On Tue, Jul 03, 2018 at 12:55:19PM -0400, Jeff King wrote: > I had prepared this as a squash-in for what became c12c9df527, but since > that's now in master, it can go on top (or get dropped, but I think it > is worth it as a style fixup). I'm ok with the patch. I thought it was already squashed i

Re: [PATCH] t5310-pack-bitmaps: fix bogus 'pack-objects to file can use bitmap' test

2018-08-27 Thread Kirill Smelkov
On Tue, Aug 14, 2018 at 01:47:21PM +0200, SZEDER Gábor wrote: > The test 'pack-objects to file can use bitmap' added in 645c432d61 > (pack-objects: use reachability bitmap index when generating > non-stdout pack, 2016-09-10) is silently buggy and doesn't check what > it's supposed to. > > In 't5310

Re: [PATCH] t5310-pack-bitmaps: fix bogus 'pack-objects to file can use bitmap' test

2018-08-27 Thread Kirill Smelkov
On Mon, Aug 27, 2018 at 07:04:52PM -0400, Jeff King wrote: > On Mon, Aug 27, 2018 at 10:22:46AM +0000, Kirill Smelkov wrote: > > > A minor comment from outside observer: running tests under something > > like > > > > -e and -o pipefail > > > > wo

Re: [PATCH 11/11] tree-diff: reuse base str(buf) memory on sub-tree recursion

2014-02-13 Thread Kirill Smelkov
On Fri, Feb 07, 2014 at 09:48:52PM +0400, Kirill Smelkov wrote: > instead of allocating it all the time for every subtree in > __diff_tree_sha1, let's allocate it once in diff_tree_sha1, and then all > callee just use it in stacking style, without memory allocations. > > T

[PATCH 0/2] Multiparent diff tree-walker + combine-diff speedup

2014-02-13 Thread Kirill Smelkov
, which for linux.git results in ~14 times speedup. I understand v1.9.0 is going to be released first, but wanted to finally send the patches, so that people could start reviewing them. Please apply on top of ks/tree-diff-more and thanks beforehand, Kirill Kirill Smelkov (2): tree-diff: rework

[PATCH 2/2] combine-diff: speed it up, by using multiparent diff tree-walker directly

2014-02-13 Thread Kirill Smelkov
.2s after 1.9s 2.4s 1.1s The result stayed the same. Signed-off-by: Kirill Smelkov --- combine-diff.c | 85 ++ diff.c | 1 + 2 files changed, 81 insertions(+), 5 deletions(-) diff --git a/combine-diff.c b/combine-di

[PATCH 1/2] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

2014-02-13 Thread Kirill Smelkov
uilding a map {} sha1-> in which commit (and date) a content was added that `-c` means also show combined diff for merges, and without them, if a merge is non-trivial (merges changes from two parents with both having separate changes to a file), or an evil one, the map will not

Re: [PATCH 00/11] More preparatory work for multiparent tree-walker

2014-02-13 Thread Kirill Smelkov
On Wed, Feb 12, 2014 at 09:25:51AM -0800, Junio C Hamano wrote: > Junio C Hamano writes: > > > Kirill Smelkov writes: > > > >> Sorry for the confusion. Could you please do the following: > >> > >> Patches should be applied over to ks/tree-diff-walk

Re: [PATCH 02/11] tree-diff: consolidate code for emitting diffs and recursion in one place

2014-02-13 Thread Kirill Smelkov
On Thu, Feb 13, 2014 at 09:43:27AM -0800, Junio C Hamano wrote: > Kirill Smelkov writes: > > > +static void show_path(struct strbuf *base, struct diff_options *opt, > > + struct tree_desc *t1, struct tree_desc *t2) > > { > > unsigned

Re: [PATCH 2/2] combine-diff: speed it up, by using multiparent diff tree-walker directly

2014-02-14 Thread Kirill Smelkov
On Thu, Feb 13, 2014 at 11:55:08AM -0800, Junio C Hamano wrote: > Kirill Smelkov writes: > > > + if (need_generic_pathscan) { > > + /* NOTE generic case also handles --stat, as it computes > > +* diff(sha1,parent_i) for all i to

Re: [PATCH 1/2] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

2014-02-14 Thread Kirill Smelkov
On Thu, Feb 13, 2014 at 11:51:19AM -0800, Junio C Hamano wrote: > Kirill Smelkov writes: > > > + /* until we go to it next round, .next holds how many bytes we > > +* allocated (for faster realloc - we don't need copying old > > data). > >

Re: [PATCH 1/2] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

2014-02-16 Thread Kirill Smelkov
On Fri, Feb 14, 2014 at 09:37:00AM -0800, Junio C Hamano wrote: > Kirill Smelkov writes: > > > Previously diff_tree(), which is now named __diff_tree_sha1(), was > > That name with two leading underscores is a rather unfortunate, > especially for a function that is not a f

[PATCH 02/19] combine-diff: move changed-paths scanning logic into its own function

2014-02-24 Thread Kirill Smelkov
faster, paths scanning implementation in the next patch. Signed-off-by: Kirill Smelkov Signed-off-by: Junio C Hamano --- ( re-posting without change ) combine-diff.c | 80 ++ 1 file changed, 53 insertions(+), 27 deletions(-) diff --git a

[PATCH v2 00/19] Multiparent diff tree-walker + combine-diff speedup

2014-02-24 Thread Kirill Smelkov
ithout sacrificing signal/noise ratio. All of them are now static though. Signoffs were left intact, if a patch was already applied to pu with one, and had not changed. Please apply and thanks, Kirill P.S. Sorry for the delay - I was very busy. Kirill Smelkov (19): combine-diff: move show_log_f

[PATCH 01/19] combine-diff: move show_log_first logic/action out of paths scanning

2014-02-24 Thread Kirill Smelkov
take care to explicitly not show anything if parents array is empty, as in fact there are some clients in Git code, which calls diff_tree_combined() in such a way. Signed-off-by: Kirill Smelkov Signed-off-by: Junio C Hamano --- ( re-posting without change ) combine-diff.c

[PATCH 13/19] tree-diff: diff_tree() should now be static

2014-02-24 Thread Kirill Smelkov
We reworked all its users to use the functionality through diff_tree_sha1 variant in recent patches (see "tree-diff: allow diff_tree_sha1 to accept NULL sha1" and what comes next). diff_tree() is now not used outside tree-diff.c - make it static. Signed-off-by: Kirill Smelkov --- (

[PATCH 09/19] tree-diff: rename compare_tree_entry -> tree_entry_pathcmp

2014-02-24 Thread Kirill Smelkov
. Signed-off-by: Kirill Smelkov Signed-off-by: Junio C Hamano --- ( re-posting without change ) tree-diff.c | 15 +-- 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/tree-diff.c b/tree-diff.c index 6207372..3345534 100644 --- a/tree-diff.c +++ b/tree-diff.c @@ -9,7 +9,14

[PATCH 11/19] tree-diff: simplify tree_entry_pathcmp

2014-02-24 Thread Kirill Smelkov
looks cleaner. Signed-off-by: Kirill Smelkov Signed-off-by: Junio C Hamano --- ( re-posting without change ) tree-diff.c | 17 ++--- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/tree-diff.c b/tree-diff.c index 20a4fda..cf96ad7 100644 --- a/tree-diff.c +++ b/tree-

[PATCH 06/19] tree-diff: consolidate code for emitting diffs and recursion in one place

2014-02-24 Thread Kirill Smelkov
patch, stay the same. Signed-off-by: Kirill Smelkov Signed-off-by: Junio C Hamano --- ( re-posting without change ) tree-diff.c | 115 1 file changed, 84 insertions(+), 31 deletions(-) diff --git a/tree-diff.c b/tree-diff.c in

[PATCH 12/19] tree-diff: remove special-case diff-emitting code for empty-tree cases

2014-02-24 Thread Kirill Smelkov
there will be several parents iterated simultaneously, and some pair of them would run to the end. Signed-off-by: Kirill Smelkov Signed-off-by: Junio C Hamano --- ( re-posting without change ) tree-diff.c | 21 + 1 file changed, 9 insertions(+), 12 deletions(-) diff --gi

[PATCH 04/19] tree-diff: no need to pass match to skip_uninteresting()

2014-02-24 Thread Kirill Smelkov
It is neither used there as input, nor the output written through it, is used outside. Signed-off-by: Kirill Smelkov Signed-off-by: Junio C Hamano --- ( re-posting without change ) tree-diff.c | 17 - 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/tree-diff.c b

[PATCH 19/19] combine-diff: speed it up, by using multiparent diff tree-walker directly

2014-02-24 Thread Kirill Smelkov
after 1.9s 2.4s 1.1s The result stayed the same. Signed-off-by: Kirill Smelkov Signed-off-by: Junio C Hamano --- ( re-posting without change ) combine-diff.c | 88 ++ diff.c | 1 + 2 files changed, 84 insertions(

[PATCH 15/19] tree-diff: no need to call "full" diff_tree_sha1 from show_path()

2014-02-24 Thread Kirill Smelkov
, as we have added a path and '/' before recursing. Signed-off-by: Kirill Smelkov Signed-off-by: Junio C Hamano --- ( re-posting without change ) tree-diff.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tree-diff.c b/tree-diff.c index f90acf5..aea0297 100644

[PATCH 17/19] Portable alloca for Git

2014-02-24 Thread Kirill Smelkov
I've changed that to now-common HAVE_ALLOCA_H=YesPlease which should be correct. Cc: Brandon Casey Cc: Marius Storm-Olsen Cc: Johannes Sixt Cc: Johannes Schindelin Cc: Ramsay Jones Cc: Gerrit Pape Cc: Petr Salinger Cc: Jonathan Nieder Cc: Thomas Schwinge Signed-off-by: Kirill Smelkov

[PATCH 03/19] tree-diff: no need to manually verify that there is no mode change for a path

2014-02-24 Thread Kirill Smelkov
diff-tree recursion), which pre-dates base_name_compare() introduction in 958ba6c9 (Introduce "base_name_compare()" helper function) by a month. Signed-off-by: Kirill Smelkov Signed-off-by: Junio C Hamano --- ( re-posting without change ) tree-diff.c | 15 +-- 1 file change

[PATCH 05/19] tree-diff: show_tree() is not needed

2014-02-24 Thread Kirill Smelkov
nto one place. Signed-off-by: Kirill Smelkov --- ( re-posting without change ) tree-diff.c | 35 +++ 1 file changed, 3 insertions(+), 32 deletions(-) diff --git a/tree-diff.c b/tree-diff.c index a8c2aec..2ad7788 100644 --- a/tree-diff.c +++ b/tree-diff.c @@ -55,25 +5

[PATCH v2 18/19] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

2014-02-24 Thread Kirill Smelkov
ilesystem, there is initial scan with `git log --reverse --raw --no-abbrev --no-renames -c` to extract log of what was created/changed when, as a result building a map {} sha1-> in which commit (and date) a content was added that `-c` means also show combined diff for merges, and

[PATCH 10/19] tree-diff: show_path prototype is not needed anymore

2014-02-24 Thread Kirill Smelkov
We moved all action-taking code below show_path() in recent HEAD~~ (tree-diff: move all action-taking code out of compare_tree_entry). Signed-off-by: Kirill Smelkov Signed-off-by: Junio C Hamano --- ( re-posting without change ) tree-diff.c | 3 --- 1 file changed, 3 deletions(-) diff --git

[PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

2014-02-24 Thread Kirill Smelkov
() to diff_tree_sha1(). As diff_tree_sha1() was already used, and the function we are talking here is its more low-level helper, let's use Linux convention for prefixing such helpers with double underscore. So the final renaming is diff_tree() -> __diff_tree_sha1() Signed-off-by: Kirill

[PATCH 08/19] tree-diff: move all action-taking code out of compare_tree_entry()

2014-02-24 Thread Kirill Smelkov
-already-there same switch/if, so the result is maybe a little bit faster. Signed-off-by: Kirill Smelkov Signed-off-by: Junio C Hamano --- ( re-posting without change ) tree-diff.c | 28 1 file changed, 12 insertions(+), 16 deletions(-) diff --git a/tree-diff.c b/tree

[PATCH 07/19] tree-diff: don't assume compare_tree_entry() returns -1,0,1

2014-02-24 Thread Kirill Smelkov
r we'll be adding support for multiparent walker and will stay that way. =0 case goes first, because it happens more often in real diffs - i.e. paths are the same. Signed-off-by: Kirill Smelkov Signed-off-by: Junio C Hamano --- ( re-posting without change ) tree-diff.c | 22

[PATCH v2 16/19] tree-diff: reuse base str(buf) memory on sub-tree recursion

2014-02-24 Thread Kirill Smelkov
t log --raw --no-abbrev --no-renames --format='%H' navy.gitlinux.git v3.10..v3.11 before 0.618s 1.903s after 0.611s 1.889s speedup 1.1%0.7% Signed-off-by: Kirill Smelkov --- Changes since v1: - don't need to

Re: [PATCH v2 00/19] Multiparent diff tree-walker + combine-diff speedup

2014-02-25 Thread Kirill Smelkov
On Tue, Feb 25, 2014 at 06:43:24AM +0700, Duy Nguyen wrote: > On Mon, Feb 24, 2014 at 11:21 PM, Kirill Smelkov wrote: > > Hello up there. > > > > Here go combine-diff speedup patches in form of first reworking diff > > tree-walker to work in general case - when a

Re: [PATCH 17/19] Portable alloca for Git

2014-02-28 Thread Kirill Smelkov
On Fri, Feb 28, 2014 at 02:50:04PM +0100, Erik Faye-Lund wrote: > On Fri, Feb 28, 2014 at 2:44 PM, Erik Faye-Lund wrote: > > On Mon, Feb 24, 2014 at 5:21 PM, Kirill Smelkov wrote: > >> diff --git a/Makefile b/Makefile > >> index dddaf4f..0334806 100644 > >>

Re: [PATCH 17/19] Portable alloca for Git

2014-03-05 Thread Kirill Smelkov
On Fri, Feb 28, 2014 at 06:19:58PM +0100, Erik Faye-Lund wrote: > On Fri, Feb 28, 2014 at 6:00 PM, Kirill Smelkov wrote: > > On Fri, Feb 28, 2014 at 02:50:04PM +0100, Erik Faye-Lund wrote: > >> On Fri, Feb 28, 2014 at 2:44 PM, Erik Faye-Lund > >> wrote: > >&g

Re: [PATCH 12/19] tree-diff: remove special-case diff-emitting code for empty-tree cases

2014-03-25 Thread Kirill Smelkov
On Mon, Mar 24, 2014 at 02:18:10PM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > via teaching tree_entry_pathcmp() how to compare empty tree descriptors: > > Drop this line, as you explain the "pretend empty compares bigger > than anything else" idea

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

2014-03-25 Thread Kirill Smelkov
On Mon, Mar 24, 2014 at 02:36:22PM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > The downside is that try_to_follow_renames(), if active, we cause > > re-reading of 2 initial trees, which was negligible based on my timings, > > That would depend on how ofte

Re: [PATCH 11/19] tree-diff: simplify tree_entry_pathcmp

2014-03-25 Thread Kirill Smelkov
On Mon, Mar 24, 2014 at 02:25:04PM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > Since an earlier "Finally switch over tree descriptors to contain a > > pre-parsed entry", we can safely access all tree_desc->entry fields > > directly inste

Re: [PATCH v2 16/19] tree-diff: reuse base str(buf) memory on sub-tree recursion

2014-03-25 Thread Kirill Smelkov
On Mon, Mar 24, 2014 at 02:43:36PM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > instead of allocating it all the time for every subtree in > > __diff_tree_sha1, let's allocate it once in diff_tree_sha1, and then all > > callee just use it in s

Re: [PATCH 12/19] tree-diff: remove special-case diff-emitting code for empty-tree cases

2014-03-26 Thread Kirill Smelkov
On Tue, Mar 25, 2014 at 10:45:01AM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > >> > static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc > >> > *t2) > >> > { > >> > struct name_entry *e1, *e2; >

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

2014-03-26 Thread Kirill Smelkov
On Tue, Mar 25, 2014 at 10:46:32AM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > What are the downsides of "__" prefix by the way? > > Aren't these names reserved for compiler/runtime implementations? Yes, but there are precedents when people don

Re: [PATCH 17/19] Portable alloca for Git

2014-03-27 Thread Kirill Smelkov
On Mon, Mar 24, 2014 at 02:47:24PM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > On Fri, Feb 28, 2014 at 06:19:58PM +0100, Erik Faye-Lund wrote: > >> On Fri, Feb 28, 2014 at 6:00 PM, Kirill Smelkov wrote: > >> ... > >> > In fact that wo

Re: [PATCH v2 18/19] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

2014-03-27 Thread Kirill Smelkov
On Mon, Feb 24, 2014 at 08:21:50PM +0400, Kirill Smelkov wrote: [...] > not changed: > > - low-level helpers are still named with "__" prefix as, imho, that is the > best > convention to name such helpers, without sacrificing signal/noise ratio. All > of them ar

Re: [PATCH v2 16/19] tree-diff: reuse base str(buf) memory on sub-tree recursion

2014-03-27 Thread Kirill Smelkov
On Tue, Mar 25, 2014 at 01:23:20PM +0400, Kirill Smelkov wrote: > On Mon, Mar 24, 2014 at 02:43:36PM -0700, Junio C Hamano wrote: > > Kirill Smelkov writes: > > > > > instead of allocating it all the time for every subtree in > > > __diff_tree_sha1, let's a

Re: [PATCH 15/19] tree-diff: no need to call "full" diff_tree_sha1 from show_path()

2014-03-27 Thread Kirill Smelkov
On Mon, Feb 24, 2014 at 08:21:47PM +0400, Kirill Smelkov wrote: > As described in previous commit, when recursing into sub-trees, we can > use lower-level tree walker, since its interface is now sha1 based. > > The change is ok, because diff_tree_sha1() only invokes > __diff_tree_s

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

2014-03-27 Thread Kirill Smelkov
On Wed, Mar 26, 2014 at 02:34:24PM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > On Tue, Mar 25, 2014 at 10:46:32AM -0700, Junio C Hamano wrote: > >> Kirill Smelkov writes: > >> > >> > What are the downsides of "__" prefix by

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

2014-03-27 Thread Kirill Smelkov
+stefanbeller On Thu, Mar 27, 2014 at 11:48:11AM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > (please keep author email) > > 8< ---- > > From: Kirill Smelkov > > Date: Mon, 24 Feb 2014 20:21:46 +0400 > > Subject: [PATCH v3a] tree-diff

Re: [PATCH v2 18/19] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

2014-04-06 Thread Kirill Smelkov
Junio, First of all thanks a lot for reviewing this patch. I'll reply inline with corrected version attached in the end. On Fri, Apr 04, 2014 at 11:42:39AM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > +extern > > +struct combine_diff_path *diff_tree_paths

Re: [PATCH v2 18/19] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

2014-04-07 Thread Kirill Smelkov
On Mon, Apr 07, 2014 at 10:29:46AM -0700, Junio C Hamano wrote: > Kirill Smelkov writes: > > > The following > > ... > > maybe looks a bit simpler, but calls tree_entry_pathcmp twice more times. > > > > Besides for important nparent=1 case we were not calli

Re: [PATCH 17/19] Portable alloca for Git

2014-04-09 Thread Kirill Smelkov
On Thu, Mar 27, 2014 at 06:22:50PM +0400, Kirill Smelkov wrote: > On Mon, Mar 24, 2014 at 02:47:24PM -0700, Junio C Hamano wrote: > > Kirill Smelkov writes: > > > > > On Fri, Feb 28, 2014 at 06:19:58PM +0100, Erik Faye-Lund wrote: > > >> On Fri, Feb 28, 20

[PATCH 2/4] diff test: Add tests for combine-diff with orderfile

2014-01-20 Thread Kirill Smelkov
In the next patch combine-diff will have special code-path for taking orderfile into account. Prepare for making changes by introducing coverage tests for that case. Signed-off-by: Kirill Smelkov --- t/t4056-diff-order.sh | 21 + 1 file changed, 21 insertions(+) diff --git

[PATCH 3/4] combine-diff: Optimize combine_diff_path sets intersection

2014-01-20 Thread Kirill Smelkov
anup path, to maintain good signal/noise ratio here. Signed-off-by: Kirill Smelkov --- combine-diff.c | 93 +- 1 file changed, 72 insertions(+), 21 deletions(-) diff --git a/combine-diff.c b/combine-diff.c index 3b92c448..98c2562 100644

[PATCH 1/4] diffcore-order: Export generic ordering interface

2014-01-20 Thread Kirill Smelkov
vided they have path accessors. The new interface is: - `struct obj_order`for describing objects to ordering routine, and - order_objects() for actually doing the ordering work. Signed-off-by: Kirill Smelkov --- diffcore-or

[PATCH 0/4] `log -c` speedup

2014-01-20 Thread Kirill Smelkov
it. Please apply. Thanks beforehand, Kirill Kirill Smelkov (4): diffcore-order: Export generic ordering interface diff test: Add tests for combine-diff with orderfile combine-diff: Optimize combine_diff_path sets intersection combine-diff: combine_diff_path.len is not needed anymore comb

[PATCH 4/4] combine-diff: combine_diff_path.len is not needed anymore

2014-01-20 Thread Kirill Smelkov
Brefore previous patch, ->len was used to speedup name compares and also to mark removed paths via len=0. Now we do significantly less strcmp and also just remove paths from list and free right after we know a path will not be needed, so ->len is not needed anymore. Signed-off-by: Kirill S

Re: [PATCH 3/4] combine-diff: Optimize combine_diff_path sets intersection

2014-01-28 Thread Kirill Smelkov
On Mon, Jan 20, 2014 at 08:20:40PM +0400, Kirill Smelkov wrote: [...] > @@ -1343,6 +1374,26 @@ void diff_tree_combined(const unsigned char *sha1, > if (p->len) > num_paths++; > } > + > + /* order paths according to diffcore_ord

Re: [PATCH 3/4] combine-diff: Optimize combine_diff_path sets intersection

2014-01-29 Thread Kirill Smelkov
On Tue, Jan 28, 2014 at 01:55:09PM -0800, Junio C Hamano wrote: > Kirill Smelkov writes: > > > diff --git a/combine-diff.c b/combine-diff.c > > index 3b92c448..98c2562 100644 > > --- a/combine-diff.c > > +++ b/combine-diff.c > > @@ -15

  1   2   >