Re: [PATCH] convert any hard coded .gitmodules file string to the MACRO

2017-08-01 Thread Jeff Hostetler
On 7/31/2017 7:11 PM, Stefan Beller wrote: I used these commands: $ cat sem.cocci @@ @@ - ".gitmodules" + GITMODULES_FILE $ spatch --in-place --sp-file sem.cocci builtin/*.c *.c *.h Feel free to regenerate or squash it in or have it as a separate commit. Signed-off-by: Stef

Re: tsan: t3008: hashmap_add touches size from multiple threads

2017-08-15 Thread Jeff Hostetler
On 8/15/2017 8:53 AM, Martin Ågren wrote: Using SANITIZE=thread made t3008-ls-files-lazy-init-name-hash.sh hit the potential race below. What seems to happen is, threaded_lazy_init_name_hash ends up using hashmap_add on the_index.dir_hash from two threads in a way that tsan considers racy. Whi

Re: tsan: t3008: hashmap_add touches size from multiple threads

2017-08-15 Thread Jeff Hostetler
On 8/15/2017 3:21 PM, Martin Ågren wrote: On 15 August 2017 at 20:48, Stefan Beller wrote: /* total number of entries (0 means the hashmap is empty) */ - unsigned int size; + /* -1 means size is unknown for threading reasons */ + int size; This double-encodes the

Re: [PATCH v2 0/4] Some ThreadSanitizer-results

2017-08-28 Thread Jeff Hostetler
On 8/21/2017 1:43 PM, Martin Ågren wrote: This is the second version of my series to try to address some issues ... 2) hashmap_add, which I could try my hands on if Jeff doesn't beat me to it -- his proposed change should fix it and I doubt I could come up with anything "better", considering

Re: [RFC 0/7] transitioning to protocol v2

2017-08-29 Thread Jeff Hostetler
On 8/25/2017 1:35 PM, Jonathan Nieder wrote: Hi, Jeff King wrote: On Thu, Aug 24, 2017 at 03:53:21PM -0700, Brandon Williams wrote: Another version of Git's wire protocol is a topic that has been discussed and attempted by many in the community over the years. The biggest challenge, as fa

Re: [RFC 0/7] transitioning to protocol v2

2017-08-30 Thread Jeff Hostetler
On 8/29/2017 11:06 PM, Jeff King wrote: On Tue, Aug 29, 2017 at 04:08:25PM -0400, Jeff Hostetler wrote: I just wanted to jump in here and say I've done some initial testing of this against VSTS and so far it seems fine. And yes, we have a custom git server. Great, thank you for che

[PATCH] hashmap: address ThreadSanitizer concerns

2017-08-30 Thread Jeff Hostetler
From: Jeff Hostetler This is to address concerns raised by ThreadSanitizer on the mailing list about threaded unprotected R/W access to map.size with my previous "disallow rehash" change (0607e10009ee4e37cb49b4cec8d28a9dda1656a4). See: https://public-inb

[PATCH] hashmap: add API to disable item counting when threaded

2017-08-30 Thread Jeff Hostetler
From: Jeff Hostetler This is to address concerns raised by ThreadSanitizer on the mailing list about threaded unprotected R/W access to map.size with my previous "disallow rehash" change (0607e10009ee4e37cb49b4cec8d28a9dda1656a4). See: https://public-inb

Re: [PATCH] hashmap: add API to disable item counting when threaded

2017-09-05 Thread Jeff Hostetler
On 9/1/2017 7:31 PM, Johannes Schindelin wrote: Hi Jeff, On Wed, 30 Aug 2017, Jeff Hostetler wrote: From: Jeff Hostetler This is to address concerns raised by ThreadSanitizer on the mailing list about threaded unprotected R/W access to map.size with my previous "disallow rehash&qu

Re: [PATCH] hashmap: add API to disable item counting when threaded

2017-09-05 Thread Jeff Hostetler
On 9/1/2017 7:50 PM, Jonathan Nieder wrote: Hi, Johannes Schindelin wrote: On Wed, 30 Aug 2017, Jeff Hostetler wrote: This is to address concerns raised by ThreadSanitizer on the mailing list about threaded unprotected R/W access to map.size with my previous "disallow rehash&qu

Re: [PATCH] hashmap: add API to disable item counting when threaded

2017-09-05 Thread Jeff Hostetler
On 9/2/2017 4:17 AM, Jeff King wrote: On Sat, Sep 02, 2017 at 01:31:19AM +0200, Johannes Schindelin wrote: Before anybody can ask for this message to be wrapped in _(...) to be translateable, let me suggest instead to add the prefix "BUG: ". Agreed on both (and Jonathan's suggestion to just

Re: [PATCH] hashmap: add API to disable item counting when threaded

2017-09-05 Thread Jeff Hostetler
On 9/2/2017 4:05 AM, Jeff King wrote: On Wed, Aug 30, 2017 at 06:59:22PM +, Jeff Hostetler wrote: From: Jeff Hostetler This is to address concerns raised by ThreadSanitizer on the mailing list about threaded unprotected R/W access to map.size with my previous "disallow rehash&qu

Re: [PATCH] hashmap: add API to disable item counting when threaded

2017-09-06 Thread Jeff Hostetler
On 9/5/2017 9:24 PM, Junio C Hamano wrote: Jeff Hostetler writes: From: Jeff Hostetler I feel somewhat stupid to say this, especially after seeing many people applaud this patch, but I do not seem to be able to even build Git with this patch. I am getting: common-main.o: In

[PATCH v2] hashmap: add API to disable item counting when threaded

2017-09-06 Thread Jeff Hostetler
From: Jeff Hostetler This is to address concerns raised by ThreadSanitizer on the mailing list about threaded unprotected R/W access to map.size with my previous "disallow rehash" change (0607e10009ee4e37cb49b4cec8d28a9dda1656a4). See: https://public-inb

[PATCH v2] hashmap: address ThreadSanitizer concerns

2017-09-06 Thread Jeff Hostetler
From: Jeff Hostetler Version 2 addresses the comments and suggestions on version 1. It removes the explicit disable/enable rehash and just relies on the state of hashmap counting. It changes the declaration of the hashmap_get_size() to be static to avoid issues seen on some compilers. It uses

Re: RFC: Design and code of partial clones (now, missing commits and trees OK)

2017-09-21 Thread Jeff Hostetler
There's a lot in this patch series. I'm still studying it, but here are some notes and questions. I'll start with direct responses to the RFC here and follow up in a second email with specific questions and comments to keep this from being too long). On 9/15/2017 4:43 PM, Jonathan Tan wrote:

Re: RFC: Design and code of partial clones (now, missing commits and trees OK) (part 2/3)

2017-09-21 Thread Jeff Hostetler
(part 2) Additional overall comments on: https://github.com/jonathantanmy/git/commits/partialclone2 {} I think it would help to split the blob-max-bytes filtering and the promisor/promised concepts and discuss them independently. {} Then we can talk about about the promisor/promised funct

Re: RFC: Design and code of partial clones (now, missing commits and trees OK) (part 3)

2017-09-21 Thread Jeff Hostetler
(part 3) Additional overall comments on: https://github.com/jonathantanmy/git/commits/partialclone2 {} WRT the code in is_promised() [1] [1] https://github.com/jonathantanmy/git/commit/7a9c2d9b6e2fce293817b595dee29a7eede0#diff-5d5d5dc185ef37dc30bb7d9a7ae0c4e8R1960 {} it looked like it

Re: [PATCH] git: add --no-optional-locks option

2017-09-22 Thread Jeff Hostetler
On 9/22/2017 12:25 AM, Jeff King wrote: On Thu, Sep 21, 2017 at 08:25:50PM +0200, Johannes Sixt wrote: +`GIT_OPTIONAL_LOCKS`:: + If set to `0`, Git will avoid performing any operations which + require taking a lock and which are not required to complete the + requested opera

[PATCH 03/13] list-objects: filter objects in traverse_commit_list

2017-09-22 Thread Jeff Hostetler
From: Jeff Hostetler Create traverse_commit_list_filtered() and add filtering interface to allow certain objects to be omitted (not shown) during a traversal. Update traverse_commit_list() to be a wrapper for the above. Filtering will be used in a future commit by rev-list and pack-objects for

[PATCH 04/13] list-objects-filter-all: add filter to omit all blobs

2017-09-22 Thread Jeff Hostetler
From: Jeff Hostetler Create a simple filter for traverse_commit_list_worker() to omit all blobs from the result. This filter will be used in a future commit by rev-list and pack-objects to create a "commits and trees" result. This is intended for partial clone and fetch support. Sig

[PATCH 02/13] oidset2: create oidset subclass with object length and pathname

2017-09-22 Thread Jeff Hostetler
From: Jeff Hostetler Create subclass of oidset where each entry has a field to store the length of the object's content and an optional pathname. This will be used in a future commit to build a manifest of omitted objects in a partial/narrow clone/fetch. Signed-off-by: Jeff Host

[PATCH 01/13] dir: refactor add_excludes()

2017-09-22 Thread Jeff Hostetler
From: Jeff Hostetler Refactor add_excludes() to separate the reading of the exclude file into a buffer and the parsing of the buffer into exclude_list items. Add add_excludes_from_blob_to_list() to allow an exclude file be specified with an OID. Signed-off-by: Jeff Hostetler --- dir.c | 53

[PATCH 00/13] RFC object filtering for parital clone

2017-09-22 Thread Jeff Hostetler
From: Jeff Hostetler This patch series contains WIP code demonstrating object (blob) filtering in rev-list and pack-objects using a common filtering API in list-objects and traverse-commit-list that allows both commands to perform the same type of filter operations. And serve as the basis of

[PATCH 05/13] list-objects-filter-large: add large blob filter to list-objects

2017-09-22 Thread Jeff Hostetler
From: Jeff Hostetler Create a filter for traverse_commit_list_worker() to omit blobs larger than a requested size from the result, but always include ".git*" special files. Signed-off-by: Jeff Hostetler @microsoft.com> --- Makefile| 1 + list-objects-filter-

[PATCH 07/13] object-filter: common declarations for object filtering

2017-09-22 Thread Jeff Hostetler
From: Jeff Hostetler Create common routines and defines for parsing object-filter-related command line arguments and pack-protocol fields. Signed-off-by: Jeff Hostetler --- Makefile| 1 + object-filter.c | 269 object-filter.h

[PATCH 06/13] list-objects-filter-sparse: add sparse-checkout based filter

2017-09-22 Thread Jeff Hostetler
From: Jeff Hostetler Create a filter for traverse_commit_list_worker() to only include the blobs the would be referenced by a sparse-checkout using the given specification. A future enhancement should be able to also omit unneeded tree objects, but that is not currently supported. Signed-off

[PATCH 09/13] rev-list: add object filtering support

2017-09-22 Thread Jeff Hostetler
From: Jeff Hostetler Teach rev-list to use the filtering provided by the traverse_commit_list_filtered() interface to omit unwanted objects from the result. This feature is only enabled when one of the "--objects*" options are used. When the "--filter-print-manifest"

[PATCH 10/13] rev-list: add filtering help text

2017-09-22 Thread Jeff Hostetler
From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- Documentation/git-rev-list.txt | 9 - Documentation/rev-list-options.txt | 32 2 files changed, 40 insertions(+), 1 deletion(-) diff --git a/Documentation/git-rev-list.txt b/Documentation/git

[PATCH 08/13] list-objects: add traverse_commit_list_filtered method

2017-09-22 Thread Jeff Hostetler
From: Jeff Hostetler Add traverse_commit_list_filtered() wrapper around the various filter methods using common data in object_filter_options. Signed-off-by: Jeff Hostetler --- list-objects.c | 34 ++ list-objects.h | 11 +++ 2 files changed, 45

[PATCH 12/13] pack-objects: add object filtering support

2017-09-22 Thread Jeff Hostetler
From: Jeff Hostetler Teach pack-objects to use the filtering provided by the traverse_commit_list_filtered() interface to omit unwanted objects from the resulting packfile. This feature is intended for partial clone/fetch. Filtering requires the use of the "--stdout" option. Sig

[PATCH 11/13] t6112: rev-list object filtering test

2017-09-22 Thread Jeff Hostetler
From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- t/t6112-rev-list-filters-objects.sh | 237 1 file changed, 237 insertions(+) create mode 100755 t/t6112-rev-list-filters-objects.sh diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list

[PATCH 13/13] pack-objects: add filtering help text

2017-09-22 Thread Jeff Hostetler
From: Jeff Hostetler Add help text for new object filtering options to pack-objects documentation. Signed-off-by: Jeff Hostetler --- Documentation/git-pack-objects.txt | 17 + 1 file changed, 17 insertions(+) diff --git a/Documentation/git-pack-objects.txt b/Documentation

Re: RFC: Design and code of partial clones (now, missing commits and trees OK)

2017-09-22 Thread Jeff Hostetler
On 9/21/2017 6:42 PM, Jonathan Tan wrote: On Thu, 21 Sep 2017 13:57:30 -0400 Jeff Hostetler wrote: There's a lot in this patch series. I'm still studying it, but here are some notes and questions. I'll start with direct responses to the RFC here and follow up in a se

Re: RFC: Design and code of partial clones (now, missing commits and trees OK) (part 2/3)

2017-09-22 Thread Jeff Hostetler
On 9/21/2017 6:51 PM, Jonathan Tan wrote: On Thu, 21 Sep 2017 13:59:43 -0400 Jeff Hostetler wrote: (part 2) Additional overall comments on: https://github.com/jonathantanmy/git/commits/partialclone2 {} I think it would help to split the blob-max-bytes filtering and the promisor

Re: RFC: Design and code of partial clones (now, missing commits and trees OK) (part 3)

2017-09-22 Thread Jeff Hostetler
On 9/21/2017 7:04 PM, Jonathan Tan wrote: On Thu, 21 Sep 2017 14:00:40 -0400 Jeff Hostetler wrote: (part 3) Additional overall comments on: https://github.com/jonathantanmy/git/commits/partialclone2 {} WRT the code in is_promised() [1] [1] https://github.com/jonathantanmy/git/commit

Re: RFC: Design and code of partial clones (now, missing commits and trees OK) (part 2/3)

2017-09-26 Thread Jeff Hostetler
On 9/22/2017 6:52 PM, Jonathan Tan wrote: On Fri, 22 Sep 2017 17:19:50 -0400 Jeff Hostetler wrote: In your specific example, how would rev-list know, on the client, to include (or exclude) a large blob in its output if it does not have it, and thus does not know its size? The client

Re: RFC: Design and code of partial clones (now, missing commits and trees OK) (part 3)

2017-09-26 Thread Jeff Hostetler
On 9/22/2017 6:58 PM, Jonathan Tan wrote: On Fri, 22 Sep 2017 17:32:00 -0400 Jeff Hostetler wrote: I guess I'm afraid that the first call to is_promised() is going cause a very long pause as it loads up a very large hash of objects. Yes, the first call will cause a long pause. (I

Re: [PATCH] git: add --no-optional-locks option

2017-09-26 Thread Jeff Hostetler
On 9/25/2017 12:17 PM, Johannes Schindelin wrote: Hi Kaartic, On Sun, 24 Sep 2017, Kaartic Sivaraam wrote: On Thursday 21 September 2017 10:02 AM, Jeff King wrote: Some tools like IDEs or fancy editors may periodically run commands like "git status" in the background to keep track of the st

Re: [PATCH 00/13] RFC object filtering for parital clone

2017-09-26 Thread Jeff Hostetler
On 9/22/2017 8:39 PM, Jonathan Tan wrote: On Fri, 22 Sep 2017 20:26:19 + Jeff Hostetler wrote: This draft contains filters to: () omit all blobs () omit blobs larger than some size () omit blobs using a sparse-checkout specification In addition to specifying the filter criteria, the

Re: [PATCH 00/13] RFC object filtering for parital clone

2017-09-26 Thread Jeff Hostetler
On 9/26/2017 10:55 AM, Jeff Hostetler wrote: On 9/22/2017 8:39 PM, Jonathan Tan wrote: On Fri, 22 Sep 2017 20:26:19 + Jeff Hostetler wrote: ... I tried applying your patches and it doesn't apply cleanly on master. Could you try rebasing? In particular, the code refer

Re: [PATCH 02/13] oidset2: create oidset subclass with object length and pathname

2017-09-27 Thread Jeff Hostetler
On 9/26/2017 6:20 PM, Jonathan Tan wrote: On Fri, 22 Sep 2017 20:26:21 + Jeff Hostetler wrote: From: Jeff Hostetler Create subclass of oidset where each entry has a field to store the length of the object's content and an optional pathname. This will be used in a future comm

Re: [PATCH 03/13] list-objects: filter objects in traverse_commit_list

2017-09-27 Thread Jeff Hostetler
On 9/26/2017 6:31 PM, Jonathan Tan wrote: On Fri, 22 Sep 2017 20:26:22 + Jeff Hostetler wrote: From: Jeff Hostetler Create traverse_commit_list_filtered() and add filtering You mention _filtered() here, but this patch contains _worker(). thanks. list-objects.h | 30

Re: [PATCH 07/13] object-filter: common declarations for object filtering

2017-09-27 Thread Jeff Hostetler
On 9/26/2017 6:39 PM, Jonathan Tan wrote: On Fri, 22 Sep 2017 20:30:11 + Jeff Hostetler wrote: Makefile| 1 + object-filter.c | 269 object-filter.h | 173 3 files changed, 443

Re: [PATCH 09/13] rev-list: add object filtering support

2017-09-27 Thread Jeff Hostetler
On 9/26/2017 6:44 PM, Jonathan Tan wrote: On Fri, 22 Sep 2017 20:30:13 + Jeff Hostetler wrote: + if (filter_options.relax) { Add some documentation about how this differs from ignore_missing_links in struct rev_info. It's unclear what that field means. (It&

Re: [PATCH 03/13] list-objects: filter objects in traverse_commit_list

2017-09-27 Thread Jeff Hostetler
On 9/27/2017 2:00 PM, Jonathan Tan wrote: On Wed, 27 Sep 2017 13:04:42 -0400 Jeff Hostetler wrote: The sparse filter is looking at pathnames and using the same rules as sparse-checkout to decide which to *include* in the result. This is essentially backwards from the other filters which

Re: [PATCH 07/13] object-filter: common declarations for object filtering

2017-09-28 Thread Jeff Hostetler
On 9/27/2017 8:05 PM, Jonathan Tan wrote: On Wed, 27 Sep 2017 13:09:42 -0400 Jeff Hostetler wrote: On 9/26/2017 6:39 PM, Jonathan Tan wrote: On Fri, 22 Sep 2017 20:30:11 + Jeff Hostetler wrote: Makefile| 1 + object-filter.c | 269

Re: [idea] File history tracking hints

2017-09-30 Thread Jeff Hostetler
On 9/29/2017 7:12 PM, Johannes Schindelin wrote: Hi Philip, On Fri, 15 Sep 2017, Philip Oakley wrote: From: "Johannes Schindelin" In light of such experiences, I have to admit that the notion that the rename detection can always be improved in hindsight puts quite a bit of insult to injur

Re: [idea] File history tracking hints

2017-10-02 Thread Jeff Hostetler
On 10/2/2017 1:41 PM, Stefan Beller wrote: It would be nice if every file (and tree) had a permanent GUID associated with it. Then the filename/pathname becomes a property of the GUIDs. Then you can exactly know about moves/renames with minimal effort (and no guessing). ... https://publi

Re: [idea] File history tracking hints

2017-10-02 Thread Jeff Hostetler
On 10/2/2017 3:18 PM, Stefan Beller wrote: On Mon, Oct 2, 2017 at 11:51 AM, Jeff Hostetler wrote: Sorry to re-re-...-re-stir up such an old topic. I wasn't really thinking about commit-to-commit hints. I think these have lots of problems. (If commit A->B does "t/* -> tests

Re: [PATCH 00/18] Partial clone (from clone to lazy fetch in 18 patches)

2017-10-03 Thread Jeff Hostetler
On 10/3/2017 4:50 AM, Junio C Hamano wrote: Christian Couder writes: Could you give a bit more details about the use cases this is designed for? It seems that when people review my work they want a lot of details about the use cases, so I guess they would also be interesting in getting this

Re: [PATCH 00/18] Partial clone (from clone to lazy fetch in 18 patches)

2017-10-04 Thread Jeff Hostetler
On 10/3/2017 7:42 PM, Jonathan Tan wrote: On Tue, Oct 3, 2017 at 7:39 AM, Jeff Hostetler wrote: As I see it there are the following major parts to partial clone: 1. How to let git-clone (and later git-fetch) specify the desired subset of objects that it wants? (A ref-relative request

[PATCH 02/13] list-objects-filter-map: extend oidmap to collect omitted objects

2017-10-24 Thread Jeff Hostetler
From: Jeff Hostetler Create helper class to extend oidmap to collect a list of omitted or missing objects during traversal. This will be used in a later commit by the list-object filtering code. Signed-off-by: Jeff Hostetler --- Makefile | 1 + list-objects-filter-map.c

[PATCH 00/13] WIP Partial clone part 1: object filtering

2017-10-24 Thread Jeff Hostetler
From: Jeff Hostetler I've been working with Jonathan Tan to combine our partial clone proposals. This patch series represents a first step in that effort and introduces an object filtering mechanism to select unwanted objects. [1] traverse_commit_list and list-objects is extended to

[PATCH 06/13] list-objects-filter-sparse: add sparse filter

2017-10-24 Thread Jeff Hostetler
From: Jeff Hostetler Create a filter for traverse_commit_list_worker() to only include the blobs the would be referenced by a sparse-checkout using the given specification. Signed-off-by: Jeff Hostetler --- Makefile | 1 + list-objects-filter-sparse.c | 241

[PATCH 08/13] list-objects: add traverse_commit_list_filtered method

2017-10-24 Thread Jeff Hostetler
From: Jeff Hostetler Add traverse_commit_list_filtered() wrapper around the various filter methods using common data in object_filter_options. Signed-off-by: Jeff Hostetler --- list-objects.c | 45 + list-objects.h | 11 +++ 2 files changed

[PATCH 03/13] list-objects: filter objects in traverse_commit_list

2017-10-24 Thread Jeff Hostetler
From: Jeff Hostetler Create traverse_commit_list_filtered() and add filtering interface to allow certain objects to be omitted (not shown) during a traversal. Update traverse_commit_list() to be a wrapper for the above. Filtering will be used in a future commit by rev-list and pack-objects for

[PATCH 07/13] list-objects-filter-options: common argument parsing

2017-10-24 Thread Jeff Hostetler
From: Jeff Hostetler Create common routines and defines for parsing list-objects-filter-related command line arguments and pack-protocol fields. Signed-off-by: Jeff Hostetler --- Makefile | 1 + list-objects-filter-options.c | 101

[PATCH 11/13] t6112: rev-list object filtering test

2017-10-24 Thread Jeff Hostetler
From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- t/t6112-rev-list-filters-objects.sh | 223 1 file changed, 223 insertions(+) create mode 100755 t/t6112-rev-list-filters-objects.sh diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list

[PATCH 01/13] dir: allow exclusions from blob in addition to file

2017-10-24 Thread Jeff Hostetler
From: Jeff Hostetler Refactor add_excludes() to separate the reading of the exclude file into a buffer and the parsing of the buffer into exclude_list items. Add add_excludes_from_blob_to_list() to allow an exclude file be specified with an OID. Signed-off-by: Jeff Hostetler --- dir.c | 51

[PATCH 09/13] extension.partialclone: introduce partial clone extension

2017-10-24 Thread Jeff Hostetler
From: Jeff Hostetler Introduce the ability to have missing objects in a repo. This functionality is guarded by new repository extension options: `extensions.partialcloneremote` and `extensions.partialclonefilter`. See the update to Documentation/technical/repository-version.txt in this

[PATCH 05/13] list-objects-filter-blobs-limit: add large blob filtering

2017-10-24 Thread Jeff Hostetler
From: Jeff Hostetler Create a filter for traverse_commit_list_worker() to omit blobs larger than a requested size from the result, but always include ".git*" special files. Signed-off-by: Jeff Hostetler --- Makefile | 1 + list-objects-filter-blobs-lim

[PATCH 04/13] list-objects-filter-blobs-none: add filter to omit all blobs

2017-10-24 Thread Jeff Hostetler
From: Jeff Hostetler Create a simple filter for traverse_commit_list_worker() to omit all blobs from the result. This filter will be used in a future commit by rev-list and pack-objects to create a "commits and trees" result. This is intended for partial clone and fetch support. Sig

[PATCH 13/13] t5317: pack-objects object filtering test

2017-10-24 Thread Jeff Hostetler
From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- t/t5317-pack-objects-filter-objects.sh | 384 + 1 file changed, 384 insertions(+) create mode 100755 t/t5317-pack-objects-filter-objects.sh diff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317

[PATCH 10/13] rev-list: add list-objects filtering support

2017-10-24 Thread Jeff Hostetler
From: Jeff Hostetler Teach rev-list to use the filtering provided by the traverse_commit_list_filtered() interface to omit unwanted objects from the result. This feature is only enabled when one of the "--objects*" options are used. Furthermore, when the "--filter-print-omitted&

[PATCH 12/13] pack-objects: add list-objects filtering

2017-10-24 Thread Jeff Hostetler
From: Jeff Hostetler Teach pack-objects to use the filtering provided by the traverse_commit_list_filtered() interface to omit unwanted objects from the resulting packfile. This feature is intended for partial clone/fetch. Filtering requires the use of the "--stdout" option. Sig

Re: [PATCH 01/13] dir: allow exclusions from blob in addition to file

2017-10-25 Thread Jeff Hostetler
On 10/25/2017 2:43 AM, Junio C Hamano wrote: Jeff Hostetler writes: +static int add_excludes_from_buffer(char *buf, size_t size, + const char *base, int baselen, + struct exclude_list *el); + /* * Given a file with

Re: [PATCH 00/13] WIP Partial clone part 1: object filtering

2017-10-25 Thread Jeff Hostetler
nk that we can and we should, as I wrote in my reply to Jeff Hostetler [1]. As for how this patch set (excluding the partialclone patch) interacts with my fsck series, they are relatively independent, as far as I can tell. I'll rebase my fsck, gc, and lazy object fetch patches (but not the fet

Re: [PATCH 02/13] list-objects-filter-map: extend oidmap to collect omitted objects

2017-10-25 Thread Jeff Hostetler
On 10/25/2017 3:10 AM, Junio C Hamano wrote: Jeff Hostetler writes: From: Jeff Hostetler Create helper class to extend oidmap to collect a list of omitted or missing objects during traversal. The reason why oidmap itself cannot be used is because the code wants to record not just the

Re: [PATCH 03/13] list-objects: filter objects in traverse_commit_list

2017-10-25 Thread Jeff Hostetler
On 10/25/2017 12:05 AM, Jonathan Tan wrote: On Tue, Oct 24, 2017 at 11:53 AM, Jeff Hostetler wrote: +enum list_objects_filter_result { + LOFR_ZERO = 0, + LOFR_MARK_SEEN = 1<<0, Probably worth documenting, something like /* Mark this object so that it is skipped f

Re: [PATCH 07/13] list-objects-filter-options: common argument parsing

2017-10-25 Thread Jeff Hostetler
On 10/25/2017 12:14 AM, Jonathan Tan wrote: On Tue, Oct 24, 2017 at 11:53 AM, Jeff Hostetler wrote: + * ::= blob:none + * blob:limit:[kmg] + * sparse:oid: + * sparse:path: I notice in the code below that there are some usages of "=" instead of "

Re: [PATCH 08/13] list-objects: add traverse_commit_list_filtered method

2017-10-25 Thread Jeff Hostetler
On 10/25/2017 12:24 AM, Jonathan Tan wrote: On Tue, Oct 24, 2017 at 11:53 AM, Jeff Hostetler wrote: +void traverse_commit_list_filtered( + struct list_objects_filter_options *filter_options, + struct rev_info *revs, + show_commit_fn show_commit, + show_object_fn

Re: [PATCH 10/13] rev-list: add list-objects filtering support

2017-10-25 Thread Jeff Hostetler
On 10/25/2017 12:41 AM, Jonathan Tan wrote: On Tue, Oct 24, 2017 at 11:53 AM, Jeff Hostetler wrote: static void finish_object(struct object *obj, const char *name, void *cb_data) { struct rev_list_info *info = cb_data; - if (obj->type == OBJ_BLOB && !has_objec

[PATCH] dir: allow exclusions from blob in addition to file

2017-10-26 Thread Jeff Hostetler
From: Jeff Hostetler Refactor add_excludes() to separate the reading of the exclude file into a buffer and the parsing of the buffer into exclude_list items. Add add_excludes_from_blob_to_list() to allow an exclude file be specified with an OID without assuming a local worktree or index exists

[PATCH] dir: allow exclusions from blob in addition to file

2017-10-26 Thread Jeff Hostetler
From: Jeff Hostetler I pulled commit 01/13 from Tuesday's partial clone part 1 patch series [1] and refactored the changes in dir.c to try to address Junio's comments in [2] WRT sharing more code with the existing read_skip_worktree_file_from_index(). This patch can be discussed ind

Re: [PATCH 01/13] dir: allow exclusions from blob in addition to file

2017-10-26 Thread Jeff Hostetler
On 10/25/2017 11:47 PM, Junio C Hamano wrote: Jeff Hostetler writes: The existing code handles use cases where you want to read the exclusion list from a pathname in the worktree -- or from blob named in the index when the pathname is not populated (presumably because of the skip-worktree

Re: [PATCH] dir: allow exclusions from blob in addition to file

2017-10-27 Thread Jeff Hostetler
On 10/26/2017 9:20 PM, Junio C Hamano wrote: Jeff Hostetler writes: From: Jeff Hostetler Refactor add_excludes() to separate the reading of the exclude file into a buffer and the parsing of the buffer into exclude_list items. Add add_excludes_from_blob_to_list() to allow an exclude file

Re: What's cooking in git.git (Oct 2017, #07; Mon, 30)

2017-10-30 Thread Jeff Hostetler
On 10/30/2017 1:31 PM, Johannes Schindelin wrote: Hi Junio, On Mon, 30 Oct 2017, Junio C Hamano wrote: * jt/partial-clone-lazy-fetch (2017-10-02) 18 commits - fetch-pack: restore save_commit_buffer after use - unpack-trees: batch fetching of missing blobs - clone: configure blobmaxbyte

RE: What's cooking in git.git (Oct 2017, #07; Mon, 30)

2017-10-31 Thread Jeff Hostetler
From: Junio C Hamano [mailto:gits...@pobox.com] Subject: Re: What's cooking in git.git (Oct 2017, #07; Mon, 30) > Jeff Hostetler writes: > >> I've been assuming that the jt/partial-clone-lazy-fetch is a >> placeholder for our next combined patch series. >

[PATCH v2 1/6] dir: allow exclusions from blob in addition to file

2017-11-02 Thread Jeff Hostetler
From: Jeff Hostetler Refactor add_excludes() to separate the reading of the exclude file into a buffer and the parsing of the buffer into exclude_list items. Add add_excludes_from_blob_to_list() to allow an exclude file be specified with an OID without assuming a local worktree or index exists

[PATCH v2 0/6] Partial clone part 1: object filtering

2017-11-02 Thread Jeff Hostetler
From: Jeff Hostetler Here is V2 of the list-object filtering. It replaces [1] and reflect a refactoring and simplification of the original. After much discussion on the "list-object-filter-map" I've replaced it with a regular oidset -- the only need for the map was to store the

[PATCH v2 2/6] oidmap: add oidmap iterator methods

2017-11-02 Thread Jeff Hostetler
From: Jeff Hostetler Add the usual map iterator functions to oidmap. Signed-off-by: Jeff Hostetler --- oidmap.h | 22 ++ 1 file changed, 22 insertions(+) diff --git a/oidmap.h b/oidmap.h index 18f54cd..d3cd2bb 100644 --- a/oidmap.h +++ b/oidmap.h @@ -65,4 +65,26 @@ extern

[PATCH v2 6/6] pack-objects: add list-objects filtering

2017-11-02 Thread Jeff Hostetler
From: Jeff Hostetler Teach pack-objects to use the filtering provided by the traverse_commit_list_filtered() interface to omit unwanted objects from the resulting packfile. This feature is intended for partial clone/fetch. Filtering requires the use of the "--stdout" option. Add

[PATCH v2 5/6] rev-list: add list-objects filtering support

2017-11-02 Thread Jeff Hostetler
From: Jeff Hostetler Teach rev-list to use the filtering provided by the traverse_commit_list_filtered() interface to omit unwanted objects from the result. This feature is intended to help with partial clone. Object filtering is only allowed when one of the "--objects*" option

[PATCH v2 3/6] oidset: add iterator methods to oidset

2017-11-02 Thread Jeff Hostetler
From: Jeff Hostetler Add the usual iterator methods to oidset. Add oidset_remove(). Signed-off-by: Jeff Hostetler --- oidset.c | 10 ++ oidset.h | 36 2 files changed, 46 insertions(+) diff --git a/oidset.c b/oidset.c index f1f874a..454c54f 100644

[PATCH v2 4/6] list-objects: filter objects in traverse_commit_list

2017-11-02 Thread Jeff Hostetler
From: Jeff Hostetler Create traverse_commit_list_filtered() and add filtering interface to allow certain objects to be omitted from the traversal. Update traverse_commit_list() to be a wrapper for the above with a null filter to minimize the number of callers that needed to be changed. Object

[PATCH 0/9] WIP Partial clone part 2: fsck and promisors

2017-11-02 Thread Jeff Hostetler
From: Jeff Hostetler This is part 2 of a proposed 3 part sequence for partial clone. Part 2 assumes part 1 [1] is in place. Part 2 is concerned with fsck, gc, initial support for dynamic object fetching, and tracking promisor objects. Jonathan Tan originally developed this code. I have moved

[PATCH 4/9] fsck: support referenced promisor objects

2017-11-02 Thread Jeff Hostetler
From: Jonathan Tan Teach fsck to not treat missing promisor objects indirectly pointed to by refs as an error when extensions.partialcloneremote is set. Signed-off-by: Jonathan Tan Signed-off-by: Jeff Hostetler --- builtin/fsck.c | 11 +++ t/t0410-partial-clone.sh | 23

[PATCH 9/9] gc: do not repack promisor packfiles

2017-11-02 Thread Jeff Hostetler
: Jeff Hostetler --- Documentation/git-pack-objects.txt | 4 +++ builtin/gc.c | 4 +++ builtin/pack-objects.c | 14 ++ builtin/prune.c| 7 + builtin/repack.c | 12 +++-- t/t0410-partial-clone.sh

[PATCH 5/9] fsck: support promisor objects as CLI argument

2017-11-02 Thread Jeff Hostetler
From: Jonathan Tan Teach fsck to not treat missing promisor objects provided on the CLI as an error when extensions.partialcloneremote is set. Signed-off-by: Jonathan Tan Signed-off-by: Jeff Hostetler --- builtin/fsck.c | 2 ++ t/t0410-partial-clone.sh | 13 + 2 files

[PATCH 7/9] introduce fetch-object: fetch one promisor object

2017-11-02 Thread Jeff Hostetler
ement and officially allow any object type to be "want"-ed. Signed-off-by: Jonathan Tan Signed-off-by: Jeff Hostetler --- Documentation/gitremote-helpers.txt | 2 ++ Makefile| 1 + builtin/fetch-pack.c| 8 builtin/index-pack.c

[PATCH 2/9] fsck: introduce partialclone extension

2017-11-02 Thread Jeff Hostetler
commits, fsck will be taught about other cases. Signed-off-by: Jonathan Tan Signed-off-by: Jeff Hostetler --- builtin/fsck.c | 2 +- cache.h | 3 +- packfile.c | 78 -- packfile.h | 13

[PATCH 1/9] extension.partialclone: introduce partial clone extension

2017-11-02 Thread Jeff Hostetler
From: Jeff Hostetler Introduce the ability to have missing objects in a repo. This functionality is guarded by new repository extension options: `extensions.partialcloneremote` and `extensions.partialclonefilter`. See the update to Documentation/technical/repository-version.txt in this

[PATCH 6/9] index-pack: refactor writing of .keep files

2017-11-02 Thread Jeff Hostetler
becomes easier. Signed-off-by: Jonathan Tan Signed-off-by: Jeff Hostetler --- builtin/index-pack.c | 99 1 file changed, 53 insertions(+), 46 deletions(-) diff --git a/builtin/index-pack.c b/builtin/index-pack.c index 8ec459f..4f305a7 100644

[PATCH 8/9] sha1_file: support lazily fetching missing objects

2017-11-02 Thread Jeff Hostetler
is also packed) - builtin/prune-packed - check if object to be pruned is packed (if not, don't prune it) - revision - used to exclude packed objects if requested by user - diff - just for optimization Signed-off-by: Jonathan Tan Signed-off-by: Jeff Hostetler --- builtin/cat-f

[PATCH 3/9] fsck: support refs pointing to promisor objects

2017-11-02 Thread Jeff Hostetler
From: Jonathan Tan Teach fsck to not treat refs referring to missing promisor objects as an error when extensions.partialclone is set. For the purposes of warning about no default refs, such refs are still treated as legitimate refs. Signed-off-by: Jonathan Tan Signed-off-by: Jeff Hostetler

[PATCH 05/14] remote-curl: add object filtering for partial clone

2017-11-02 Thread Jeff Hostetler
From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- Documentation/gitremote-helpers.txt | 4 remote-curl.c | 10 -- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/Documentation/gitremote-helpers.txt b/Documentation/gitremote-helpers.txt

[PATCH 08/14] fetch: add from_promisor and exclude-promisor-objects parameters

2017-11-02 Thread Jeff Hostetler
From: Jonathan Tan Teach fetch to use from-promisor and exclude-promisor-objects parameters with sub-commands. Initialize fetch_if_missing global variable. Signed-off-by: Jonathan Tan Signed-off-by: Jeff Hostetler --- builtin/fetch.c | 9 ++--- connected.c | 3 +++ 2 files changed

[PATCH 10/14] t5601: test for partial clone

2017-11-02 Thread Jeff Hostetler
From: Jonathan Tan Signed-off-by: Jonathan Tan Signed-off-by: Jeff Hostetler --- builtin/clone.c | 17 ++--- t/t5601-clone.sh | 49 + 2 files changed, 63 insertions(+), 3 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c

[PATCH 14/14] index-pack: silently assume missing objects are promisor

2017-11-02 Thread Jeff Hostetler
From: Jeff Hostetler Teach index-pack to not complain about missing objects when the --promisor flag is given. The assumption is that index-pack is currently building the idx and promisor data and the is_promisor_object() query would fail anyway. Signed-off-by: Jeff Hostetler --- builtin

<    2   3   4   5   6   7   8   9   10   11   >