Junio C Hamano wrote:
> Eric Wong writes:
>
> >> [primeclone]
> >>url = http://location/pack-$NAME.pack
> >>filetype = pack
> >
> > If unconfigured, I wonder if a primeclone pack can be inferred by
> > the existence of a pack bitmap (or merely being the biggest+oldest
> > pack for dumb H
un "git checkout -b" to create an initial checkout; this step
> would require us to have parsed the command line for --branch,
> etc.
>
> Even though the current code conceptually does the above, these
> steps are not cleanly separated as such. I think our update
equire us to have parsed the command line for --branch,
etc.
Even though the current code conceptually does the above, these
steps are not cleanly separated as such. I think our update to gain
"resumable clone" feature on the client side need to start by
refactoring the curre
On Mon, Sep 19, 2016 at 08:52:34PM +0700, Duy Nguyen wrote:
>
> A brief overview for this service in
> Documentation/technical/http-protocol.txt (and maybe
> Documentation/gitremote-helpers.txt as well) would be great help. It's
> a bit hard to follow because at this point I don't know anything ab
On Fri, Sep 16, 2016 at 03:45:44PM -0700, Junio C Hamano wrote:
> > @@ -1136,7 +1138,10 @@ static int handle_curl_result(struct slot_results
> > *results)
> > curl_easy_strerror(results->curl_result),
> > sizeof(curl_errorstr));
> > #endif
>
ote_refs(const struct ref
> > *local_refs)
> > die("%s", err.buf);
> >
> > for (r = local_refs; r; r = r->next) {
> > - if (!r->peer_ref)
> > + if (!r->peer_ref || ref_exists(r->peer_ref->name))
> >
er
> to follow if you separate out some code move (I think I've seen some,
> not sure). I'll try to have another look when I find time. But it's
> great to hear from you again, the pleasant surprise in my inbox today,
> as I thought we lost you ;-) There's hope for
On Mon, Sep 19, 2016 at 08:15:00PM +0700, Duy Nguyen wrote:
> We also have an exception for select_getanyfile() below. I think it's
> time we add a function callback in struct rpc_service to run each
> service the way they want. Then prime-clone won't need an exception
> (neither does select_anyfil
On Fri, Sep 16, 2016 at 01:53:15PM -0700, Junio C Hamano wrote:
> Kevin Wern writes:
>
> > Create git-prime-clone, a program to be executed on the server that
> > returns the location and type of static resource to download before
> > performing the rest of a clone.
> >
> > Additionally, as this
Eric Wong writes:
>> [primeclone]
>> url = http://location/pack-$NAME.pack
>> filetype = pack
>
> If unconfigured, I wonder if a primeclone pack can be inferred by
> the existence of a pack bitmap (or merely being the biggest+oldest
> pack for dumb HTTP).
That would probably be a nice
Kevin Wern wrote:
> Hey, all,
>
> It's been a while (sent a very short patch in May), but I've
> still been working on the resumable clone feature and checking up on
> the mailing list for any updates. After submitting the prime-clone
> service alone, I figured implem
code move (I think I've seen some,
> not sure). I'll try to have another look when I find time. But it's
> great to hear from you again, the pleasant surprise in my inbox today,
> as I thought we lost you ;-) There's hope for resumable clone maybe
> before 2018 aga
). I'll try to have another look when I find time. But it's
great to hear from you again, the pleasant surprise in my inbox today,
as I thought we lost you ;-) There's hope for resumable clone maybe
before 2018 again.
--
Duy
On Fri, Sep 16, 2016 at 7:12 AM, Kevin Wern wrote:
> +static void prime_clone(void)
> +{
> + char *result, *result_full, *line;
> + size_t result_len;
> + int err = 0, one_successful = 0;
> +
> + if (request_service("git-prime-clone", &result_full, &result,
> +
On Fri, Sep 16, 2016 at 7:12 AM, Kevin Wern wrote:
> static struct daemon_service daemon_service[] = {
> { "upload-archive", "uploadarch", upload_archive, 0, 1 },
> { "upload-pack", "uploadpack", upload_pack, 1, 1 },
> { "receive-pack", "receivepack", receive_pack, 0, 1 },
Kevin Wern writes:
> Use transport_download_primer and transport_prime_clone in git clone.
> This only supports a fully connected packfile.
>
> transport_prime_clone and transport_download_primer are executed
> completely independent of transport_(get|fetch)_remote_refs, et al.
> transport_downlo
Kevin Wern writes:
> +int http_download_primer(const char *url, const char *out_file)
> +{
> + int ret = 0, try_count = HTTP_TRY_COUNT;
> + struct http_get_options options = {0};
> + options.progress = 1;
> +
> + if (file_exists(out_file)) {
> + fprintf(stderr,
> +
Kevin Wern writes:
> Create git-prime-clone, a program to be executed on the server that
> returns the location and type of static resource to download before
> performing the rest of a clone.
>
> Additionally, as this executable's location will be configurable (see:
> upload-pack and receive-pac
Kevin Wern writes:
> It's been a while (sent a very short patch in May), but I've
> still been working on the resumable clone feature and checking up on
> the mailing list for any updates. After submitting the prime-clone
> service alone, I figured implementing the whole t
Add method for transport to call when parsing primeclone output from
stdin. Suppress stderr when using git_connect with ssh, unless output
is verbose.
Signed-off-by: Kevin Wern
---
connect.c | 47 +++
connect.h | 10 ++
2 files changed, 53 inse
Create transport_prime_clone API, as well as all internal methods.
Create representations of alt_resource and prime-clone path options.
The intention of get_alt_res_helper is solely to parse the output of
remote-curl because transport-helper does not handle verbose options
or speaking to the user
Create resumable download procedure and progress display function.
The conversion from B to KB occurs because otherwise the byte counts
for large repos (i.e. Linux) overflow calculating percentage.
The download protocol includes the resource's URL, and the directory
the resource will be downloaded
t;)),
+ OPT_STRING('p', "prime-clone", &option_prime_clone, N_("path"),
+ N_("path to git-prime-clone on the remote")),
OPT_STRING(0, "depth", &option_depth, N_("depth"),
N_(
Create function transport_download_primer and components
to invoke and pass commands to remote-curl.
Signed-off-by: Kevin Wern
---
transport-helper.c | 24
transport.c| 9 +
transport.h| 7 +++
3 files changed, 40 insertions(+)
diff --git a
Add function and interface to handle prime-clone input, extracting
and using duplicate functionality from discover_refs as function
request_service.
Because part of our goal is for prime_clone to recover from errors,
HTTP errors are only optionally printed to screen and never cause
death in this c
Create git-prime-clone, a program to be executed on the server that
returns the location and type of static resource to download before
performing the rest of a clone.
Additionally, as this executable's location will be configurable (see:
upload-pack and receive-pack), add the program to
BINDIR_PR
Add logic to serve git-prime-clone to git and http clients.
Do not pass --stateless-rpc and --advertise-refs options to
prime-clone. It is inherently stateless and an 'advertisement'.
Signed-off-by: Kevin Wern
---
Documentation/git-daemon.txt | 7 +++
Documentation/git-http-backend.t
Hey, all,
It's been a while (sent a very short patch in May), but I've
still been working on the resumable clone feature and checking up on
the mailing list for any updates. After submitting the prime-clone
service alone, I figured implementing the whole thing would be the best
way to
Create a bare-bones version of git-prime-clone, which returns the
location of an alternate resource specified by the server that the
client should fetch and build before returning to perform an incremental
fetch.
At this point, no validation is performed of the file's existence, the
file's validit
Add endpoints for prime-clone in http service and git daemon
---
daemon.c | 7 +++
http-backend.c | 1 +
2 files changed, 8 insertions(+)
diff --git a/daemon.c b/daemon.c
index 8d45c33..2ddc7f7 100644
--- a/daemon.c
+++ b/daemon.c
@@ -475,10 +475,17 @@ static int receive_pack(void)
Junio,
> Yes, that is very close to what I said in the "what remains?"
> section, but with a crucial difference in a detail. Perhaps reading
> the message you are respoinding to again more carefully will clear
> the confusion. This is what we want to allow the server to say
> (from the message y
Duy Nguyen writes:
> Yeah the determination is tricky, it depends on server setup. Let's
> start with select the pack for download first because there could be
> many of them. A heuristic (*) of choosing the biggest one in
> $GIT_DIR/objects/pack is probably ok for now (we don't need full
> histo
packfile contains
> the object it needs.
You sound as if you are describing how a fetch over the dumb commit
walker http transport works. That does not have anything to do with
the discussion of resumable clone, though, so I am not sure where
you are going with this.
> What I believe *doesn&
On Tue, Mar 8, 2016 at 10:33 AM, Kevin Wern wrote:
> Hey Junio and Duy,
>
> Thank you for your thorough responses! I'm new to git dev, so it's
> extremely helpful.
>
>> - The server side endpoint does not have to be, and I think it
>> should not be, implemented as an extension to the current
>> up
Hey Junio and Duy,
Thank you for your thorough responses! I'm new to git dev, so it's
extremely helpful.
> - The server side endpoint does not have to be, and I think it
> should not be, implemented as an extension to the current
> upload-pack protocol. It is perfectly fine to add a new "git
> pr
Johannes Schindelin writes:
> First of all: my main gripe with the discussed approach is that it uses
> bundles. I know, I introduced bundles, but they just seem too klunky and
> too static for the resumable clone feature.
We should make the mechanism extensible so that we can late
Duy Nguyen writes:
> One thing Junio didn't mention in his summary is the use of pack
> bitmap [1]. Jeff talked about GitHub specific needs,...
Do not take it as the "summary of the whole discussion".
I deliberately tried to limit the list to absolute minimum to allow
building a workable initia
On Sun, Mar 6, 2016 at 3:49 PM, Duy Nguyen wrote:
> One thing Junio didn't mention in his summary is the use of pack
> bitmap [1].
>
> [1] http://thread.gmane.org/gmane.comp.version-control.git/288205/focus=288222
Oops, wrong link. Should be this one
http://article.gmane.org/gmane.comp.vers
On Sun, Mar 6, 2016 at 2:59 PM, Johannes Schindelin
wrote:
> First of all: my main gripe with the discussed approach is that it uses
> bundles. I know, I introduced bundles, but they just seem too klunky and
> too static for the resumable clone feature.
One thing Junio didn't
Hi Junio & Duy,
On Sat, 5 Mar 2016, Junio C Hamano wrote:
> Duy Nguyen writes:
>
> > Resumable clone is happening. See [1] for the basic idea, [2] and [3]
> > for some preparation work. I'm sure you can help. Once you've gone
> > through at least [1
Junio C Hamano writes:
> So what remains? Here is a rough and still slushy outline:
>
> - A new method, prime_clone(), in "struct transport" for "git
>clone" client to first call to learn the location of the
>"alternate resource" from the server.
>
>...
>- The format of the retu
Duy Nguyen writes:
> Resumable clone is happening. See [1] for the basic idea, [2] and [3]
> for some preparation work. I'm sure you can help. Once you've gone
> through at least [1], I think you can pick something (e.g. finalizing
> the protocol, update the server side, or
anize the feature when writing patches?
Resumable clone is happening. See [1] for the basic idea, [2] and [3]
for some preparation work. I'm sure you can help. Once you've gone
through at least [1], I think you can pick something (e.g. finalizing
the protocol, update the server side
Hey, all,
A while ago, I noticed that if a clone is interrupted, it has to be
restarted completely. When I looked up the issue, I saw that someone
suggested resumable clones as a feature as early as 2007. It doesn't
seem like any progress has been made on this issue, so I began looking
at relevan
On Sun, Feb 14, 2016 at 9:05 AM, Jeff King wrote:
>
> I'm also not happy about having an HTTP-only feature in the protocol. I
> liked Stefan's proposal for the "v2" protocol that would let the two
> sides exchange capabilities before the ref advertisement. Then the
> client, having seen the server
= it can shrink the ref advertisement to 1
>> line containing the capabilities.
>
> I'm slightly wary of this. The client advertising "resumable=1" does not
> mean "I will definitely do a resumable clone". It means "I support the
> resumable keyword, an
slightly wary of this. The client advertising "resumable=1" does not
mean "I will definitely do a resumable clone". It means "I support the
resumable keyword, and if you share a resumable URL with me, I _might_
use it, depending on things that are none of your business"
oday and thought a
>> variation makes more sense:
>>
>> 1. Clients attempting clone ask for /info/refs?service=git-upload-pack
>> like they do today.
>>
>> 2. Servers that support resumable clone include a "resumable"
>> capability in the advertis
On Fri, Feb 12, 2016 at 07:40:43PM -0600, Blake Burkhart wrote:
> On Wed, Feb 10, 2016 at 3:49 PM, Jeff King wrote:
> >> 2. Servers that support resumable clone include a "resumable"
> >> capability in the advertisement.
> >
> > Because the magic happen
Duy Nguyen writes:
> On Thu, Feb 11, 2016 at 4:01 AM, Jonathan Nieder wrote:
>>
>>
>> I really like this design. I'm tempted to implement it (since it
>> lacks a bunch of the downsides of clone.bundle).
>
> A bit disappointed we still do not address resumable fetch.
As you already said, t
On Wed, Feb 10, 2016 at 3:49 PM, Jeff King wrote:
>> 2. Servers that support resumable clone include a "resumable"
>> capability in the advertisement.
>
> Because the magic happens in the git protocol, that would mean this does
> not have to be limited to git-ov
people excited and one of them even tempted to
implement. This thread sounds like we'll see resumable clone before
2017 :)
--
Duy
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
think it would be helpful for normal bundle use.
It _could_ be helpful in the context we're talking about here, though.
If I create the split-bundle so that people can resumable-clone from me,
they can only clone up to that bundle's creation point (and
incrementally fetch the rest). But
Jeff King writes:
> ... One
> alternative would be to amend the bundle format so that rather than a
> single file, you get a bundle header whose end says "...and my matching
> packfile is 1234-abcd". And then the client knows that they can fetch
> that separately from the same source.
I would im
On Wed, Feb 10, 2016 at 7:43 PM, Junio C Hamano wrote:
> Jonathan Nieder writes:
>
>> I really like this design. I'm tempted to implement it (since it
>> lacks a bunch of the downsides of clone.bundle).
>
> Just to see people are not stepping on each others toe, implementing
> slightly different
Jonathan Nieder writes:
> I really like this design. I'm tempted to implement it (since it
> lacks a bunch of the downsides of clone.bundle).
Just to see people are not stepping on each others toe, implementing
slightly different components in parallel based on the same theme to
solve the same
On Wed, Feb 10, 2016 at 02:17:58PM -0800, Jonathan Nieder wrote:
> > Because the magic happens in the git protocol, that would mean this does
> > not have to be limited to git-over-http. It could be "resumable="
> > to point the client anywhere (the same server over a different protocol,
> > anoth
Jeff King writes:
> Clients do not have to _just_ fetch a packfile. They could get a bundle
> file that contains the roots along with the packfile. I know that one of
> your goals is not duplicating the storage of the packfile on the server,
> but it would not be hard for the server to store the
;> like they do today.
>>
>> 2. Servers that support resumable clone include a "resumable"
>> capability in the advertisement.
>
> Because the magic happens in the git protocol, that would mean this does
> not have to be limited to git-over-http. It could be "r
On Wed, Feb 10, 2016 at 01:22:07PM -0800, Jonathan Nieder wrote:
> > I am not quite sure if that is an advantage, though. The second
> > message proposes that the lost-found computation to be done by the
> > client using *.pack, but any client, given the same *.pack, will
> > compute the same res
pting clone ask for /info/refs?service=git-upload-pack
> like they do today.
>
> 2. Servers that support resumable clone include a "resumable"
> capability in the advertisement.
Because the magic happens in the git protocol, that would mean this does
not have to be limited to
Junio C Hamano wrote:
> I somehow doubt it. Both index-pack and lost-found need to trace
> "object A depends on object B", but the similarity ends there.
You already responded about this in another side-thread, so the
following is mostly redundant. :) I think this message can still be
useful to
Jonathan Nieder writes:
> --strict + --check-self-contained-and-connected check that the pack
> is self-contained. In the process they mark each object that is
> reachable from another object in the pack with FLAG_LINK.
>
> The objects not marked with FLAG_LINK are the roots.
Ahh, OK, if we alr
e they do today.
>>
>> 2. Servers that support resumable clone include a "resumable"
>> capability in the advertisement.
>
> like "resumable-token=hash" similar to a push cert advertisement?
It could just be the string 'resumable'.
But I
Stefan Beller writes:
>> 6. Once stored and indexed with .idx, clients run `git fsck
>> --lost-found` to discover the roots of the pack it downloaded. These
>> are saved as temporary references.
>
> jrn:
>> I suspect we can do even faster by making index-pack do the work
I somehow doubt it. Bot
ask for /info/refs?service=git-upload-pack
> like they do today.
>
> 2. Servers that support resumable clone include a "resumable"
> capability in the advertisement.
like "resumable-token=hash" similar to a push cert advertisement?
>
> 3. Updated clients on c
umable clone include a "resumable"
capability in the advertisement.
3. Updated clients on clone request GET /info/refs?service=git-resumable-clone.
4. The server may return a 302 Redirect to its current "mostly whole"
pack file. This can be more flexible than "refs/heads/*
Sorry, no code, only words today. Previously people have proposed a
few different options for resumable clone, like "clone.bundle"
(currently used by Android as a hack) or "skip start of regenerated
pack". Here is another option.
We could implement resumable clone by making
I was reminded by LWN [1] about this. Annoyed in fact because it's
called a bug while it looks more like an elephant. So let's see if we
can agree this is a good way to go.
This is the stable pack approach. The client initiates "resumable
mode" and forces the server to generate a resumable, even t
On Tue, Mar 3, 2015 at 6:47 AM, Junio C Hamano wrote:
> Koosha Khajehmoogahi writes:
>
>> Among GSoC 2011 ideas of git [1], it was proposed that a GSoC project
>> could be implementing resumable clone for git. AFAIK, this feature is
>> still missing in git but cou
Koosha Khajehmoogahi writes:
> Among GSoC 2011 ideas of git [1], it was proposed that a GSoC project
> could be implementing resumable clone for git. AFAIK, this feature is
> still missing in git but could be a great idea to be implemented. Does
> that sound OK to the community?
&g
Among GSoC 2011 ideas of git [1], it was proposed that a GSoC project
could be implementing resumable clone for git. AFAIK, this feature is
still missing in git but could be a great idea to be implemented. Does
that sound OK to the community?
[1]: https://git.wiki.kernel.org/index.php
72 matches
Mail list logo
