Re: [gentoo-dev] The infinite git migration
On Wed, Jun 11, 2014 at 5:59 AM, Patrick Lauer wrote: > Another part: Git wasn't ready. > The first migration attempt failed after consuming nearly 100GB of RAM! > When it did work it took obscene amounts of time, and the result was > unusably large (e.g. initial checkout would take 16GB RAM on the server, > thus not allowing a few hundred devs to do checkouts the same day). > The current state is almost usable, but it is still obscenely slow (e.g. > initial clone taking ~10 CPU-minutes just to figure out what to do), but > we can just throw more hardware at it. > (10 minutes @ 3.6Ghz, so on my notebook it'll take about 4h to just > clone the friggin repository. Too awesome!) Since v1.9.0 we can clone from a shallow repository. We can host two repos on the server: a full repo and a shallow one, containing history of only last year. Most of the time spent in initial clone is to verify the history. Shorter history would shorten that time. But you need to try out to see how long it actually is. I'm not sure if that 16GB includes cloning, or just plain checkout. If the latter, Git has a problem. -- Duy
Re: [gentoo-dev] The infinite git migration
On Wed, Jun 11, 2014 at 4:38 PM, Sergey Popov wrote: > 11.06.2014 04:48, Duy Nguyen пишет: >> On Wed, Jun 11, 2014 at 5:59 AM, Patrick Lauer wrote: >>> Another part: Git wasn't ready. >>> The first migration attempt failed after consuming nearly 100GB of RAM! >>> When it did work it took obscene amounts of time, and the result was >>> unusably large (e.g. initial checkout would take 16GB RAM on the server, >>> thus not allowing a few hundred devs to do checkouts the same day). >>> The current state is almost usable, but it is still obscenely slow (e.g. >>> initial clone taking ~10 CPU-minutes just to figure out what to do), but >>> we can just throw more hardware at it. >>> (10 minutes @ 3.6Ghz, so on my notebook it'll take about 4h to just >>> clone the friggin repository. Too awesome!) >> >> Since v1.9.0 we can clone from a shallow repository. We can host two >> repos on the server: a full repo and a shallow one, containing history >> of only last year. Most of the time spent in initial clone is to >> verify the history. Shorter history would shorten that time. But you >> need to try out to see how long it actually is. I'm not sure if that >> 16GB includes cloning, or just plain checkout. If the latter, Git has >> a problem. >> > > Not sure if you can commit into that shallow repo(IIRC, you can not). Not before v1.9.0. Since 1.9, shallow repos are no different than full ones. You can fetch from a shallow repo, to a shallow repo, as well as push from/to a shallow repo. -- Duy
Re: [gentoo-dev] git security (SHA-1)
On Tue, Sep 16, 2014 at 5:11 AM, Gordon Pettey wrote: > On Mon, Sep 15, 2014 at 7:02 AM, hasufell wrote: >> >> hasufell: >> > >> > * there is no known SHA-1 collision afais >> > * calculating one isn't that hard. NSA might be able to do it in >> > reasonable time >> > * however, the algorithms to do that will come up with random garbage, >> > so it's a completely different thing to hide a useful vulnerability >> > behind a SHA-1 collision >> > >> >> That said... an attacker who has that much resources to calculate a >> _random_ hash collision in reasonable time would certainly have a lot of >> easier attack vectors than forging a _non-random_ hash collision that >> contains actual working code (which, afaiu doesn't effectively work with >> the current attack algorithms on SHA-1). >> >> He could simply break into one of the ~200 developer computers. There's >> a pretty high chance at least one of them is running windows or known >> vulnerable versions of the kernel or other random packages. >> >> No need to waste millions of dollars on SHA-1. > > > Even if you wanted to burn the money to find that magical collision that > actually contains working code, you've still got to somehow propagate that > to other repositories, since they'll just ignore it for having the same hash > as an already-existing object. In the fetch/pull case, if you receive the "same" object that you already have, git performs byte-to-byte comparison and warns loudly if the "new"object does not match yours. -- Duy
Re: [gentoo-dev] git security (SHA-1)
On Tue, Sep 16, 2014 at 5:41 AM, Duy Nguyen wrote: >> Even if you wanted to burn the money to find that magical collision that >> actually contains working code, you've still got to somehow propagate that >> to other repositories, since they'll just ignore it for having the same hash >> as an already-existing object. > > In the fetch/pull case, if you receive the "same" object that you > already have, git performs byte-to-byte comparison and warns loudly if > the "new"object does not match yours. Correction, in the push case as well. You have to find another way to inject this collided object (perhaps via a server that is not C Git). -- Duy
