[ Following to debian-security-tracker@ldo ] Sylvain Beucler wrote...
> On 07/06/2026 21:24, Christoph Biedl wrote:
> > This I can confirm, the auto-gc brought this down to 5.3G last night.
> [...]
> > Still, this is a lot worse than ealier: On April 7th the total size of
> > .git/objects/ was just 1571M here - a day later things started to go
> > downhill.
>
> Interesting. Do you graph the .git usage somehow, and do you have more data?
This is a by-product of my backup strategy: rsync into a filesystem with
snapshots. Such snapshots are done daily, and are later expired in a
non-linear way, to keep weekly ones, later monthly. Counting over those
that are still around:
846M 20250201
2,4G 20250301
690M 20250401
1,2G 20250501
876M 20250601
1,8G 20250701
1,1G 20250801
1,9G 20250901
1,1G 20251001
1,6G 20251013
1,8G 20251020
1,9G 20251027
2,0G 20251101
2,2G 20251103
631M 20251110
886M 20251117
1,4G 20251124
1,5G 20251201
1,8G 20251208
2,0G 20251215
2,2G 20251222
2,8G 20251229
3,1G 20260101
3,3G 20260105
703M 20260112
954M 20260119
1,1G 20260126
1,4G 20260201
1,4G 20260202
1,6G 20260209
1,8G 20260216
1,9G 20260223
2,0G 20260301
2,1G 20260302
770M 20260309
913M 20260316
1,1G 20260320
1,1G 20260321
1,1G 20260322
1,2G 20260323
1,2G 20260324
1,3G 20260325
1,3G 20260326
1,3G 20260327
1,3G 20260328
1,4G 20260329
1,4G 20260330
1,4G 20260331
1,4G 20260401
1,5G 20260402
1,5G 20260403
1,5G 20260404
1,5G 20260405
1,6G 20260406
1,6G 20260407
1,6G 20260408
1,9G 20260409 <== here things start to become interesting
2,6G 20260410
2,7G 20260411
3,7G 20260412
3,7G 20260413
4,4G 20260414
4,8G 20260415
4,8G 20260416
5,9G 20260417
6,0G 20260418
6,0G 20260419
6,6G 20260420
7,1G 20260421
8,0G 20260422
8,3G 20260423
8,2G 20260424
8,6G 20260425
9,1G 20260426
9,1G 20260427
9,6G 20260428
10G 20260429
11G 20260430
12G 20260501
(...)
Actually, the increasing run time of the daily backup was the first
thing I noticed.
> I believe that Salsa is still sending very suboptimal packs over the
> network, which are then reused by the local Git.
Possibly that's core.bigFileThreshold, and and among other things it's
also a (mild) DoS protection.
> 'git gc --aggressive --keep-largest-pack' repacked the most recents packs
> from last week MUCH more efficiently (1.6GB -> 12MB), which tends to confirm
> this hypothesis.
>
> 'git gc --aggressive' is supposed to repack everything, but I'm not able to
> run it on my computer anymore at this currently OOMs at >20GB RAM, which
> means previous packs may stay fat and non-optimized forever.
Interesting to learn more about git internals - for the practical part
however there's no doubt the current is already beyond the
limits of the tool. And as there is no realistic replacement for the
tool (git), this means adjusting the workflow. I'd prefer this to happen
before the file crosses the 90 Mbyte size.
> I also see that files >50MB now get a warning at GitHub (before getting
> rejected at >100MB), so with our data/CVE/list at 53MB we're starting to hit
> all kinds of hard limits, confirming we have to do something sooner than
> later.
Well, this is not Github :=) Still, the warning should be taken as an
advice, and it is.
> Maybe this is something where the LTS Team can help with, let's wait for the
> bookworm LTS handover to take place this week, and I'll try and reach out to
> involved teams.
Likely I'm missing something here - in my understanding the obstacles
are, in ascending order:
* Spilt the file and re-create the git history.
* Adjust the tooling.
* Seek consensus on the future workflow.
Christoph
signature.asc
Description: PGP signature
