On 10/6/26 12:23, Divya Ranjan Pattanaik wrote:
Hello everyone,
I didn't want to derail the conversation too much in the GCD 008 PR, so let us
discuss the details here. Thank you Thanos and Hugo for sharing with us the
ways you guys have been using LLMs.
Thank you for trying to create something positive out of the discussion!
I'll see this as "we need to ensure that people can keep contributing
without an LLM".
Lets break it up into parts. The two main parts I see are:
1. The 'do everything autonomous' part. You can tell an agent "please
keep python-astropy at the latest version, refresh any outdated
dependency, and fix any dependent that breaks" (so recursion both ways).
I don't have too much experience with taking things that far, but
there is quite some power there. There is also quite some automation
possible without LLMs, especially assuming 'the happy path' where
everything works well together.
2. Fixing the non-happy path, what if a package is broken? This is the
part that you are focusing on. But it ties into the first part, because
if possible, the best fix is usually to refresh the package. Refreshing
the package requires the recursion part to be automated as well.
This can be split up in two parts as well:
2a. Figuring out what is wrong. Or guessing. A tool that could point
out "hey, this is weird" (either from the package itself or from the
build log) would already be very useful, even if it doesn't fix anything.
2b. Fixing the broken thing. This is way harder.
You asked for things we use an LLM for. But that is not necessarily the
right approach. Any commit we did manually would be a target for
automation, and most of those commits would probably be more suitable.
That is, the more we automate, the more human time we have for the
'complicated' cases.
That is, now we do 50% (a guess) with a simple "guix refresh", 49%
manually, and 1% with an LLM. If we can change that to "80%" with "guix
refresh-plus-plus" and 20% manually, then that is also a win. (Browsing
through the commits, it seems that only about 10% can currently be done
with 'guix refresh'.)
So I think the best way to approach this LLM-free automation is take any
commit that could possibly be automated, and automate that. I think it
is going to be really hard to make a difference though. But maybe if we
have a framework, then people can add their own 'favorite fixes'. That
is, for any manual fix we do, we also try to automate it for next time.
Nevertheless, here is a full list of everything Guix-related that I
actually used an LLM for. It is kinda dumb actually. Note that the
code was written by me by hand, the agent only investigated.
- [1] Calibre had a hardcoded python3.11 path in it. Embarrassing; I
just didn't read properly. But relevant, because I then grepped the
entire code base, and there were more packages like that. So to some
extend this check can be automated.
https://codeberg.org/guix/guix/commit/495a81ada81a472db9061609e5a6d18d5362a8d1
- [2] I misunderstood what happened with openshot. The build failure
was due to the pyproject migration, and not due to the Python 3.12
migration. Therefore I searched for the wrong solution. The agent
immediately noticed what was going on (and even consulted the git
history to show it was right). The problem was an outdated phase
modification should be removed. Seems hard to automate without an LLM,
but it can be generalized. E.g. phases that do substitutes might be
outdated. So disabling phases could be a strategy.
https://codeberg.org/guix/guix/commit/9abfa7c62fa0808e580d85ceeb99da769eebf26e
I did an experiment a few months back where I asked the agent four times
to "pick a random failing package and fix it", it fixed all four. Here
the code was written by the LLM. Those I did not make a P.R. for for
various reasons. I can't say whether the fixes are good; the goal was
to see whether it worked at all.
- [3] Adding headers to qdl with a subsitute (still broken on master).
The detection of the problem could probably be automated, but maybe it
would be better to add a make-flag; not sure. (I recall I changed the
URLs by hand in this commit.)
https://codeberg.org/hugobuddel/guix-mirror/commit/8e7277063f48a951deb2e62c638da703f1c62b1d
- [4] Fixing a pointer type in perl-text-iconv (still broken on master).
I've done such a fix by hand once, so maybe it is something that could
be automated.
https://codeberg.org/hugobuddel/guix-mirror/commit/d61b4914be636c283bec2f65f51ca42dcbf7106f
- [5] Adding a missing module to
go-github-com-matttproud-golang-protobuf-extensions-v2. Exactly the
same fix on master. I don't know the go ecosystem, but it is weird that
this is suddenly a problem. There probably has been a migration (e.g.
to a new go version), so this could be a known problem, and thus be
automated.
https://codeberg.org/hugobuddel/guix-mirror/commit/04901453566e4af57305e87d9360955e5cdbc6bb
https://codeberg.org/guix/guix/commit/9cfcaf82649b4adc5b95919a5480e158a7ae818c
git log | grep 'native-inputs.*go-github-com-google-go-cmp'
- [6] Adding a missing header to audmes. This was fixed on master by
refreshing the package.
https://codeberg.org/hugobuddel/guix-mirror/commit/434384988392f117608592b99e4ebbf7dda1b7d8
https://codeberg.org/guix/guix/commit/ec3cc2458a6f0ae3656bf1b4221105875dd6819f
Hugo
Here are the concrete tasks I think that are not feasible by existing tools for
which LLMs have been sought for:
```org
* Build Failures
- Compiler Failures
- Missing packages
- Problematic flags (test, arch, etc.)
- Package requires path to binaries as variables (cf. emacs-elfeed)
* Build Phase Issues
- Changes to existing phase(s)
- Requires new simple phase(s)
- Complicated phases such as subtituting ENV variables, config etc.
* Patch Coordination with upstream
```
This list is of course not exhaustive yet, please remind me what other kinds of
issues we fall into, and an example commit or package definition would be great.
One immediate free improvement we can get is to "autonomously" have a bot
decide when the build failure is because of a dependency change and requires upgrade and
then it should automatically call `guix refresh` for that and prepare the patch properly.
This should not be _that_ difficult.
Regarding compiler failures, the ones that are straightforward such as the one
that Thanos' bot did for GCC14, can be also automated by parsing properly the
output of the compiler. In that case, it missed -Wno-error=array-bounds flag, a
simple bot can parse through the compiler output, check for which flags are
missing and suggest to add them to (arguments) in the package definition.
Let us try to find out such issues where we know statically what is wrong and
that is provided to us by the compiler, environment variables, etc. and see
what we can do with them.
Regards,
