The LLM I used to produce that exact news report was gpt-4o-mini,
from openai. ChatGPT is the name of openai's LLM web interface and
its underlying LLM model name could change. It took roughly 3
minutes to perform the bulk API calls.
That said, I basically implemented support for all commonly seen
LLM inference services:
(4 commercial ones)
openai, anthropic, google, xai,
(4 self-hosted)
llamafile, ollama, vllm, zmq (built-in but kind of outdated.)
Other services missing from the list are also supported as long
as it has compatibility mode to the openai api.
For the particular use case like summarizing a mailing list, self-hosted
one will be much slower to respond to the bulk API call unless it is
hosted on a GPU cluster :-)
Small LLMs are not necessarily smart enough. The open llm leaderboard[3]
is a good reference for figuring out the best open-access llm for
self-hosting.
In terms of "Debian hosted computer with AMD GPU for LLM inference" --
that is exactly one of the long term goals of debian deep learning
team (debian-ai@l.d.o). Team members are working to prepare the ROCm
packages and the ROCm version of pytorch.
I find ollama[1] and llamafile[2] quite handy to use locally if do not
mind using software from outside of debian archive, with a spare GPU.
[1] https://github.com/ollama/ollama
[2] https://github.com/Mozilla-Ocho/llamafile
[3] https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard
On 11/9/24 05:19, PICCA Frederic-Emmanuel wrote:
is it via ChatGPT or an llm self hosted ?
Can we imagine having a Debian hosted computer with and AMD GPU dedicated to
this use case ?
Se should provide these summaries letter for most of our mailing list :)
cheers
Fred
----- Le 9 Nov 24, à 14:09, Hector Oron zu...@debian.org a écrit :
Hello Lumin,
El sáb, 9 nov 2024 a las 10:27, DebGPT (<lu...@debian.org>) escribió:
This is an experiment, by letting LLM go through all 369 emails from
debian-devel on Oct. The command for producing the news report
is included below. Use debgpt's git HEAD if you want to try.
First time I see this kind of email, I thought time ago that'd be a
really cool use of AI, to produce a summary of mailing lists - since I
struggle to read everything.
I just want to thank you for putting this together and, at least from
my side, this is very much appreciated.
Regards
--
Héctor Orón -.. . -... .. .- -. -.. . ...- . .-.. --- .--. . .-.
