Hi Mo, before going into criticizing things, I would like to thank you for your continued work in the AI space. In particular, your way of classifying models into different degrees of freedom demonstrates how much you care about Debian's values. I see that your focus is on enabling users and that's amazing!
On Sun, Jan 12, 2025 at 05:01:33PM -0500, M. Zhou wrote: > One long term goal of debian deep learning team is to host an LLM with > the team's AMD GPUs and expose it to the members. That said, the necessary > packages to run that kind of service are still missing from our archive. > It is a good way to use existing GPUs any way. Disregarding the ecological and political aspects that have been discussed elsewhere at length, let me also note that successfully using a LLM is not trivial. Whether you get something useful very much depends on what you ask. Typically, you should expect that something between 30% and 70% of answers are factually wrong in some regard. As a result, asking questions where the answer is not verifiable is not very useful. Using the answers without verifying them poses a risk to us as a community that myths are propagated and become harder to falsify. We are talking about ai dementia already and can observe it happening. How do you see us mitigating this problem? That said, it can be useful to use a LLM if your premise is that you verify the answer. For instance if you are searching for a library function that does something particular, searching the documentation can be time consuming. Describing your function to a LLM and then looking up the documentation of the presented suggestions has a significant chance of turning up something useful (in addition to the rubbish included). Likewise, searching for a particular statement in a mailing list thread can be a daunting task where a LLM may be able to interpret your vague memory sufficiently well that it can provide a few candidates. This way of using LLMs is effectively limiting it to behave as a search engine with a different query language. I suspect that most of us use search engines all day without thinking much about them being driven by corporate actors running big clusters. Is using LLMs in this way that much different from using a search engine? Ecological and economical reasons aside, would anyone see an issue with providing an AI-driven search engine to Debian's documentation, mailing lists, bugs and the wiki? > Even if we get no commercial sponsorship of API calls, we will eventually > experiment and evaluate one with the team's infrastructure. We are still > working towards that. I am actually looking forward to this as I trust you to do it in a responsible way given your earlier work. > First, DebGPT is designed to conveniently put any particular information > whether or not Debian-specific, to the context of LLM. I have also implemented > some mapreduce algorithm to let the LLM deal with extremely overlength > context such as a whole ratt buildlog directory. Indeed. I can imagine that a LLM may be able to also suggest particular lines in a build log that may hint at the cause of a failure whereas we now have a set of patterns (e.g. "Waiting for unfinished jobs") and hope that one of them locates the problem. > My intention is always to explore possible and potential ways to make LLM > useful in any possible extent. To support my idea, I wrote DebGPT, and > I tend to only claim things that is *already implemented* and *reproducible* > in DebGPT. I have not experimented with DebGPT yet. Maybe I should. Unless already done so, let me suggest that you engineer the default prompt in such a way that your LLM references its information sources whenever possible. Also including some text that asks the user to verify the answer using external sources may help (in an opt-out way). Practically speaking, the use of neural networks is not something we can stop even if we wanted to. In ten years, all of us will be using neural networks on a daily basis. Even today, they're difficult to avoid (e.g. when dealing with customer service of any corporation). The best we can do here is making the use of them as free as possible and that's what I see Mo is doing. Helmut
