Hi Andreas, I completely agree with the point you make about correctness. I personally feel LLMs have a use for exploration and bug finding. Even if I agree completely with the motivations of the GCD.
>Just yesterday I asked several LLMs a simple question about a project >I know: >"What is the default precision of MPFR?". Several of them replied that >there is none, claiming that it must be given by the user on variable >initialisation (both wrong); one went on to state that there is a default >rounding mode (which is wrong, it needs to be given by the user at each >function invocation). And it gave code snippets that initialised variables, >but did not free them. I think this example mischaracterizes a bit these tools, I suggest you to use them in agent mode (they work fine inside a guix shell container, I usually share only /tmp or a subset of it). I am not familiar with MPFR, I assume it's a git repository. I would probably have asked: "clone the source of MPFR, study it and tell me the default precision of MPFR. Provide evidence for any statement you make about the code in the file_name:line_number format." This has worked well for me and gave a correct answer with evidence. Of course one has always to assume that they are hallucinating, but asking for evidence makes checking their statements easier. cheers, giacomo
