Christian Moe <[email protected]> writes: > Jim Porter <[email protected]> writes: >> One issue here is review burden [...] With LLM-generated code, the >> patch is often cleaner (at least superficially), which in my >> experience requires much closer attention from the reviewer. > > I lack experience in reviewing code, but this resonates with my > experience in my field (translation and copy-editing).
I think these two aspects are actually quite different. For LLM-generated texts, the major issues (AFAIK) are tendency to overclaim things and tendency to write generic hand-waving instead of stating precise facts and statements. The former can be corrected with correct prompting. The latter is rather hard to fight because LLMs tend to generalize (naturally). I personally prefer LLMs to critique my own writing instead of letting them write things. Maybe translations are somewhat more similar to the code. For translations, LLMs sometimes lack broader (beyond the text given) contextual understanding and use incorrect words for translation. However, under the hood, this is often caused by underlying ambiguity of the text that is taken out of broader (company or field) context. The LLM is forced to give /some/ output, and it gives it, without clarifying ambiguities, instead randomly choosing one possible way to translate. There are ways to fight it - most important is giving the extended context, but an alternative is also prompting to surface the ambiguities. For the code, the problem with ambiguity is one of the major causes of bad quality code. When forced to decide about implementation details, LLMs often choose randomly (and incorrectly). That happens unless sufficient context is provided - very detailed instructions about what exactly needs to be written. I think I once had to write several pages of instructions to produce a useful non-trivial code. -- Ihor Radchenko // yantar92, Org mode maintainer, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92>
