On Sun, Dec 15, 2024 at 3:32 PM Noah Misch <n...@leadboat.com> wrote: > For PostgreSQL, I expect the most obvious problems will arise for rolname and > datname containing non-UTF8. For example, pg_dumpall relies on > appendShellString() to call pg_dump for arbitrary datname. pg_dumpall would > get "database ... does not exist".
Right, those catalogues have undefined encoding (the initial problem my CLUSTER ENCODING proposal started trying to fix) and could even be different for every row, and Windows wants all strings used in non-wide environ, argv, file APIs, etc to be valid in the ACP (because it converts them to UTF-16). We would get away with it if UTF-8 weren't so picky, but come to think of it, so is SJIS, so maybe this is not a new problem with $SUBJECT? Wild guess: 文字化け (= mojibake) when encoded as UTF-8 and then passed in a command line to CreateProcess() with ACP=SJIS might show the problem (I just gave that string to iconv -f SJIS -t UTF-8 and it rejected it, I'm assuming that means it'd do the same sort of thing in that context). It's a shame the implicit conversion here doesn't fail with EILSEQ. I can't imagine how anything good can ever have come from lossy, non-error-raising implicit conversions anywhere near argv[]. On the other hand, on Unix we have other problems stemming from the undefinedness. What does "copy ... to '/tmp/café.txt" do inside a LATIN1 database? macOS: EILSEQ, can't open that file, Linux: sure, now you have a file whose name is displayed as caf�.txt in your UTF-8 terminal or other software (U+FFFD REPLACEMENT CHARACTER). > 2. Just fail if the system option is enabled and we would appendShellString() > a non-UTF8 value. I guess the general version is just: fail if the string is not valid in the ACP (MB_ERR_INVALID_CHARS). With the ACP-matching idea for CLUSTER ENCODING, it *think* it should become unreachable in the two recommended modes: either those strings would be pure ASCII, or they'd be in database encoding (same encoding for all databases enforced) and the ACP would match, so it would all be aligned without any new conversions being required. It also has an UNDEFINED mode so a failed encoding validation there would still be reachable that way. Still thinking about it all though.
