Hi Peter,
At 2023-08-27T00:06:39-0400, Peter Schaffter wrote:
> On Sat, Aug 26, 2023, Bjarni Ingi Gislason wrote:
> > Lesson to learn: always use the information you have and give it to
> > the program, for example "groff" with option "-K.
> >
> > groff -Kutf8 -V file
> >
> > preconv -eutf8 file | troff -Tps | grops
>
> Thanks, Bjarni. Odd to be discovering this for the first time after
> decades of preparing documents. Live and learn.
In groff 1.23.0, the preconv(1) man pages attempts to address this
subject more directly.
iconv support
While preconv recognizes all of the coding tags listed above, it
is capable on its own of interpreting only three encodings:
Latin‐1, code page 1047, and UTF‐8. If iconv support is
configured at compile time and available at run time, all others
are passed to iconv library functions, which may recognize many
additional encoding strings. The command “preconv -v” discloses
whether iconv support is configured.
The use of iconv means that characters in the input that encode
invalid code points for that encoding may be dropped from the
output stream or mapped to the Unicode replacement character
(U+FFFD). Compare the following examples using the input “café”
(note the “e” with an acute accent), which due to its short
length challenges inference of the encoding used.
printf 'caf\351\n' | LC_ALL=en_US.UTF-8 preconv
printf 'caf\351\n' | preconv -e us-ascii
printf 'caf\351\n' | preconv -e latin-1
The fate of the accented “e” differs in each case. In the first,
uchardet fails to detect an encoding (though the library on your
system may behave differently) and preconv falls back to the
locale settings, where octal 351 starts an incomplete UTF‐8
sequence and results in the Unicode replacement character. In
the second, it is not a representable character in the declared
input encoding of US‐ASCII and is discarded by iconv. In the
last, it is correctly detected and mapped.
Regards,
Branden
signature.asc
Description: PGP signature
