On 05/24/2017 02:52 AM, Ronald Fischer wrote: > I have a file X which contains ASCII text, but also in some lines German > umlaut characters. The file is classified as: > > $ file X > X: ISO-8859 text, with CRLF line terminators
In ISO-8859, a German umlaut occupies one byte with the high-bit set. > > However, if LANG is set to C.UTF-8, two things happen: > > - grep classifies the file as binary file and produces the error message > "Binary file X matches" In UTF-8, any one-byte sequence with the high bit set in isolation is an encoding error (all high-bit bytes in UTF-8 occur in 2-or-more byte sequences). According to POSIX, grep is only required to operate on text files, and the definition of a text file includes a requirement that ALL bytes in the file form valid encodings of characters in the current locale. Yes, this means that there are files that are valid text files in some locales and invalid in others (such as your file here). Once you violate the POSIX constraint of passing a non-text file to grep, all bets are off, and grep can do whatever it wants, including telling you that a binary file matches. > > - Both the grepped lines (i.e. in our example the non-empty lines) AND > the error message end up in the standard output (i.e. in file Y). Yes, that's the current intended behavior in upstream grep. It's not unique to Cygwin, so complaining here won't change it. > > IMO, there are several problems with this: > > 1. It's hard to see, why an umlaut character makes the file X binary > under encoding C.UTF-8, Because it's not a valid UTF-8 encoding. Use iconv to convert your file from ISO-8859 to UTF-8 if you want to grep it under C.UTF-8. > but not under encoding UTF-8 or C.en_EN Those aren't valid locale names. But if you mean that it does what you want under LC_ALL=C, that's because in the straight C locale, there are no multi-byte characters, and therefore no encoding errors are possible, and therefore you can't get a binary file in that locale due merely to an encoding error. > > 2. If grep classifies a file as binary, I think the desired behaviour > would be to NOT produce any output, unless the -a flag has been > supplied. Once behavior is in the realm of the undefined, it's hard to say what the desired behavior should be. But again, if you want the current behavior changed, it's an upstream issue to complain about on bug-grep, and not something that I'm going to change for Cygwin in isolation. > > 3. If grep writes a message "Binary file ... matches", this message > should go to stderr, not stdout. The stdout is supposed to contain only > a subset of the input lines. The message "Binary file ... matches" has always gone to stdout, even before upstream was tightened to flag more encoding errors as binary files. Whether the behavior of mixing it with actual output is desirable is a question for upstream. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
signature.asc
Description: OpenPGP digital signature
