On Jan 31 09:58, Eric Blake wrote: > > 2) Code that uses mbrtowc() or wcrtomb() is also likely to malfunction. > > On Cygwin >= 1.7 mbrtowc() and wcrtomb() is implemented in an > > intelligent > > but somewhat surprising way: wcrtomb() may return 0, that is, produce > > no > > output bytes when it consumes a wchar_t. > > > Now with a chinese character outside the BMP: > > $ > > 1 4 > > $ printf 'a \xf0\xa1\x88\xb4 b\n' | wc -w -m > > 3 6 > > > > On Cygwin 1.7.5 (with LANG=C.UTF-8 and 'wc' from GNU coreutils 8.5): > > > > $ printf 'a\xf0\xa1\x88\xb4b\n' | wc -w -m > > 1 5 > > $ printf 'a \xf0\xa1\x88\xb4 b\n' | wc -w -m > > 2 7 > > > > So both the number of characters and the number of words are counted > > wrong as soon as non-BMP characters occur. > > > > Does this represent a bug in cygwin's mbrtowc routines that could be > fixed by cygwin? > > Or, does this represent a bug in coreutils for using mbrtowc one > character at a time instead of something like mbsrtowcs to do bulk > conversions?
Just to clarify a bit. This has been discussed on the cygwin-developer mailing list back in 2009. The original code which handled UTF-16 surrogates always wrote at least 1 byte to the destination UTF-8 string. However, the problem is that Windows filenames may contain lone surrogate pairs, even though the filename is usually interpreted as UTF-16. So the current code returns 0 bytes for the first surrogate half and only writes the full UTF-8 sequence after the second surrogate half has been evaluated. In the case where a lone high surrogate is still pending, but the low surrogate is missing, we can just write out the high surrogate in CESU-8 encoding. This would not have been possible if we had already written the first byte of the UTF-8 string. Lone low surrogates are written as CESU-8 sequence immediately so they are nothing to worry about. As for wctomb/wcrtomb returning 0: Even if this looks like kind of a stretch, this should not be a problem per POSIX. A return value of 0 from wctomb/wcrtomb has no special meaning(*). Even in the case where the incoming wide char is L'\0', the resulting \0 is written and 1 is returned. Since 0 bytes have been written to the destination string, returning 0 is perfectly valid. If a calling function misinterprets the return value of 0 as an error or EOF, it's not a bug in wctomb/wcrtomb. For the original discussion, see http://cygwin.com/ml/cygwin-developers/2009-09/msg00065.html Corinna (*) http://pubs.opengroup.org/onlinepubs/9699919799/functions/wcrtomb.html -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
