2009/9/22 Corinna Vinschen: >> > As you might know, invalid bytes >= 0x80 are translated to UTF-16 by >> > transposing them into the 0xdc00 - 0xdcff range by just or'ing 0xdc00. >> > The problem now is that readdir() will return the transposed characters >> > as if they are the original characters. >> >> Yep, that's where the bug is. Those 0xDC?? words represent invalid >> UTF-8 bytes. They do not represent CP1252 or ISO-8859-1 characters. >> >> Therefore, when converting a UTF-16 Windows filename to the current >> charset, 0xDC?? words should be treated like any other UTF-16 word >> that can't be represented in the current charset: it should be encoded >> as a ^N sequence. > > How? Just like the incoming multibyte character didn't represent a valid > UTF-8 char, a single U+DCxx value does not represent a valid UTF-16 char. > Therefore, the ^N conversion will fail since U+DCxx can't be converted > to valid UTF-8.
True, but that's an implementation issue rather than a design issue, i.e. the ^N conversion needs to do the UTF-8 conversion itself rather than invoke the __utf8 functions. Shall I look into creating a patch? >> > So it looks like the current mechanism to handle invalid multibyte >> > sequences is too complicated for us. As far as I can see, it would be >> > much simpler and less error prone to translate the invalid bytes simply >> > to the equivalent UTF-16 value. That creates filenames with UTF-16 >> > values from the ISO-8859-1 range. >> >> This won't work correctly, because different POSIX filenames will map >> to the same Windows filename. For example, the filenames "\xC3\xA4" >> (valid UTF-8 for a-umlaut) and "\xC4" (invalid UTF-8 sequence that >> represents a-umlaut in 8859-1), will both map to Windows filename >> "U+00C4", i.e a-umlaut in UTF-16. Furthermore, after creating a file >> called "\xC4", a readdir() would show that file as "\xC3\xA4". > > Right, but using your above suggestion will also lead to another filename > in readdir, it would just be \x0e\xsome\xthing. I don't think the suggestion above is directly relevant to the problem I tried to highlight here. Currently, with UTF-8 filename encodings, "\xC3xA4" turns into U+00C4 on disk, while "\xC4" turns into U+DCC4, and converting back yields the original separate filenames. If I understand your proposal correctly, both "\xC3\xA4" and "\xC4" would turn into U+00C4, hence converting back would yield "\xC3\xA4" for both. This is wrong. Those filenames shouldn't be clobbering each other, and a filename shouldn't change between open() and readdir(), certainly not without switching charset inbetween. Having said that, if you did switch charset from UTF-8 e.g. to ISO-8859-1, the on-disk U+DCC4 would indeed turn into "\x0E\xsome\xthing". However, that issue applies to any UTF-16 character not in the target charset, not just those funny U+DC?? codes for representing invalid UTF-8 bytes. The only way to avoid the POSIX filenames changing depending on locale would be to assume UTF-8 for filenames no matter the locale charset. That's an entirely different can of worms though, extending the compatibility problems discussed on the "The C locale" thread to all non-UTF-8 locales, and putting the onus for converting filenames on applications. Andy -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
