I believe that Cygwin displays certain UTF-8 characters incorrectly. To
see the problem, first save the attached "utf-8_test.sed" text file to
your desktop. Then run "mintty," and set its options by right clicking
in its title bar, selecting "Options" and then "Text." On the Text page
set "Locale" to "en_US" and "Character set" to "UTF-8," and then
"Save." Now exit and restart mintty. Change directory to your desktop
and run the editor "vim" on the utf-8_test.sed file. Once inside vim do
a ":set fileencoding=utf-8". You should now see that vim displays
correctly a sample of one-, two-, and three-byte UTF-8 character
encodings in the test file. Vim fails, however, on the three-byte
encodings for the "en" dash, the "em" dash, and the ellipsis, each of
which displays incorrectly as a filled-in rectangle. Now exit vim and
do a "less" or "cat" on the utf-8_test.sed file. You should see most of
the sample UTF-8 encoded characters displayed correctly, except once
again for the en dash, em dash, and ellipsis. So it looks like a
problem in the underlying Cygwin run-time libraries rather than in vim,
less, or cat. I haven't tested this on four-byte UTF-8 character
encodings, but assume Cygwin will have similar problems.
# This is file "utf-8_test.sed"
#
# It's used by the "sed" utility program
# to convert XML-encoded filenames to UTF-8
# Match longest strings first
# Three-byte encodings:
# En dash
s/%[Ee]2%80%93/–/g
# Em dash
s/%[Ee]2%80%94/—/g
# Horizontal ellipsis
s/%[Ee]2%80%[Aa]6/…/g
# Less-than-or-equal sign
s/%[Ee]2%89%[Aa]4/≤/g
# Euro symbol
s/%[Ee]2%82%[Aa][Cc]/€/g
# Two-byte encodings:
# Non-break space
s/%[Cc]2%[Aa]0/⎵/g
# Lowercase a with acute accent
s/%[Cc]3%[Aa]1/á/g
# Lowercase a with umlaut (a.k.a. diaeresis)
s/%[Cc]3%[Aa]4/ä/g
# Lowercase e with acute accent
s/%[Cc]3%[Aa]9/é/g
# Lowercase i with acute accent
s/%[Cc]3%[Aa]D/í/g
# Lowercase o with acute accent
s/%[Cc]3%[Bb]3/ó/g
# Lowercase n with tilde
s/%[Cc]3%[Bb]1/ñ/g
# Lowercase c with acute accent
s/%[Cc]4%87/ć/g
# Lowercase o with long accent (a.k.a. macron)
s/%[Cc]5%8[Dd]/ō/g
# One-byte encodings:
# "And" sign (a.k.a. ampersand)
s/&/\&/g
# Space
s/%20/ /g
# Sharp (or pound) sign
s/%23/#/g
# Percent sign
s/%25/%/g
# Left square bracket
s/%5[Bb]/[/g
# Right square bracket
s/%5[Dd]/]/g
# End of file "utf-8_test.sed"
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
