Martin, Thanks, food for thought indeed. > On Unix, yes. On Windows, NTFS and VFAT represent file names as Unicode > strings always, independent of locale. POSIX file names are byte > strings, and there isn't any good support for recording what their > encoding is. I get my filenames from two sources: 1. A wxPython treeview control (unicode build) 2. os.listdir() with a unicode path passed to it
I have found that os.listdir() does not always return unicode objects when passed a unicode path. Sometimes "byte strings" are returned in the list, mixed-in with unicodes. I will try the technique given on:http://www.pyzine.com/Issue008/Section_Articles/article_Encodings.html#guessing-the-encoding Perhaps that will help. Re os.listdir(): > If you think you may have file names with mixed locales, and > the current locale might not match the file name's locale, you should > be using the byte string variant on Unix (which it seems you are already > doing). I gather you mean that I should get a unicode path, encode it to a byte string and then pass that to os.listdir Then, I suppose, I will have to decode each resulting byte string (via the detect routines mentioned in the link above) back into unicode - passing those I simply cannot interpret. > Then, if the locale's encoding cannot decode the file names, you have > several options > a) don't try to interpret the file names as character strings, i.e. > don't decode them. Not sure why you need the file names - if it's > only to open the files, and never to present the file name to the > user, not decoding them might be feasible So, you reckon I should stick to byte-strings for the low-level file open stuff? It's a little complicated by my using Python Imaging to access the font files. It hands it all over to Freetype and really leaves my sphere of savvy. I'll do some testing with PIL and byte-string filenames. I wish my memory was better, I'm pretty sure I've been down that road and all my results kept pushing me to stick to unicode objects as far as possible. > b) guess an encoding. For file names on Linux, UTF-8 is fairly common, > so it might be a reasonable guess. > c) accept lossy decoding, i.e. decode with some encoding, and use > "replace" as the error handler. You'll have to preserve the original > file names along with the decoded versions if you later also want to > operate on the original file. Okay, I'm getting your drift. > That's not true. Try open("\xff","w"), then try interpreting the file > name as UTF-8. Some byte strings are not meaningful UTF-8, hence that > approach cannot work. Okay. > That's correct, and there is no solution (not in Python, not in any > other programming language). You have to made trade-offs. For that, > you need to analyze precisely what your requirements are. I would say the requirements are: 1. To open font files from any source (locale.) 2. To display their filename on the gui and the console. 3. To fetch some text meta-info (family etc.) via PIL/Freetype and display same. 4. To write the path and filename to text files. 5. To make soft links (path + filename) to another path. So, there's a lot of unicode + unicode and os.path.join and so forth going on. > > I went through this exercise recently and had no joy. It seems the string > > I chose to use simply would not render - even under 'ignore' and > > 'replace'. > I don't understand what "would not render" means. I meant it would not print the name, but constantly throws ascii related errors. I don't know if the character will survive this email, but the text I was trying to display (under LANG=C) in a python script (not the immediate-mode interpreter) was: "MÖgul". The second character is a capital O with an umlaut (double-dots I think) above it. For some reason I could not get that to display as "M?gul" or "Mgul". BTW, I just made that up - it means nothing (to me). I hope it's not a swear word in some other language :) > As for font files - I don't know what encoding the family is in, but > I would sure hope that the format specification of the font file format > would also specify what the encoding for the family name is, or that > there are at least established conventions. You'd think. It turns out that font file are anything but simple. I am doing my best to avoid being sucked-into the black hole of complexity they represent. I must stick to what PIL/Freetype can do. The internals of font-files are waaaaaay over my head. > >> I would avoid locale.getlocale. It's a pointless function (IMO). > As a consequence, it will return None if it doesn't know better. > If all you want is the charset of the locale, use > locale.getpreferredencoding(). Brilliant summary - thanks a lot for that. > You could just leave out the languages parameter, and trust gettext > to find some message catalog. Right - I'll give that a go. > > This would mean cutting-out a percentage of the external font files that > > can be used by the app. > See above. There are other ways to trade-off. Alternatively, you could > require that the program finds a richer locale, and bail out if the > locale is just "C". That's kind of what the OP is all about. If I make this a 'design decision' then it means I have a problem with the Debian packaging (and RPM?) rules that require a "C" locale support. I think I shall have to break the links between my setup.py and the rest of my app - so that setup.py will allow LANG=C but the app (when run) will not. > That doesn't help. For Turkish in particular, the UTF-8 locale is worse > than the ISO-8859-9 locale, as the lowercase I takes two bytes in UTF-8, > so tolower can't really work in the UTF-8 locale (but can in the > ISO-8859-9 locale). Wow. I still get cold chills -- but I assume that once the right encoding is known this sort of thing will be okay. Thanks again. It's coming together slowly. \d -- http://mail.python.org/mailman/listinfo/python-list
