On Wed, Dec 11, 2013 at 5:01 PM, Ned Batchelder <n...@nedbatchelder.com>wrote:
> On 12/11/13 6:39 PM, Dan Stromberg wrote: > >> >> On Wed, Dec 11, 2013 at 3:24 PM, Steven D'Aprano >> <steve+comp.lang.pyt...@pearwood.info >> <mailto:steve+comp.lang.pyt...@pearwood.info>> wrote: >> >> On Wed, 11 Dec 2013 15:07:35 -0800, Dan Stromberg wrote: >> >> > $ chardet mpaa-ratings-reasons.list >> > mpaa-ratings-reasons.list: windows-1255 (confidence: 0.97) >> > >> > I'm aware that chardet is playing guessing games, though one >> would hope >> > it would guess well most of the time, and give a reasonable >> confidence >> > rating. >> >> What reason do you have for thinking that Windows-1255 isn't a >> reasonable >> guess? If the bulk of the text is Latin-1 except perhaps for one or >> two >> Hebrew characters (or what chardet thinks are Hebrew characters), it >> may >> actually be a reasonable guess. >> >> >> I get a traceback if I try to read the file as Windows-1255. I don't >> get a traceback if I read it as ISO-8859-1. >> >> If it is a poor guess, perhaps you ought to report it to the chardet >> maintainers as a good example of a poor guess. >> >> I was considering that, and may do so. >> >> I've also been wondering if ISO-8859-1 is just an octet-oriented codec, >> so it'll read about anything. There are clearly non-7-bit-ASCII >> characters in the file that look like line noise in an mrxvt. >> > > Both ISO-8859-1 and Windows-1255 are octet-oriented, I don't see why one > would raise an exception when the other didn't. Unless the exception isn't > on the decode, but instead on your attempt to output the result. Can you > show the full traceback you're seeing? > $ ./movie-ratings Traceback (most recent call last): File "./movie-ratings", line 85, in <module> main() File "./movie-ratings", line 68, in main ratings = get_ratings('/home/dstromberg/src/home-svn/movie-ratings/trunk/mpaa-ratings-reasons.list') File "./movie-ratings", line 52, in get_ratings for line in ratings_file: File "/usr/local/cpython-3.3/lib/python3.3/encodings/cp1255.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0xfc in position 1225: character maps to <undefined> BTW, other than satisfying our respective curiosities, I consider this project finished. It's probably not getting ratings for my entire movie collection, but it is getting them for a significant fraction, which is all I was really looking for. Now I know which ones are rated PG, so I can decide whether to let my 8 year old watch them. This is with cpython-3.3. Thanks. ^_^
-- https://mail.python.org/mailman/listinfo/python-list
