On Wednesday, July 25, 2012 11:02:01 AM UTC+2, Walter Dörwald wrote:
> On 25.07.12 08:09, Ulrich Eckhardt wrote:
>
> > Am 24.07.2012 17:01, schrieb cpppw...@gmail.com:
> >> reader = codecs.getreader(encoding)
> >> lines = []
> >> with open(filename, 'rb') as f:
> >> lines = reader(f,
> 'strict').readlines(keepends=False)
> >>
> >> where encoding == 'utf-16-be'
> >> Everything works fine, except that lines[0] is equal to
> >> codecs.BOM_UTF16_BE
> >> Is this behaviour correct, that the BOM is still present?
> >
> > Yes, assuming the first line only contains that BOM. Technically
> it's a
> > space character, and why should those be removed?
>
> If the first "character" in the file is a BOM the file encoding is
> probably not utf-16-be but utf-16.
>
> Servus,
> Walter
The byte order mark, if present, is nothing else than
an encoded
>>> ud.name('\ufeff')
'ZERO WIDTH NO-BREAK SPACE'
*code point*.
Five "BOM" are possible (Unicode consortium). utf-8-sig, utf-16-be,
utf-16-le, utf-32-be, utf-32-le. The codecs module provide many
aliases.
The fact that utf-16/32 does correspond to -le or to -be may
vary according to the platforms, the compilers, ...
>>> sys.version
'3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit
(Intel)]'
>>> codecs.BOM_UTF16_BE
b'\xfe\xff'
>>> codecs.BOM_UTF16_LE
b'\xff\xfe'
>>> codecs.BOM_UTF16
b'\xff\xfe'
>>>
---
As far as I know, Py 2.7 or Py 3.2 never return a "BOM" when
a file is read correctly.
>>> with open('a-utf-16-be.txt', 'r', encoding='utf-16-be') as f:
... r = f.readlines()
... for zeile in r:
... print(zeile.rstrip())
...
abc
élève
cœur
€uro
>>>
jmf
--
http://mail.python.org/mailman/listinfo/python-list
