[issue13618] bytes.deocde() UnicodeEncodeError on Apple iOS characters
New submission from Silverback Networks : I've searched high and low to find a way to make Python accept Apple's iOS characters, but it looks like Python is not supporting greater than 16-bit characters correctly. If you look at the leading character of each group, it's \xf0, indicating a 4-character sequence, which also indicates greater than 16-bit characters. I've tried all three "errors" arguments to decode - ignore, replace, and strict - and still get this error each time: UnicodeEncodeError: 'charmap' codec can't encode characters in position 140: character maps to So I have no way to proceed short of rolling my own corrected unicode decoder. My assumption is that Python should convert a character regardless of whether it's found in the internal lookup database, or at a minimum there should be a way to signal Python to do so. Below is a sample bytes string that will reproduce the problem: b'\n \n \n average-user-rating\n \n \n1\n \n \n text\n \n \n \xf0\x9f\x8e\x84\xf0\x9f\x8e\x85\xf0\x9f\x8e\x81\xf0\x9f\x8e\x84\xf0\x9f\x8e\x85\xf0\x9f\x8e\x81 if you haven't checked this out yet please do. download APP TRAILERS and go to videos use promo code FREE4U and enjoy free apps courtesy of apple MERRY CHRISTMAS \xf0\x9f\x8e\x84\xf0\x9f\x8e\x85\xf0\x9f\x8e\x81\xf0\x9f\x8e\x84\xf0\x9f\x8e\x85\xf0\x9f\x8e\x81\n \n \ntitle\n \n \n4. IF YOU LOVE FREE STUFF (v1.5)\n \n \ntype\n \n \n review\n \n \nuser-name\n \n \n Freenesss on Dec 16, 2011\n \n \n \n \n average-user-rating\n \n \n0.8\n \n \n text\n \n \nThis application is very cool .. I hope only be added to the dictionary other languages \xe2\x80\x8b\xe2\x80\x8b..\n \n \ntitle\n \n \n8. the dictionary (v1.5)\n \n \ntype\n \n \nreview\n \n \nuser-name\n \n \nRnaa on Dec 16, 2011\n \n \n \n \naverage-user-rating\n \n \n1\n \n \ntext\n \n \nHey I'm 13 trying to b discovered plz check my 1st video out on you tube its called speak now cover by Bekka burton thnx and I luv luv luv this app\n \n \ntitle\n \n \n9. Love this app+check me out on you tube (v1.5)\n \n \ntype\n \n \n review\n \n \nuser-name\n \n \n Lol\xee\x84\x86 on Dec 16, 2011\n \n \n' (Obviously, stripped down to not-well-formed XML, but for conversion purposes that's irrelevant.) -- components: Unicode messages: 149659 nosy: ezio.melotti, silverbacknet priority: normal severity: normal status: open title: bytes.deocde() UnicodeEncodeError on Apple iOS characters type: behavior versions: Python 3.3 ___ Python tracker <http://bugs.python.org/issue13618> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13618] bytes.decode() UnicodeEncodeError on Apple iOS (>16-bit) characters
Changes by Silverback Networks : -- title: bytes.deocde() UnicodeEncodeError on Apple iOS characters -> bytes.decode() UnicodeEncodeError on Apple iOS (>16-bit) characters ___ Python tracker <http://bugs.python.org/issue13618> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13618] bytes.decode() UnicodeEncodeError on Apple iOS (>16-bit) characters
Silverback Networks added the comment: I feel like a 'tard now, it was because I was trying to print() it at the same time I decoded it, which is what threw up. Well, sorry about that, next time I'll be a little more careful to separate every step before I go reporting it. -- resolution: -> invalid status: open -> closed ___ Python tracker <http://bugs.python.org/issue13618> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15772] Unresolved symbols in Windows 64-bit python
Silverback Networks added the comment: Same errors in 3.3. Some names are decorated, some aren't, seemingly at random. For instance, python32.lib contains: 35B20 _PyObject_Dump 35B20 __imp__PyObject_Dump and just below that, 2924A PyObject_Free 2924A __imp_PyObject_Free But the Visual Studio 2010 compiler always looks for the underscored names for x64, resulting in, for instance: error LNK2019: unresolved external symbol __imp__PyObject_Free referenced in function _WRdealloc I understand that the x64 isn't supposed to use underscore prefixes as all, but it is and Python partially is. I have no idea why. The preprocessed output of including python.h, btw, is: __declspec(dllimport) void PyObject_Free(void *); -- nosy: +silverbacknet ___ Python tracker <http://bugs.python.org/issue15772> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15772] Unresolved symbols in Windows 64-bit python
Changes by Silverback Networks : -- versions: +Python 3.3 ___ Python tracker <http://bugs.python.org/issue15772> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15772] Unresolved symbols in Windows 64-bit python
Silverback Networks added the comment: Boy, do I feel like an idiot now. An update to VC 2010 Express SP1 had clobbered by x64 build capabilities completely, and even reinstalling the SDK did nothing. It turns out that there's a specific hotfix you have to install to get x64 builds working again: http://support.microsoft.com/kb/2519277 When I did that and fixed up the build environment variables, Python extensions all built again, despite the apparently random decorating. -- ___ Python tracker <http://bugs.python.org/issue15772> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16296] Patch to fix building on Win32/64 under VS 2010
New submission from Silverback Networks: Once I got my broken environment fixed, this was the only thing that didn't work. The bug is that VS 2010 no longer creates a manifest by default, despite the documentation, and there are confirmation posts around the internet. /Manifest has to be forced starting with VS 2010. Here is a patch to fix that: --- Lib/distutils/msvc9compiler.py2011-08-14 11:17:42.0 -0700 +++ Lib/distutils/msvc9compiler.py2012-10-21 10:38:42.257682200 -0700 @@ -411,10 +411,16 @@ '/Z7', '/D_DEBUG'] self.ldflags_shared = ['/DLL', '/nologo', '/INCREMENTAL:NO'] +if self.__version >= 10: +self.ldflags_shared = ['/DLL', '/nologo', '/INCREMENTAL:NO', '/Manifest'] if self.__version >= 7: self.ldflags_shared_debug = [ '/DLL', '/nologo', '/INCREMENTAL:no', '/DEBUG', '/pdb:None' ] +if self.__version >= 10: +self.ldflags_shared = [ +'/DLL', '/nologo', '/INCREMENTAL:NO', '/DEBUG', '/pdb:None', '/Manifest' +] self.ldflags_static = [ '/nologo'] self.initialized = True -- components: Build messages: 173464 nosy: silverbacknet priority: normal severity: normal status: open title: Patch to fix building on Win32/64 under VS 2010 type: compile error versions: Python 3.1, Python 3.2, Python 3.3, Python 3.4 ___ Python tracker <http://bugs.python.org/issue16296> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16296] Patch to fix building on Win32/64 under VS 2010
Silverback Networks added the comment: oops, add _debug on the second part of the patch. -- ___ Python tracker <http://bugs.python.org/issue16296> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16296] Patch to fix building on Win32/64 under VS 2010
Silverback Networks added the comment: Sure. I got this patch from Mercurial, just in case, but it looks the same. -- keywords: +patch Added file: http://bugs.python.org/file27648/msvc9manifest.diff ___ Python tracker <http://bugs.python.org/issue16296> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17011] ElementPath ignores different namespace mappings for the same path expression
Changes by Silverback Networks : -- nosy: +silverbacknet ___ Python tracker <http://bugs.python.org/issue17011> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17088] ElementTree incorrectly refuses to write attributes without namespaces when default_namespace is used
New submission from Silverback Networks:
ET reads a default-namespaced (xmnls="whatever") file correctly but won't write
it back out.
The error given is:
ValueError: cannot use non-qualified names with default_namespace option
The XML reference is reasonably clear on this:
http://www.w3.org/TR/REC-xml-names/#defaulting
"Default namespace declarations do not apply directly to attribute names;"
"The namespace name for an unprefixed attribute name always has no value."
Therefore, it is not an error to write non-qualified _attribute_ names with a
default namespace; they're just considered un-namespaced anyway. The trivial
case where a file is read in with a default namespace and written out with the
same one should make it obvious:
from xml.etree.ElementTree import *
register_namespace('svg', 'http://www.w3.org/2000/svg')
svg = ElementTree(XML("""
http://www.w3.org/2000/svg"; version="1.1">
"""))
svg.write('simple_new.svg',encoding='UTF-8',default_namespace='svg')
Yet this will fail with the error above. By leaving off default_namespace,
every element is pointlessly prefixed by 'svg:' in the resulting file, but it
does work.
--
components: XML
messages: 181005
nosy: silverbacknet
priority: normal
severity: normal
status: open
title: ElementTree incorrectly refuses to write attributes without namespaces
when default_namespace is used
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5
___
Python tracker
<http://bugs.python.org/issue17088>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22009] pdb.set_trace() crashes when binary data is input via stdin
New submission from Silverback Networks: sample.py: import sys, pdb infile = sys.stdin.buffer pdb.set_trace() command line: python -m pdb sample.py < binary.exe output: > c:\users\me\dropbox\sample.py(1)() -> import sys, pdb (Pdb) Traceback (most recent call last): File "C:\dev\Python\x86\lib\pdb.py", line 1661, in main pdb._runscript(mainpyfile) File "C:\dev\Python\x86\lib\pdb.py", line 1542, in _runscript self.run(statement) File "C:\dev\Python\x86\lib\bdb.py", line 431, in run exec(cmd, globals, locals) File "", line 1, in File "c:\users\me\dropbox\test.py", line 1, in import sys, pdb File "c:\users\me\dropbox\test.py", line 1, in import sys, pdb File "C:\dev\Python\x86\lib\bdb.py", line 48, in trace_dispatch return self.dispatch_line(frame) File "C:\dev\Python\x86\lib\bdb.py", line 66, in dispatch_line self.user_line(frame) File "C:\dev\Python\x86\lib\pdb.py", line 259, in user_line self.interaction(frame, None) File "C:\dev\Python\x86\lib\pdb.py", line 346, in interaction self._cmdloop() File "C:\dev\Python\x86\lib\pdb.py", line 319, in _cmdloop self.cmdloop() File "C:\dev\Python\x86\lib\cmd.py", line 126, in cmdloop line = input(self.prompt) File "C:\dev\Python\x86\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 72: character maps to Uncaught exception. Entering post mortem debugging Running 'cont' or 'step' will restart the program > c:\dev\python\x86\lib\encodings\cp1252.py(23)decode() -> return codecs.charmap_decode(input,self.errors,decoding_table)[0] (Pdb) Traceback (most recent call last): File "C:\dev\Python\x86\lib\pdb.py", line 1661, in main pdb._runscript(mainpyfile) File "C:\dev\Python\x86\lib\pdb.py", line 1542, in _runscript self.run(statement) File "C:\dev\Python\x86\lib\bdb.py", line 431, in run exec(cmd, globals, locals) File "", line 1, in File "c:\users\me\dropbox\test.py", line 1, in import sys, pdb File "c:\users\me\dropbox\test.py", line 1, in import sys, pdb File "C:\dev\Python\x86\lib\bdb.py", line 48, in trace_dispatch return self.dispatch_line(frame) File "C:\dev\Python\x86\lib\bdb.py", line 66, in dispatch_line self.user_line(frame) File "C:\dev\Python\x86\lib\pdb.py", line 259, in user_line self.interaction(frame, None) File "C:\dev\Python\x86\lib\pdb.py", line 346, in interaction self._cmdloop() File "C:\dev\Python\x86\lib\pdb.py", line 319, in _cmdloop self.cmdloop() File "C:\dev\Python\x86\lib\cmd.py", line 126, in cmdloop line = input(self.prompt) File "C:\dev\Python\x86\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 72: character maps to During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\dev\Python\x86\lib\runpy.py", line 171, in _run_module_as_main "__main__", mod_spec) File "C:\dev\Python\x86\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\dev\Python\x86\lib\pdb.py", line 1685, in pdb.main() File "C:\dev\Python\x86\lib\pdb.py", line 1677, in main pdb.interaction(None, t) File "C:\dev\Python\x86\lib\pdb.py", line 346, in interaction self._cmdloop() File "C:\dev\Python\x86\lib\pdb.py", line 319, in _cmdloop self.cmdloop() File "C:\dev\Python\x86\lib\cmd.py", line 126, in cmdloop line = input(self.prompt) File "C:\dev\Python\x86\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 57: character maps to python version: Python 3.4.0 (v3.4.0:04f714765c13, Mar 16 2014, 19:24:06) [MSC v.1600 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. -- components: Windows messages: 223453 nosy: silverbacknet priority: normal severity: normal status: open title: pdb.set_trace() crashes when binary data is input via stdin type: crash versions: Python 3.4 ___ Python tracker <http://bugs.python.org/issue22009> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22009] pdb.set_trace() crashes with UnicodeDecodeError when binary data is input via stdin
Silverback Networks added the comment: Just verified the problem on 3.4.1 as well. -- title: pdb.set_trace() crashes when binary data is input via stdin -> pdb.set_trace() crashes with UnicodeDecodeError when binary data is input via stdin ___ Python tracker <http://bugs.python.org/issue22009> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22009] pdb.set_trace() crashes with UnicodeDecodeError when binary data is input via stdin
Silverback Networks added the comment: OK, now that I've screwed my thinking cap back on, it's obvious that pdb in the example is pulling from stdin. However, that isn't be the case in my original problem, so modify the example to: sample.py: import sys, pdb infile = sys.stdin.buffer.read() pdb.set_trace() Same error despite nothing being left in stdin's buffer. -- ___ Python tracker <http://bugs.python.org/issue22009> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
