[Twisted-Python] print unicode
Hello!
I'm using Twisted 10.0 and as usually sometime print debug infos with
myunicodestr.encode('UTF-8') which are saved to logfile, but since
using twisted 10 I'm getting
UnicodeEncodeError: 'ascii' codec can't encode characters...
type(myunicodestr) returns
What is the problem here?
Thanks!
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] print unicode
On Wednesday 05 May 2010, Pet wrote:
> I'm using Twisted 10.0 and as usually sometime print debug infos with
> myunicodestr.encode('UTF-8') which are saved to logfile, but since
> using twisted 10 I'm getting
>
> UnicodeEncodeError: 'ascii' codec can't encode characters...
UTF-8 uses the full 8 bits of a byte, while ASCII only uses 7, so writing
Unicode encoded as UTF-8 to an ASCII stream is not valid.
I think recent Python versions are more strict about what is written to
stdout/stderr than older versions, it might not be related to Twisted
itself. You can specify a different encoding for stdin/out/err by setting
the PYTHONIOENCODING environment variable.
Bye,
Maarten
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] print unicode
On Wed, May 5, 2010 at 2:05 PM, Maarten ter Huurne
wrote:
> On Wednesday 05 May 2010, Pet wrote:
>
>> I'm using Twisted 10.0 and as usually sometime print debug infos with
>> myunicodestr.encode('UTF-8') which are saved to logfile, but since
>> using twisted 10 I'm getting
>>
>> UnicodeEncodeError: 'ascii' codec can't encode characters...
>
> UTF-8 uses the full 8 bits of a byte, while ASCII only uses 7, so writing
> Unicode encoded as UTF-8 to an ASCII stream is not valid.
>
> I think recent Python versions are more strict about what is written to
> stdout/stderr than older versions, it might not be related to Twisted
> itself. You can specify a different encoding for stdin/out/err by setting
> the PYTHONIOENCODING environment variable.
Hi Maarten!
Thanks for help!
Unfortunately, my Python installation is 2.5.2 and PYTHONIOENCODING is
introduced in 2.6
Pet
>
> Bye,
> Maarten
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] print unicode
On 05/05/10 13:31, Pet wrote:
> On Wed, May 5, 2010 at 2:05 PM, Maarten ter Huurne
> wrote:
>> On Wednesday 05 May 2010, Pet wrote:
>>
>>> I'm using Twisted 10.0 and as usually sometime print debug infos with
>>> myunicodestr.encode('UTF-8') which are saved to logfile, but since
>>> using twisted 10 I'm getting
>>>
>>> UnicodeEncodeError: 'ascii' codec can't encode characters...
>>
>> UTF-8 uses the full 8 bits of a byte, while ASCII only uses 7, so writing
>> Unicode encoded as UTF-8 to an ASCII stream is not valid.
>>
>> I think recent Python versions are more strict about what is written to
>> stdout/stderr than older versions, it might not be related to Twisted
>> itself. You can specify a different encoding for stdin/out/err by setting
>> the PYTHONIOENCODING environment variable.
>
> Hi Maarten!
>
> Thanks for help!
> Unfortunately, my Python installation is 2.5.2 and PYTHONIOENCODING is
> introduced in 2.6
I think this is highly dependent on your OS environment. For example:
Python 2.4.3 (#1, Oct 23 2006, 14:19:47)
[GCC 4.1.1 20060525 (Red Hat 4.1.1-1)] o
Type "help", "copyright", "credits" or "
>>> import sys
>>> sys.getdefaultencoding()
'ascii'
>>> print unichr(163)
£
[p...@wildfire ~]$ echo $LANG
en_GB.UTF-8
...but:
LANG=C python
Python 2.4.3 (#1, Oct 23 2006, 14:19:47)
[GCC 4.1.1 20060525 (Red Hat 4.1.1-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print unichr(163)
Traceback (most recent call last):
File "", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa3' in
position 0: ordinal not in range(128)
...i.e. here I can just print unicode characters, with nothing
particularly special, provided my environment variables are set right.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] print unicode
On Wed, 2010-05-05 at 13:45 +0200, Pet wrote:
> Hello!
>
> I'm using Twisted 10.0 and as usually sometime print debug infos with
> myunicodestr.encode('UTF-8') which are saved to logfile, but since
> using twisted 10 I'm getting
>
> UnicodeEncodeError: 'ascii' codec can't encode characters...
>
> type(myunicodestr) returns
>
>
> What is the problem here?
This works fine for me (Twisted trunk):
$ python2.5 -c "import sys; from twisted.python import log; \
log.startLogging(file('/tmp/log', 'w')); print \
u'\u1234'.encode('UTF-8')"
$ cat /tmp/log
2010-05-05 08:48:40-0400 [-] Log opened.
2010-05-05 08:48:40-0400 [-] ሴ
Can you include a minimal reproducing example?
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] SQLAlchemy and Twisted
Hi all, I've been doing some searching about how to get SQLAlchemy and Twisted working together in a Twisted application. Though I've found a lot of information, I haven't seen (or figured out) a good working solution or definitive answer. The most promising one I've run across concerns running the SQLAlchemy queries in a separate process (rather than a separate thread) and communicating the queries between the Twisted application in one process and the SQLAlchemy application in another. Does anyone have any good pointers, suggestions, ideas, links to how I might go about setting something like this up? Here's a couple questions that come to mind: 1) Would the SQLAlchemy process also be a Twisted application with all the queries running as deferreds in the main thread, and blocking? 2) How would the Twisted process communicate with the SQLAlchemy process, using something like XMLRPC, calling methods to perform the queries? Or would the XMLRPC methods convey something more generic like SQL? Thanks in advance for any help! Doug ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] print unicode
On Wed, May 5, 2010 at 2:47 PM, Phil Mayers wrote:
> On 05/05/10 13:31, Pet wrote:
>> On Wed, May 5, 2010 at 2:05 PM, Maarten ter Huurne
>> wrote:
>>> On Wednesday 05 May 2010, Pet wrote:
>>>
I'm using Twisted 10.0 and as usually sometime print debug infos with
myunicodestr.encode('UTF-8') which are saved to logfile, but since
using twisted 10 I'm getting
UnicodeEncodeError: 'ascii' codec can't encode characters...
>>>
>>> UTF-8 uses the full 8 bits of a byte, while ASCII only uses 7, so writing
>>> Unicode encoded as UTF-8 to an ASCII stream is not valid.
>>>
>>> I think recent Python versions are more strict about what is written to
>>> stdout/stderr than older versions, it might not be related to Twisted
>>> itself. You can specify a different encoding for stdin/out/err by setting
>>> the PYTHONIOENCODING environment variable.
>>
>> Hi Maarten!
>>
>> Thanks for help!
>> Unfortunately, my Python installation is 2.5.2 and PYTHONIOENCODING is
>> introduced in 2.6
>
> I think this is highly dependent on your OS environment. For example:
>
> Python 2.4.3 (#1, Oct 23 2006, 14:19:47)
> [GCC 4.1.1 20060525 (Red Hat 4.1.1-1)] o
> Type "help", "copyright", "credits" or "
> >>> import sys
> >>> sys.getdefaultencoding()
> 'ascii'
> >>> print unichr(163)
> £
on python console it works for me, but not in application, if the
string to be printed is fetched from db which is utf-8
>
>
> [p...@wildfire ~]$ echo $LANG
> en_GB.UTF-8
>
> ...but:
>
> LANG=C python
> Python 2.4.3 (#1, Oct 23 2006, 14:19:47)
> [GCC 4.1.1 20060525 (Red Hat 4.1.1-1)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> print unichr(163)
> Traceback (most recent call last):
> File "", line 1, in ?
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xa3' in
> position 0: ordinal not in range(128)
>
> ...i.e. here I can just print unicode characters, with nothing
> particularly special, provided my environment variables are set right.
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] print unicode
On Wed, May 5, 2010 at 2:49 PM, Itamar Turner-Trauring
wrote:
> On Wed, 2010-05-05 at 13:45 +0200, Pet wrote:
>> Hello!
>>
>> I'm using Twisted 10.0 and as usually sometime print debug infos with
>> myunicodestr.encode('UTF-8') which are saved to logfile, but since
>> using twisted 10 I'm getting
>>
>> UnicodeEncodeError: 'ascii' codec can't encode characters...
>>
>> type(myunicodestr) returns
>>
>>
>> What is the problem here?
>
> This works fine for me (Twisted trunk):
>
> $ python2.5 -c "import sys; from twisted.python import log; \
> log.startLogging(file('/tmp/log', 'w')); print \
> u'\u1234'.encode('UTF-8')"
> $ cat /tmp/log
> 2010-05-05 08:48:40-0400 [-] Log opened.
> 2010-05-05 08:48:40-0400 [-] ሴ
>
> Can you include a minimal reproducing example?
If I print as you do it works, but my string is fetched from database
and only then it fails
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Reactor Spinning?
Hey Itamar, the problem is that this only occurs under heavy load in a production environment (of course!), so I don't really have any way of testing a simplified version. The code itself is fairly simple, however, and is almost identical to the code that ships in twisted.web.proxy. This is to say that I'm simply shuttling data between an HTTPClient object and a Request object. I never call transport.resumeProducing() or startWriting() myself, and I only call transport.write() in exactly one place (when sending POST data via HTTPClient). The rest of the time I'm writing to either the Request object (not its transport) when communicating back to the clients, or availing myself of higher-level methods like HTTPClient.sendHeader() when writing to the server side. The code is available here: http://www.googlesharing.net/server/googleshare-0.9.tar.gz - moxie -- http://www.thoughtcrime.org Itamar Turner-Trauring wrote: > Can you produce a minimal reproducing example of a program that has this > issue? I can imagine a code bug where you're calling > transport.resumeProducing() or startWriting() sometime when you shouldn't, > except the empty write() does look wrong... a minimal code sample would > help. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Reactor Spinning?
I've confirmed that when spinning the FD corresponds with a client connection. What would you suggest logging that might provide some insight? Given what I understand about manholes, the fact that this only occurs in a production environment which must remain high performance means that might not be an option for me. - moxie -- http://www.thoughtcrime.org exar...@twistedmatrix.com wrote: > It might be interesting to learn what's in the reactor that's associated > with this file descriptor and what the state of its send buffer is. > Some strategically placed log messages might help with that, or a > manhole in the process that you can connect to when you notice the > problem has started in order to poke around. > > Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] print unicode
On Wed, May 5, 2010 at 4:15 PM, Pet wrote:
> On Wed, May 5, 2010 at 2:49 PM, Itamar Turner-Trauring
> wrote:
>> On Wed, 2010-05-05 at 13:45 +0200, Pet wrote:
>>> Hello!
>>>
>>> I'm using Twisted 10.0 and as usually sometime print debug infos with
>>> myunicodestr.encode('UTF-8') which are saved to logfile, but since
>>> using twisted 10 I'm getting
>>>
>>> UnicodeEncodeError: 'ascii' codec can't encode characters...
>>>
>>> type(myunicodestr) returns
>>>
>>>
>>> What is the problem here?
>>
>> This works fine for me (Twisted trunk):
>>
>> $ python2.5 -c "import sys; from twisted.python import log; \
>> log.startLogging(file('/tmp/log', 'w')); print \
>> u'\u1234'.encode('UTF-8')"
>> $ cat /tmp/log
>> 2010-05-05 08:48:40-0400 [-] Log opened.
>> 2010-05-05 08:48:40-0400 [-] ሴ
>>
>> Can you include a minimal reproducing example?
>
> If I print as you do it works, but my string is fetched from database
> and only then it fails
It's pretty weird. I've send as parameter {'s': u'c\u0142a'} to
twisted xml-rpc server after it was restarted and it has printed
param['s'].encode('UTF-8') without errors. Immidiately after that I've
send the same request again and it failed to print it. I've restarted
the server again and at the first request it prints without errors,
all other requests raise exceptions. So it has nothing to do with
database.
>
>>
>>
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com
>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] print unicode
On Wed, 2010-05-05 at 16:15 +0200, Pet wrote: > If I print as you do it works, but my string is fetched from database > and only then it fails Strings coming out of the database are usually just strings. There's nothing database (or Twisted) specific about them. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] print unicode
On Wed, May 5, 2010 at 4:29 PM, Pet wrote:
> On Wed, May 5, 2010 at 4:15 PM, Pet wrote:
>> On Wed, May 5, 2010 at 2:49 PM, Itamar Turner-Trauring
>> wrote:
>>> On Wed, 2010-05-05 at 13:45 +0200, Pet wrote:
Hello!
I'm using Twisted 10.0 and as usually sometime print debug infos with
myunicodestr.encode('UTF-8') which are saved to logfile, but since
using twisted 10 I'm getting
UnicodeEncodeError: 'ascii' codec can't encode characters...
type(myunicodestr) returns
What is the problem here?
>>>
>>> This works fine for me (Twisted trunk):
>>>
>>> $ python2.5 -c "import sys; from twisted.python import log; \
>>> log.startLogging(file('/tmp/log', 'w')); print \
>>> u'\u1234'.encode('UTF-8')"
>>> $ cat /tmp/log
>>> 2010-05-05 08:48:40-0400 [-] Log opened.
>>> 2010-05-05 08:48:40-0400 [-] ሴ
>>>
>>> Can you include a minimal reproducing example?
>>
>> If I print as you do it works, but my string is fetched from database
>> and only then it fails
>
> It's pretty weird. I've send as parameter {'s': u'c\u0142a'} to
> twisted xml-rpc server after it was restarted and it has printed
> param['s'].encode('UTF-8') without errors. Immidiately after that I've
> send the same request again and it failed to print it. I've restarted
> the server again and at the first request it prints without errors,
> all other requests raise exceptions. So it has nothing to do with
> database.
Now, I'm getting Exception with
File
"/usr/local/tw10/lib/python2.5/site-packages/Twisted-10.0.0-py2.5-linux-x86_64.egg/twisted/python/log.py",
line 555, in write
d = (self.buf + data).split('\n')
exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte
0xc5 in position 4: ordinal not in range(128)
>
>>
>>>
>>>
>>> ___
>>> Twisted-Python mailing list
>>> Twisted-Python@twistedmatrix.com
>>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>>
>>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Reactor Spinning?
On 05/05/10 15:22, Moxie Marlinspike wrote: > > I've confirmed that when spinning the FD corresponds with a client > connection. What would you suggest logging that might provide some insight? > > Given what I understand about manholes, the fact that this only occurs > in a production environment which must remain high performance means > that might not be an option for me. A manhole is really just a listening connection which spawns a python interpreter. It doesn't consume any CPU adding a manhole to a twisted server, and very little CPU accessing it (unless you do something *in* the manhole that consumes a lot of CPU) ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] print unicode
On Wed, 2010-05-05 at 16:47 +0200, Pet wrote:
> Now, I'm getting Exception with
>
> File
> "/usr/local/tw10/lib/python2.5/site-packages/Twisted-10.0.0-py2.5-linux-x86_64.egg/twisted/python/log.py",
> line 555, in write
> d = (self.buf + data).split('\n')
> exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte
> 0xc5 in position 4: ordinal not in range(128)
Are you logging/printing unencoded unicode strings (i.e. type(s) ==
unicode)? Twisted does not support that.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] print unicode
On Wed, May 5, 2010 at 4:54 PM, Itamar Turner-Trauring
wrote:
> On Wed, 2010-05-05 at 16:47 +0200, Pet wrote:
>> Now, I'm getting Exception with
>>
>> File
>> "/usr/local/tw10/lib/python2.5/site-packages/Twisted-10.0.0-py2.5-linux-x86_64.egg/twisted/python/log.py",
>> line 555, in write
>> d = (self.buf + data).split('\n')
>> exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte
>> 0xc5 in position 4: ordinal not in range(128)
>
> Are you logging/printing unencoded unicode strings (i.e. type(s) ==
> unicode)? Twisted does not support that.
No, this exception occurs if I do print myunicodestring.encode('UTF-8')
As I said before, it doesn't happen at first request after server restart.
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] SQLAlchemy and Twisted
Doug Farrell wrote: > I’ve been doing some searching about how to get SQLAlchemy and Twisted > working together in a Twisted application. Though I’ve found a lot of > information, I haven’t seen (or figured out) a good working solution or > definitive answer. The most promising one I’ve run across concerns > running the SQLAlchemy queries in a separate process (rather than a > separate thread) and communicating the queries between the Twisted > application in one process and the SQLAlchemy application in another. > Does anyone have any good pointers, suggestions, ideas, links to how I > might go about setting something like this up? The best advice I've gotten was from David Bolen; you can find an interchange between him and me on the SA list on October 22 and 23 of last year. The upshot of it is, it shouldn't be a problem to use the SQL level if you're careful to keep the database accesses in a separate thread (and Bolen has done that); using the ORM level, however, can be problematic if you're tempted to access ORM objects in the main thread (since you're not directly in control of when database accesses occur). > Here’s a couple questions that come to mind: > > 1) Would the SQLAlchemy process also be a Twisted application with > all the queries running as deferreds in the main thread, and blocking? There might be value in reworking the SA concepts into a Twisted package, so that the asynchrony is "built in". I haven't heard of any indications of that happening. > 2) How would the Twisted process communicate with the SQLAlchemy > process, using something like XMLRPC, calling methods to perform the > queries? Or would the XMLRPC methods convey something more generic like SQL? Well, Bolen used a dedicated worker thread to do the SA operations (all SQL level), passing functions to it to be executed in that context. He also used a single connection in the thread to do all DB operations. -- Don Dwiggins Advanced Publishing Technology ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] SQLAlchemy and Twisted
Doug Farrell wrote: > > I’ve been doing some searching about how to get SQLAlchemy and Twisted > working together in a Twisted application. Short version: to be safe, anything that touches any SQLAlchemy-mapped object needs to be run in its own thread. Any query or access of an attribute of a mapped object may result in a blocking sql query. (aka: twisted doesn't play nice with orms) > definitive answer. The most promising one I’ve run across concerns > running the SQLAlchemy queries in a separate process (rather than a > separate thread) and communicating the queries between the Twisted > application in one process and the SQLAlchemy application in another. That seems a little odd. What would be the IPC? How would the "sqlachemy application" be run? > 1) Would the SQLAlchemy process also be a Twisted application with > all the queries running as deferreds in the main thread, and blocking? What do you men by "all the queries"? > Thanks in advance for any help! In my case, since most of the app I'm working on is "web requested" (either xmlrpc or http), I just agve up and used a good wsgi stack (repoze.bfg in my case) and munge other incoming requests into wsgi requests. Twisted's wsgi server runs each request in its own thread, so be it. cheers, Chris ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Reactor Spinning?
> > I've confirmed that when spinning the FD corresponds with a client > connection. What would you suggest logging that might provide some > insight? What is the transport's writeSomeData() method doing? Is it actually trying to write an empty string? Transition from/to writeable state may also be interesting, along with logging size of transport's buffer (so logging in stopWriting/startWriting). ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
