[Twisted-Python] Gtk2Reactor: what are the reasons for avoiding GLib functions?

[Jason Heeris]

I'd like to try and get to the bottom of the various problems with
Twisted + GTK + Windows (mainly issue #4932[1], but see also issues
#4376[2], #4862[3], #1759[4]).

On issue #4376 there's a comment from Exarkun:

"glib handles timeouts much less efficiently than any of the existing
Twisted reactors. Letting glib take care of these would probably make
the glib-based reactors unusable for applications with many timeouts."

Is there a benchmark or piece of code somewhere that verifies this (or
at least gives some sort of demonstration of "unuseable")?

If not, is there some alternative implementation lying around in
version control that tried to use GLib's mainloop functions and was
thrown out for performance reasons?

I ask because my first approach to fixing this would be to ignore
what's already there and try to write something that's quite closely
coupled to the GLib/GTK mainloop. But it sounds like this was already
tried and abandoned, and I want to be able to make sure I'm not (a)
writing something that was already written, or (b) performs too poorly
to be of any use. It would be good to have a metric available.

Cheers,
Jason

[1] http://twistedmatrix.com/trac/ticket/4932
[2] http://twistedmatrix.com/trac/ticket/4376
[3] http://twistedmatrix.com/trac/ticket/4862
[4] http://twistedmatrix.com/trac/ticket/1759

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Gtk2Reactor: what are the reasons for avoiding GLib functions?

[Andrew Bennetts]

Jason Heeris wrote:
[…]
> "glib handles timeouts much less efficiently than any of the existing
> Twisted reactors. Letting glib take care of these would probably make
> the glib-based reactors unusable for applications with many timeouts."
> 
> Is there a benchmark or piece of code somewhere that verifies this (or
> at least gives some sort of demonstration of "unuseable")?

…and if someone does have such a benchmark, it'd be great to add it to
 :)

-Andrew.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Gtk2Reactor: what are the reasons for avoiding GLib functions?

[Itamar Turner-Trauring]

On Tue, 2011-04-05 at 15:08 +0800, Jason Heeris wrote:
> I'd like to try and get to the bottom of the various problems with
> Twisted + GTK + Windows (mainly issue #4932[1], but see also issues
> #4376[2], #4862[3], #1759[4]).

Thanks for looking into this!

> On issue #4376 there's a comment from Exarkun:
> 
> "glib handles timeouts much less efficiently than any of the existing
> Twisted reactors. Letting glib take care of these would probably make
> the glib-based reactors unusable for applications with many timeouts."
> 
> Is there a benchmark or piece of code somewhere that verifies this (or
> at least gives some sort of demonstration of "unuseable")?

https://bugzilla.gnome.org/show_bug.cgi?id=143061

I don't think it's ever going to get fixed on the gtk side, I filed that
bug in 2004. The issues on Windows may be different than this, for that
matter.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Gtk2Reactor: what are the reasons for avoiding GLib functions?

[Žiga Seilnacht]

Hello,

On 5.4.2011 9:08, Jason Heeris wrote:
> I'd like to try and get to the bottom of the various problems with
> Twisted + GTK + Windows (mainly issue #4932[1], but see also issues
> #4376[2], #4862[3], #1759[4]).
>
> On issue #4376 there's a comment from Exarkun:
>
> "glib handles timeouts much less efficiently than any of the existing
> Twisted reactors. Letting glib take care of these would probably make
> the glib-based reactors unusable for applications with many timeouts."
>
> Is there a benchmark or piece of code somewhere that verifies this (or
> at least gives some sort of demonstration of "unuseable")?
>

I think that the main reason for PortableGtkReactor being used on Windows
is due to glib's IO channels limitations on Windows. You can only have 63
channels in the event loop at the same time due to g_poll's use of
MsgWaitForMultipleObject.

In addition, IO channels are quite buggy on Windows when used for sockets,
see the comment at the top of the glib/giowin32.c file in the glib source
for a list of issues. E.g. I suspect that the second problem described in:

https://bugzilla.gnome.org/show_bug.cgi?id=357674

is responsible for Twisted's ticket #3371, because a similar problem
can be seen with win32eventreactor (ticket #4950), which also uses
WSAEventSelect for socket monitoring.

Best regards,
Ziga

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Gtk2Reactor: what are the reasons for avoiding GLib functions?

[Jason Heeris]

On 5 April 2011 20:06, Žiga Seilnacht  wrote:
> I think that the main reason for PortableGtkReactor being used on Windows
> is due to glib's IO channels limitations on Windows. You can only have 63
> channels in the event loop at the same time due to g_poll's use of
> MsgWaitForMultipleObject.

The trouble is, PortableGtkReactor is just out-and-out unusable for
certain applications. In my case, it turned a 3 minute comms session
into a 25 hour session — OR, I could have my app completely max out
the CPU while it's running. Neither is workable in any realistic
context. But I'm pretty sure I don't need > 63 I/O channels open at
once, unless I'm missing something about the inner workings of
Twisted.

(I mean no offense, I can see why it has to work the way it does...
but its weakness shows up in exactly the kind of protocol I need to
use.)

> In addition, IO channels are quite buggy on Windows when used for sockets,
> see the comment at the top of the glib/giowin32.c file in the glib source
> for a list of issues. E.g. I suspect that the second problem described in:
>
> https://bugzilla.gnome.org/show_bug.cgi?id=357674
>
> is responsible for Twisted's ticket #3371, because a similar problem
> can be seen with win32eventreactor (ticket #4950), which also uses
> WSAEventSelect for socket monitoring.

Hmm. I may be in over my head here. But I won't give up just yet,
since it *would* be nice to be able to use PyGTK for the things I need
to do.

Thanks for the info,
Jason

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Twisted 11.0 and Foolscap incompatibility

[Bill Noon]

It looks like the new ssl/tls code doesn't work with Foolscap.  The 
ConnectionMixin is treating the self.protocol
object as a transport and calls write() and loseConnection() on it.

Here is a traceback:
Traceback (most recent call last):
  File 
"/usr/lib64/python2.6/site-packages/foolscap-0.6.1_-py2.6.egg/foolscap/slicers/root.py",
 line 107, in send
d.callback(None)
  File 
"/usr/lib64/python2.6/site-packages/Twisted-11.0.0_r31548-py2.6-linux-x86_64.egg/twisted/internet/defer.py",
 line 361, in callback
self._startRunCallbacks(result)
  File 
"/usr/lib64/python2.6/site-packages/Twisted-11.0.0_r31548-py2.6-linux-x86_64.egg/twisted/internet/defer.py",
 line 455, in _startRunCallbacks
self._runCallbacks()
  File 
"/usr/lib64/python2.6/site-packages/Twisted-11.0.0_r31548-py2.6-linux-x86_64.egg/twisted/internet/defer.py",
 line 542, in _runCallbacks
current.result = callback(current.result, *args, **kw)
---  ---
  File 
"/usr/lib64/python2.6/site-packages/foolscap-0.6.1_-py2.6.egg/foolscap/banana.py",
 line 216, in produce
self.pushSlicer(slicer, obj)
  File 
"/usr/lib64/python2.6/site-packages/foolscap-0.6.1_-py2.6.egg/foolscap/banana.py",
 line 352, in pushSlicer
openID = self.sendOpen()
  File 
"/usr/lib64/python2.6/site-packages/foolscap-0.6.1_-py2.6.egg/foolscap/banana.py",
 line 481, in sendOpen
int2b128(openID, self.transport.write)
  File 
"/usr/lib64/python2.6/site-packages/foolscap-0.6.1_-py2.6.egg/foolscap/banana.py",
 line 24, in int2b128
stream(chr(0))
  File 
"/usr/lib64/python2.6/site-packages/Twisted-11.0.0_r31548-py2.6-linux-x86_64.egg/twisted/internet/_newtls.py",
 line 176, in write
self.protocol.write(bytes)
exceptions.AttributeError: 'Broker' object has no attribute 'write'


Any workarounds or suggestions?  Reverting to Twisted 10.2 works for now.

--Bill Noon
Northeast Regional Climate Center
Cornell University


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted SVN HEAD and Foolscap incompatibility

[Glyph Lefkowitz]

On Apr 5, 2011, at 3:03 PM, Bill Noon wrote:

> File 
> "/usr/lib64/python2.6/site-packages/Twisted-11.0.0_r31548-py2.6-linux-x86_64.egg/twisted/internet/_newtls.py",
>  line 176, in write

Nobody panic :).

r31548 of trunk is not Twisted 11.0.0.  Twisted 11.0 diverged from trunk as of 
.

Perhaps more tellingly, there is no _newtls.py in Twisted 11.0: 
.
  (Plus, I triple-checked the tarballs just now to make sure that they were 
released from the right place, and they were.)

That revision of trunk would include 
 though, which is some new SSL 
code _not_ included in the 11.0 release.

So - yes, this is a bug, but NOT in the 11.0 release.

Please file a bug, noting that it's a regression.  Depending on what level of 
nastiness foolscap is getting up to in order to trigger this behavior, we may 
have to revert .

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 11.0 and Foolscap incompatibility

[Glyph Lefkowitz]

On Apr 5, 2011, at 3:03 PM, Bill Noon wrote:

> Reverting to Twisted 10.2 works for now.

You should just revert to the actual released version of 11.0.  How did you 
install 11.0?  Is easy_install screen-scraping the wrong link again, or 
something like that?

By the way; correct version number or not, thanks for bringing this to our 
attention.  It's good to get feedback from user testing on features like this 
_before_ they make it into a release ;-).

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted SVN HEAD and Foolscap incompatibility

[Bill Noon]

On Apr 5, 2011, at 3:23 PM, Glyph Lefkowitz wrote:

> 
> On Apr 5, 2011, at 3:03 PM, Bill Noon wrote:
> 
>>File 
>> "/usr/lib64/python2.6/site-packages/Twisted-11.0.0_r31548-py2.6-linux-x86_64.egg/twisted/internet/_newtls.py",
>>  line 176, in write
> 
> Nobody panic :).
Sorry about misstating that this was in the 11.0 release.

> 
> r31548 of trunk is not Twisted 11.0.0.  Twisted 11.0 diverged from trunk as 
> of .
> 
> Perhaps more tellingly, there is no _newtls.py in Twisted 11.0: 
> .
>   (Plus, I triple-checked the tarballs just now to make sure that they were 
> released from the right place, and they were.)
> 
> That revision of trunk would include 
>  though, which is some new SSL 
> code _not_ included in the 11.0 release.
> 
> So - yes, this is a bug, but NOT in the 11.0 release.
> 
> Please file a bug, noting that it's a regression.  Depending on what level of 
> nastiness foolscap is getting up to in order to trigger this behavior, we may 
> have to revert .

Ok.  It is ticket 5028, http://twistedmatrix.com/trac/ticket/5028

--Bill

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 11.0 and Foolscap incompatibility

[Bill Noon]

On Apr 5, 2011, at 4:02 PM, Glyph Lefkowitz wrote:

> 
> On Apr 5, 2011, at 3:03 PM, Bill Noon wrote:
> 
>> Reverting to Twisted 10.2 works for now.
> 
> You should just revert to the actual released version of 11.0.  How did you 
> install 11.0?  Is easy_install screen-scraping the wrong link again, or 
> something like that?

No, my fault for running from svn_head.  

> By the way; correct version number or not, thanks for bringing this to our 
> attention.  It's good to get feedback from user testing on features like this 
> _before_ they make it into a release ;-).

Just doing my part...

--Bill


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [Twisted-web] Creating a hybrid server in Twisted

[Glyph Lefkowitz]

On Apr 4, 2011, at 7:51 PM, David Bolen wrote:

> Jashank Jeremy  writes:
> 
>> So I'm pretty much stuck in a rut.  I don't want to totally reinvent the
>> wheel just to be able to protocol-switch; I'd prefer to make use of
>> existing code from Twisted.  How do I hijack Twisted.Web to add protocol
>> switching?
> 

> To insert your own protocol code, you want to set the "protocol"
> attribute of your Site (or HTTPFactory if using that directly)
> instance to your own "hybrid" protocol class.

It would be slightly better to override buildProtocol, because that way you can 
call the superclass's implementation more idiomatically.  But, the idea is the 
same.

Note that we'd really like to have protocol switch functionality built into 
Twisted proper so that you don't have to jump through all these hoops.  Please 
feel free to contribute patches to fix this ticket: 
.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted SVN HEAD and Foolscap incompatibility

[Glyph Lefkowitz]

On Apr 5, 2011, at 4:04 PM, Bill Noon wrote:

> Ok.  It is ticket 5028, http://twistedmatrix.com/trac/ticket/5028

As exarkun already noted here: 
, this is caused by 
foolscap manipulating an attribute which probably shouldn't have been public in 
the first place, and *definitely* isn't mutable.  This means that there are 
several other types of instrumentation which foolscap will break.

This is a popular ticket this week - I just mentioned it in another thread: 
.  If foolscap needs to switch 
protocols mid-stream, it either needs to do what AMP does, or contribute a real 
fix for that ticket.  (And then require a new version of Twisted :).)

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python