[Twisted-Python] kqueue reactor / ticket #1918

[Tobias Oberstein]

> Is there even a kqueue-based Twisted reactor?
> 
> Yes... ish.  More detail here: .

Hi Glyph,

first, thanks as always!

The pykqueue 1.3 and 2.0 referenced in that ticket seem to be no longer
maintained.

On the other hand, Python has built-in support for kqueue since Python 2.6
in the select module.

I have adjusted the Twisted kqreactor to select.kqueue and made a package:

https://github.com/oberstet/txkqreactor

I have tested that reactor successfully with Autobahn WebSockets on FreeBSD 8.2
(running as a VirtualBox x86 VM). When I'm home later, I can check out Mac.

Doing the Autobahn tests helped me finding the only non-canonical change 
required

https://github.com/oberstet/txkqreactor/commit/27302b4f643cd3967289ff11d811309a0b05f630

Is there any regression/test/stresstest I could run to verify the stuff 
actually works?

The Autobahn WS tests do quite some stuff, but they do not test concurrency and
high connections numbers ..

Cheers,
Tobias

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] kqueue reactor / ticket #1918

[exarkun]

On 02:11 pm, tobias.oberst...@tavendo.de wrote:

Is there even a kqueue-based Twisted reactor?

Yes... ish.  More detail here: 
.


Hi Glyph,

first, thanks as always!

The pykqueue 1.3 and 2.0 referenced in that ticket seem to be no longer
maintained.

On the other hand, Python has built-in support for kqueue since Python 
2.6

in the select module.

I have adjusted the Twisted kqreactor to select.kqueue and made a 
package:


https://github.com/oberstet/txkqreactor

I have tested that reactor successfully with Autobahn WebSockets on 
FreeBSD 8.2
(running as a VirtualBox x86 VM). When I'm home later, I can check out 
Mac.


Doing the Autobahn tests helped me finding the only non-canonical 
change required


https://github.com/oberstet/txkqreactor/commit/27302b4f643cd3967289ff11d811309a0b05f630

Is there any regression/test/stresstest I could run to verify the stuff 
actually works?


You can run the Twisted test suite:

   trial --reactor kqueue twisted

(assuming you've hooked up the name "kqueue" to your new version of the 
kqueue reactor).


This is what needs to work in order for the new reactor to be added to 
Twisted itself (ie, for us to resolve #1918).


Jean-Paul
The Autobahn WS tests do quite some stuff, but they do not test 
concurrency and

high connections numbers ..

Cheers,
Tobias

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] kqueue reactor / ticket #1918

[Tobias Oberstein]

> >Is there any regression/test/stresstest I could run to verify the stuff
> >actually works?
> 
> You can run the Twisted test suite:
> 
> trial --reactor kqueue twisted
> 
> (assuming you've hooked up the name "kqueue" to your new version of the
> kqueue reactor).
> 
> This is what needs to work in order for the new reactor to be added to
> Twisted itself (ie, for us to resolve #1918).

Hello Jean-Paul,

I've ran the new one on a Python 2.7.1/Twisted 11.0 (stock) by patching

twisted_reactor.py

#kqueue = Reactor(
#'kqueue', 'twisted.internet.kqreactor', 'kqueue(2)-based reactor.')
kqueue = Reactor(
'kqueue', 'txkqreactor.kqreactor', 'NEW kqueue(2)-based reactor.')

and running both that, and the select() reactor.

Uploaded all logs to the ticket.

The diff between both:

http://twistedmatrix.com/trac/attachment/ticket/1918/select_vs_new_kqueue_diff.log

Summary:

< Ran 6515 tests in 102.948s
---
> Ran 6515 tests in 343.371s
9292c9354
< FAILED (skips=977, expectedFailures=11, failures=1, errors=7, successes=5519)
---
> FAILED (skips=977, expectedFailures=11, failures=2, errors=13, successes=5516)

< = select
> = new kqueue

Is that good, bad or ugly?


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] kqueue reactor / ticket #1918

[Kevin Horn]

On Fri, Nov 4, 2011 at 9:11 AM, Tobias Oberstein <
tobias.oberst...@tavendo.de> wrote:

> > Is there even a kqueue-based Twisted reactor?
> >
> > Yes... ish.  More detail here: <
> http://twistedmatrix.com/trac/ticket/1918>.
>
> Hi Glyph,
>
> first, thanks as always!
>
> The pykqueue 1.3 and 2.0 referenced in that ticket seem to be no longer
> maintained.
>
> On the other hand, Python has built-in support for kqueue since Python 2.6
> in the select module.
>
> I have adjusted the Twisted kqreactor to select.kqueue and made a package:
>
> https://github.com/oberstet/txkqreactor
>
> I have tested that reactor successfully with Autobahn WebSockets on
> FreeBSD 8.2
> (running as a VirtualBox x86 VM). When I'm home later, I can check out Mac.
>
> Doing the Autobahn tests helped me finding the only non-canonical change
> required
>
>
> https://github.com/oberstet/txkqreactor/commit/27302b4f643cd3967289ff11d811309a0b05f630
>
> Is there any regression/test/stresstest I could run to verify the stuff
> actually works?
>
> The Autobahn WS tests do quite some stuff, but they do not test
> concurrency and
> high connections numbers ..
>
> Cheers,
> Tobias
>
>
This is awesome, BUT...

...would probably have to be distributed outside of Twisted proper (at
least for the moment), as Twisted supports versions of Python < 2.6, where
select.kqueue would not exist.

(Maybe this is your plan, but I'm just pointing it out.)

Kevin Horn
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] kqueue reactor / ticket #1918

[Itamar Turner-Trauring]

> This is awesome, BUT...
>
> ...would probably have to be distributed outside of Twisted proper (at
> least for the moment), as Twisted supports versions of Python < 2.6, where
> select.kqueue would not exist.
>
> (Maybe this is your plan, but I'm just pointing it out.)

We've said e.g. Windows only supports Python 2.5 or later even when
everything else supports Python 2.4. And we have inlineCallbacks which
only runs on Python 2.5 or later. Saying "kqueue reactor only works on
Python 2.6 and later" seems fine to me.

-Itamar


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] kqueue reactor / ticket #1918

[exarkun]

On 04:49 pm, kevin.h...@gmail.com wrote:
>On Fri, Nov 4, 2011 at 9:11 AM, Tobias Oberstein <
>tobias.oberst...@tavendo.de> wrote:
>> > Is there even a kqueue-based Twisted reactor?
>> >
>> > Yes... ish.  More detail here: <
>>http://twistedmatrix.com/trac/ticket/1918>.
>>
>>Hi Glyph,
>>
>>first, thanks as always!
>>
>>The pykqueue 1.3 and 2.0 referenced in that ticket seem to be no 
>>longer
>>maintained.
>>
>>On the other hand, Python has built-in support for kqueue since Python 
>>2.6
>>in the select module.
>>
>>I have adjusted the Twisted kqreactor to select.kqueue and made a 
>>package:
>>
>>https://github.com/oberstet/txkqreactor
>>
>>I have tested that reactor successfully with Autobahn WebSockets on
>>FreeBSD 8.2
>>(running as a VirtualBox x86 VM). When I'm home later, I can check out 
>>Mac.
>>
>>Doing the Autobahn tests helped me finding the only non-canonical 
>>change
>>required
>>
>>
>>https://github.com/oberstet/txkqreactor/commit/27302b4f643cd3967289ff11d811309a0b05f630
>>
>>Is there any regression/test/stresstest I could run to verify the 
>>stuff
>>actually works?
>>
>>The Autobahn WS tests do quite some stuff, but they do not test
>>concurrency and
>>high connections numbers ..
>>
>>Cheers,
>>Tobias
>This is awesome, BUT...
>
>...would probably have to be distributed outside of Twisted proper (at
>least for the moment), as Twisted supports versions of Python < 2.6, 
>where
>select.kqueue would not exist.
>
>(Maybe this is your plan, but I'm just pointing it out.)

It would be fine for Twisted to offer this feature only on Python 2.6 
and newer.

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] kqueue reactor / ticket #1918

[Tobias Oberstein]

> >This is awesome, BUT...
> >
> >...would probably have to be distributed outside of Twisted proper (at
> >least for the moment), as Twisted supports versions of Python < 2.6,
> >where select.kqueue would not exist.
> >
> >(Maybe this is your plan, but I'm just pointing it out.)
> 
> It would be fine for Twisted to offer this feature only on Python 2.6 and
> newer.

Would be great of course to have it in Twisted.

In the meantime, I've packaged it up

http://pypi.python.org/pypi/txkqreactor

Caveat: as attached to the ticket, there are some errs when running trial
which don't happen with select reactor.

Whether those are significant I can't tell .. if there is anything I can do to
clarify (like i.e. running on non-virtualized hardware FreeBSD) let me know.




___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] kqueue reactor / ticket #1918

[Glyph]

On Nov 4, 2011, at 1:11 PM, Tobias Oberstein wrote:

>>> This is awesome, BUT...
>>> 
>>> ...would probably have to be distributed outside of Twisted proper (at
>>> least for the moment), as Twisted supports versions of Python < 2.6,
>>> where select.kqueue would not exist.
>>> 
>>> (Maybe this is your plan, but I'm just pointing it out.)
>> 
>> It would be fine for Twisted to offer this feature only on Python 2.6 and
>> newer.
> 
> Would be great of course to have it in Twisted.

Yes :).

> In the meantime, I've packaged it up
> 
> http://pypi.python.org/pypi/txkqreactor

Thanks, but we should really try to integrate it ASAP. :)

> Caveat: as attached to the ticket, there are some errs when running trial
> which don't happen with select reactor.

Do you think you could start fixing some of those errors?  Do they make sense 
to you?  If you're interested in moving this ticket forward, I'm sure that 
folks on this mailing list (me included) would love to provide you some 
guidance.

> Whether those are significant I can't tell ..

If they're failing tests, they're significant :).  In order for stuff to be 
added to Twisted, the full test suite needs to run.

> if there is anything I can do to clarify (like i.e. running on 
> non-virtualized hardware FreeBSD) let me know.

Have you run the tests on OS X as well?

-glyph


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python