Re: [Twisted-Python] kqueue reactor / ticket #1918
> > if there is anything I can do to clarify (like i.e. running on
> > non-virtualized
> hardware FreeBSD) let me know.
>
> Have you run the tests on OS X as well?
I'll do that.
> > Caveat: as attached to the ticket, there are some errs when running
> > trial which don't happen with select reactor.
>
> Do you think you could start fixing some of those errors? Do they make
I've looked at the logs in more detail, the error _diff_ between select/kqueue
yields:
twisted.test.test_tcp.HalfCloseBuggyApplicationTests.test_readNotificationRaises
twisted.test.test_tcp.HalfCloseTestCase.testCloseWriteCloser
twisted.test.test_tcp.HalfCloseTestCase.testWriteCloseNotification
which all seem to be testing TCP Half-Close.
I've looked at the case code .. guess I can grasp that after reading some
background.
But I have no clue where to start "fixing" / looking for why this works with
the select reactor vs kqueue.
> sense to you? If you're interested in moving this ticket forward, I'm sure
> that
> folks on this mailing list (me included) would love to provide you some
> guidance.
>
Ok, will see what I can do.
Need some advice/hints though.
Generally, there are a couple of skipped tests missing modules (i.e. pycrypto)
which I could install. Should I do that, or are those not necessary?
Note, that there are packages I could, but would not like to install: GTK etc.
Then, whats the difference between FAIL and ERROR?
Then, there are a couple of tests which are skipped because
xxx_KQueueReactor
No module named kqsyscall
The kqsyscall should now be select.kqueue/kevent .. so I guess those
test cases needed to be adapted as well, right?
Then, baseline. I've run the trial using select reactor in the expectation of
finding all tests passed.
This is not the case, there are a couple of failed ones. Those are also failed
with the kqueue reactor, so I need to know whether I need to inspect those
or those are problems unrelated to the reactor, but of general (platform?)
nature.
[1]
twisted.internet.test.test_posixprocess
FileDescriptorTests
test_expectedFDs ... [FAIL]
[2]
twisted.internet.test.test_process
PTYProcessTestsBuilder_PollReactor
test_openFileDescriptors ...[ERROR]
PTYProcessTestsBuilder_SelectReactor
test_openFileDescriptors ...[ERROR]
ProcessTestsBuilder_PollReactor
test_openFileDescriptors ...[ERROR]
ProcessTestsBuilder_SelectReactor
test_openFileDescriptors ...[ERROR]
[3]
twisted.python.test.test_components
RegistrationTestCase
test_duplicateAdapterForInterfaceAllowed ...[ERROR]
test_registerAdapterForInterface ...[ERROR]
test_subclassAdapterRegistrationForInterface ...[ERROR]
Now, [2] could be a consequence of [3], since they all fail because of
exceptions.ValueError: an adapter ( at 0x2c0dc41c>) was
already registered.
See:
twisted.internet.test.test_process.PTYProcessTestsBuilder_PollReactor.test_openFileDescriptors
twisted.internet.test.test_process.PTYProcessTestsBuilder_SelectReactor.test_openFileDescriptors
twisted.internet.test.test_process.ProcessTestsBuilder_PollReactor.test_openFileDescriptors
twisted.internet.test.test_process.ProcessTestsBuilder_SelectReactor.test_openFileDescriptors
===
[ERROR]
Traceback (most recent call last):
File
"/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/python/test/test_components.py",
line 351, in test_duplicateAdapterForInterfaceAllowed
return self._duplicateAdapterForClassOrInterfaceAllowed(TheOriginal)
File
"/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/python/test/test_components.py",
line 312, in _duplicateAdapterForClassOrInterfaceAllowed
components.registerAdapter(firstAdapter, original, TheInterface)
File
"/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/python/components.py",
line 93, in registerAdapter
raise ValueError("an adapter (%s) was already registered." % (factory, ))
exceptions.ValueError: an adapter ( at 0x2c0dc41c>) was
already registered.
twisted.python.test.test_components.RegistrationTestCase.test_duplicateAdapterForInterfaceAllowed
===
[ERROR]
Traceback (most recent call last):
File
"/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/python/test/test_components.py",
line 268, in test_registerAdapterForInterface
return self._registerAdapterForClassOrInterface(TheOriginal)
F
Re: [Twisted-Python] kqueue reactor / ticket #1918
> We've said e.g. Windows only supports Python 2.5 or later even when > everything else supports Python 2.4. And we have inlineCallbacks which only > runs on Python 2.5 or later. Saying "kqueue reactor only works on Python 2.6 > and later" seems fine to me. For completeness: there seem to be 2 packages providing the select.kqueue/epoll interface backported for Python 2.3-2.5: http://pypi.python.org/pypi/select26/0.1a3 http://pypi.python.org/pypi/select_backport/0.2 However, both seem to be based on Python svn at some point in time, and not actively maintained. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] kqueue reactor / ticket #1918
On Sat, 2011-11-05 at 04:48 -0700, Tobias Oberstein wrote: > Then, whats the difference between FAIL and ERROR? Fail means an assertion failed. Error means an unexpected exception was throwhn. > Then, there are a couple of tests which are skipped because > > xxx_KQueueReactor > No module named kqsyscall > > The kqsyscall should now be select.kqueue/kevent .. so I guess those > test cases needed to be adapted as well, right? Yes. > Then, baseline. I've run the trial using select reactor in the expectation of > finding all tests passed. > > This is not the case, there are a couple of failed ones. Those are also failed > with the kqueue reactor, so I need to know whether I need to inspect those > or those are problems unrelated to the reactor, but of general (platform?) > nature. Are you using Twisted 11? Many of these issues are, if I'm not mistaken, fixed in trunk (though perhaps not all). So trunk is probably a better baseline to work off of. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] kqueue reactor / ticket #1918
> > Then, baseline. I've run the trial using select reactor in the
> > expectation of finding all tests passed.
> >
> > This is not the case, there are a couple of failed ones. Those are
> > also failed with the kqueue reactor, so I need to know whether I need
> > to inspect those or those are problems unrelated to the reactor, but of
> general (platform?) nature.
>
> Are you using Twisted 11? Many of these issues are, if I'm not mistaken,
> fixed in trunk (though perhaps not all). So trunk is probably a better
> baseline
> to work off of.
Ok, I'm using trunk now.
1)
The errors in
twisted.python.test.test_components
are gone.
2)
The errors in
twisted.internet.test.test_posixprocess
twisted.internet.test.test_process
are still there ... and they show up for select, poll and kqueue reactors.
They all somehow relate to "open FDs":
twisted.internet.test.test_posixprocess
FileDescriptorTests
test_expectedFDs ... [FAIL]
twisted.internet.test.test_process
PTYProcessTestsBuilder_KQueueReactor
test_openFileDescriptors ...[ERROR]
PTYProcessTestsBuilder_PollReactor
test_openFileDescriptors ...[ERROR]
PTYProcessTestsBuilder_SelectReactor
test_openFileDescriptors ...[ERROR]
ProcessTestsBuilder_SelectReactor
test_openFileDescriptors ...[ERROR]
This might more a FreeBSD related thing. ?
3)
I think I've nailed down the Half-Close stuff.
4)
These are kqueue specific. Need to investigate further.
twisted.internet.test.test_fdset
ReactorFDSetTestsBuilder_KQueueReactor
test_lostFileDescriptor ... [ERROR]
test_removedFromReactor ... [FAIL]
The first one comes up, since
removeReader()
is called for a FD which is gone, and leads to a socket exception from fd =
reader.fileno()
5)
These are kqueue specific. Need to investigate further.
twisted.conch.test.test_conch
The fail on the kqeue control call .. see below.
Any ideas what could be the reason?
[autobahn@autobahnhub ~/temp]$ trial -r select twisted.conch.test.test_conch
twisted.conch.test.test_conch
CmdLineClientTestCase
test_exec ... [OK]
test_localToRemoteForwarding ... [OK]
test_remoteToLocalForwarding ... [OK]
OpenSSHClientForwardingTestCase
test_exec ... [OK]
test_localToRemoteForwarding ... [OK]
test_remoteToLocalForwarding ... [OK]
OpenSSHClientRekeyTestCase
test_clientRekey ... [OK]
StdioInteractingSessionTests
test_eofReceived ... [OK]
---
Ran 8 tests in 4.707s
PASSED (successes=8)
[autobahn@autobahnhub ~/temp]$ trial -r kqueue twisted.conch.test.test_conch
twisted.conch.test.test_conch
CmdLineClientTestCase
test_exec ... [OK]
test_localToRemoteForwarding ...[ERROR]
[ERROR]
test_remoteToLocalForwarding ... [OK]
OpenSSHClientForwardingTestCase
test_exec ... [OK]
test_localToRemoteForwarding ...[ERROR]
[ERROR]
test_remoteToLocalForwarding ... [OK]
OpenSSHClientRekeyTestCase
test_clientRekey ... [OK]
StdioInteractingSessionTests
test_eofReceived ... [OK]
===
[ERROR]
Traceback (most recent call last):
Failure: twisted.internet.defer.TimeoutError:
(test_localToRemoteForwarding) still
running at 120.0 secs
twisted.conch.test.test_conch.CmdLineClientTestCase.test_localToRemoteForwarding
===
[ERROR]
Traceback (most recent call last):
File
"/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/python/log.py",
line 84, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File
"/home/autobahn/python/lib/python2.7/site-packages/Twisted-11.0.0_r33077-py2.7-freebsd-8.2-RELEASE-p3-i386.egg/twisted/python/log.py",
line 69, in callWithContext
re
[Twisted-Python] Weekly Bug Summary
Bug summary __ Summary for 2011-10-30 through 2011-11-06 Opened Closed Total Change Enhancements: 11 6770 +5 Defects: 13 4519 +9 Tasks: 2 0 73 +2 Regressions: 0 1 3 -1 Total:26 11 1365+15 |== Type Changes |== Priority Changes |== Component Changes |Defect: +9 |Highest: +1 |Conch: +2 |Enhancement: +5 |High: +2 |Core: +11 |Regression: -1 |Normal: +14 |Ftp: +0 |Task: +2 |Low: -2 |Names: +0 |Release Management: +1 |Web: +1 |Words: +0 Total Tickets Open Tickets New / Reopened Bugs __ = Highest = [#5350] Elliptic Curve cryptography for public key authentication (opened by termim) defect conch http://twistedmatrix.com/trac/ticket/5350 = High = [#5347] registerProducer and unregisterProducer need special treatment in twisted.internet._newtls (opened by itamar) defect core http://twistedmatrix.com/trac/ticket/5347 [#5368] epoll (and perhaps other reactors) busy loops when file descriptor limit is hit (opened by itamar) defect core http://twistedmatrix.com/trac/ticket/5368 = Normal = [#5345] it's very difficult to debug response generation with twisted.web.client.Agent (opened by glyph) (CLOSED, duplicate) enhancement core http://twistedmatrix.com/trac/ticket/5345 [#5346] log.msg goes into infinite loop if it has two recursive observers (opened by zseil) defect core http://twistedmatrix.com/trac/ticket/5346 [#5348] Use sort(key=...) in SRVConnector (opened by antoine) (CLOSED, duplicate) enhancement names http://twistedmatrix.com/trac/ticket/5348 [#5349] Avoid relying on the str() of a class object (opened by antoine) (CLOSED, fixed) enhancement conch http://twistedmatrix.com/trac/ticket/5349 [#5351] MultiService class documentation makes the nonsense claim that "No service will wait for another" (opened by exarkun) defect core http://twistedmatrix.com/trac/ticket/5351 [#5352] do not register overlapping old DH key exchange messages (opened by antoine) defect conch http://twistedmatrix.com/trac/ticket/5352 [#5353] provide a "fake" IReactorCore for testing (opened by glyph) enhancement core http://twistedmatrix.com/trac/ticket/5353 [#5354] Create a real, tested, in-memory loopback transport that has fine-grained controls and doesn't run the real reactor (opened by glyph) enhancement core http://twistedmatrix.com/trac/ticket/5354 [#5355] twisted.internet.protocol.FileWrapper and twisted.test.proto_helpers.StringTransport are redundant (opened by glyph) taskcore http://twistedmatrix.com/trac/ticket/5355 [#5356] test_iocp's _acceptAddressTest should use port as SO_UPDATE_ACCEPT_CONTEXT (opened by zseil) defect core http://twistedmatrix.com/trac/ticket/5356 [#5357] build-news doesn't work properly between pre-releases (opened by therve) defect release management http://twistedmatrix.com/trac/ticket/5357 [#5358] tcp client endpoint syntax should support positional arguments (opened by glyph) enhancement core http://twistedmatrix.com/trac/ticket/5358 [#5359] when the threadpool is too busy to do what I want, it should log what it's doing so I can fix my program (opened by glyph) enhancement core http://twistedmatrix.com/trac/ticket/5359 [#5360] twistd has no option to run without a pidfile (opened by habnabit) enhancement core http://twistedmatrix.com/trac/ticket/5360 [#5361] Add persistent connection support to `twisted.web.client.ProxyAgent` (opened by itamar) enhancement webhttp://twistedmatrix.com/trac/ticket/5361 [#5362] Fix Windows detection in t.p.lockfile (opened by antoine) (CLOSED, fixed) defect core http://twistedmatrix.com/trac/ticket/5362 [#5363] IRC client howto (opened by jonathanj) enhancement words http://twistedmatrix.com/trac/ticket/5363 [#5364] Avoid indexing WindowsError objects (opened by antoine) defect core http://twistedmatrix.com/trac/ticket/5364 [#5365] _pollingfile makes wrong assumption about WriteFile (opened by antoine) defect core http://twistedmatrix.com/trac/ticket/5365 [#5366] Expand threading section of logging howto (opened by itamar) defect core
