Hi,
I am doing network performance tests using netperf on a trivial Twisted
TCP echo server (code at the end).
One of the tests that netperf offers is throughput, and I am running
into an issue with this.
When running the test (loopback) for 10 seconds on the test box I get a
throughput of 11.5 Gb/s (which is not bad):
[oberstet@brummer1 ~]$ netperf -N -H 127.0.0.1 -t TCP_STREAM -l 10 -- -P
9000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 9000 AF_INET to 127.0.0.1 ()
port 9000 AF_INET
: no control : histogram : interval : dirty data : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
0 32768 32768 10.02 11517.89
[Sidenote: when running against the netperf server, the box does 46Gb/s
on that test. More results here:
https://github.com/crossbario/crossbar/wiki/Stream-Testee#netperf]
However, when I run the test for 60 s (changing to "-l 60" in above will
do), the test server is killed by the OS due to out-of-memory.
This screenshot
http://picpaste.com/pics/Clipboard03-DllXR7QE.1420107551.png
shows that the server at the time immediately before of killing
allocated >30GB RAM.
In fact, memory also runs away with 10 sec test .. it's just that the
machine has enough RAM to cope with that. So it's a "general" issue.
I tested with:
* CPython 2.7.9 and PyPy 2.4
* select, poll and kqueue reactors
all on FreeBSD 10.1. Same behavior for all combinations.
===
Now, my suspicion is that Twisted is reading off the TCP stack from the
kernel and buffering in userspace faster than the echo server is pushing
out stuff to the TCP stack into the kernel. Hence, no TCP backpressure
results, netperf happily sends more and more, and the memory of the
Twisted process runs away.
I am aware of
http://twistedmatrix.com/documents/14.0.0/core/howto/producers.html, but
that seems to cover the sending side only.
What's the cause? What can I do?
How do I prevent Twisted to read off sockets from kernel as the
userspace buffer grows?
E.g. can I set a limit on the userspace buffer, so Twisted won't read
out the sockets until the app has consumed more of the already buffered
stuff?
Any hints appreciated,
Cheers,
/Tobias
TCP Echo Server used ====>
from twisted.internet import kqreactor
kqreactor.install()
#from twisted.internet import selectreactor
#selectreactor.install()
#from twisted.internet import pollreactor
#pollreactor.install()
from twisted.internet import protocol, reactor, endpoints
class Echo(protocol.Protocol):
def dataReceived(self, data):
self.transport.write(data)
class EchoFactory(protocol.Factory):
def buildProtocol(self, addr):
return Echo()
endpoints.serverFromString(reactor, "tcp:9000").listen(EchoFactory())
print "running on ", reactor.__class__
reactor.run()
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
