[Twisted-Python] Twisted receiving buffers swamped?

[Tobias Oberstein]

Hi,

I am doing network performance tests using netperf on a trivial Twisted 
TCP echo server (code at the end).


One of the tests that netperf offers is throughput, and I am running 
into an issue with this.


When running the test (loopback) for 10 seconds on the test box I get a 
throughput of 11.5 Gb/s (which is not bad):


[oberstet@brummer1 ~]$ netperf -N -H 127.0.0.1 -t TCP_STREAM -l 10 -- -P 
9000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 9000 AF_INET to 127.0.0.1 () 
port 9000 AF_INET 
: no control : histogram : interval : dirty data : demo

Recv   SendSend
Socket Socket  Message  Elapsed
Size   SizeSize Time Throughput
bytes  bytes   bytessecs.10^6bits/sec

 0  32768  3276810.0211517.89

[Sidenote: when running against the netperf server, the box does 46Gb/s 
on that test. More results here: 
https://github.com/crossbario/crossbar/wiki/Stream-Testee#netperf]


However, when I run the test for 60 s (changing to "-l 60" in above will 
do), the test server is killed by the OS due to out-of-memory.


This screenshot

http://picpaste.com/pics/Clipboard03-DllXR7QE.1420107551.png

shows that the server at the time immediately before of killing 
allocated >30GB RAM.


In fact, memory also runs away with 10 sec test .. it's just that the 
machine has enough RAM to cope with that. So it's a "general" issue.


I tested with:

* CPython 2.7.9 and PyPy 2.4
* select, poll and kqueue reactors

all on FreeBSD 10.1. Same behavior for all combinations.

===

Now, my suspicion is that Twisted is reading off the TCP stack from the 
kernel and buffering in userspace faster than the echo server is pushing 
out stuff to the TCP stack into the kernel. Hence, no TCP backpressure 
results, netperf happily sends more and more, and the memory of the 
Twisted process runs away.


I am aware of 
http://twistedmatrix.com/documents/14.0.0/core/howto/producers.html, but 
that seems to cover the sending side only.


What's the cause? What can I do?

How do I prevent Twisted to read off sockets from kernel as the 
userspace buffer grows?


E.g. can I set a limit on the userspace buffer, so Twisted won't read 
out the sockets until the app has consumed more of the already buffered 
stuff?


Any hints appreciated,

Cheers,
/Tobias



TCP Echo Server used >

from twisted.internet import kqreactor
kqreactor.install()
#from twisted.internet import selectreactor
#selectreactor.install()
#from twisted.internet import pollreactor
#pollreactor.install()

from twisted.internet import protocol, reactor, endpoints

class Echo(protocol.Protocol):
def dataReceived(self, data):
self.transport.write(data)

class EchoFactory(protocol.Factory):
def buildProtocol(self, addr):
return Echo()

endpoints.serverFromString(reactor, "tcp:9000").listen(EchoFactory())

print "running on ", reactor.__class__
reactor.run()


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Silverberg CQL Driver

[James Broadhead]

On 29 December 2014 at 19:11, Jason J. W. Williams <
jasonjwwilli...@gmail.com> wrote:

> Hi James,
>
> Have you guys tried Silverberg? Wondering if its less flakly.
>
> -J
>

We haven't -- when I said that, I meant "on some occasions, after 3 years
of continuous use". It's a sporadic failure mode, and hasn't happened
enough times for us to investigate deeply. That said, we're in our own DC,
so the environment is pretty stable. If you're in AWS or similar, you might
want to run some testing around failure cases (let us know what you find!)
All that said, we've been very happy with it.

The likelihood is that we'll be working on migrating to Manhattan[1] rather
than trying out a new Cassandra library.
Not quite sure which way we'll go, but one option may be to write a
telephus-like wrapper around the thrift bindings.


[1]
https://blog.twitter.com/2014/manhattan-our-real-time-multi-tenant-distributed-database-for-twitter-scale
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted receiving buffers swamped?

[Glyph]

> On Jan 1, 2015, at 2:21 AM, Tobias Oberstein  
> wrote:
> 
> I am aware of 
> http://twistedmatrix.com/documents/14.0.0/core/howto/producers.html 
> , but 
> that seems to cover the sending side only.

It covers the receiving side as well.  If you pauseProducing() on a transport, 
it stops calling dataReceived on its transport.

> What's the cause? What can I do?

My initial hypothesis is that netperf is sending traffic but not bothering to 
receive it.

If this hypothesis is correct, then 
self.transport.registerProducer(self.transport) should solve the problem.  
Presuming that there is no problem with crossing the streams - I don't think 
i've ever done that particular incantation, and I'm almost shocked it's taken 
this long to come up :).

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python