Re: [Twisted-Python] Twisted receiving buffers swamped?

2015-01-02 Thread Tobias Oberstein
>>I am aware of  >>http://twistedmatrix.com/documents/14.0.0/core/howto/producers.html, but that >>seems to cover the sending side only. >It covers the receiving side as well.  If you pauseProducing() on a transport, >it stops calling dataReceived on its transport. Not sure I understand that. B

Re: [Twisted-Python] Twisted receiving buffers swamped?

2015-01-02 Thread Glyph
> On Jan 2, 2015, at 2:25 AM, Tobias Oberstein > wrote: > > Unfortunately, it doesn't seem to work (the problem persists): Your "streaming" flag is wrong. A TCP transport is an IPushProducer (it will produce data without being asked). Try setting it to True and see if that helps? -g__

Re: [Twisted-Python] Twisted receiving buffers swamped?

2015-01-02 Thread exarkun
On 1 Jan, 10:21 am, tobias.oberst...@tavendo.de wrote: Hi, I am doing network performance tests using netperf on a trivial Twisted TCP echo server (code at the end). One of the tests that netperf offers is throughput, and I am running into an issue with this. ? [snip] Now, my suspicion is

Re: [Twisted-Python] Twisted receiving buffers swamped?

2015-01-02 Thread Tobias Oberstein
>>Unfortunately, it doesn't seem to work (the problem persists): >Your "streaming" flag is wrong.  A TCP transport is an IPushProducer (it will >produce data without being asked).  Try setting it to True and see if that >helps? With streaming == True and cProfile added https://github.com/oberst

Re: [Twisted-Python] Twisted receiving buffers swamped?

2015-01-02 Thread Tobias Oberstein
> >Now, my suspicion is that Twisted is reading off the TCP stack from the > >kernel and buffering in userspace faster than the echo server is > >pushing out stuff to the TCP stack into the kernel. Hence, no TCP > >backpressure results, netperf happily sends more and more, and the > >memory of the

Re: [Twisted-Python] Twisted receiving buffers swamped?

2015-01-02 Thread exarkun
On 12:06 pm, tobias.oberst...@tavendo.de wrote: Fact is: somehow memory runs away. How do I track down _where_ exactly the mem is spent? Probably that leads to the "why" then .. There are memory profiling tools for Python. For example, memory_profiler: https://pypi.python.org/pypi/mem

Re: [Twisted-Python] Twisted receiving buffers swamped?

2015-01-02 Thread Glyph Lefkowitz
> On Jan 2, 2015, at 4:06 AM, Tobias Oberstein > wrote: > > Fact is: somehow memory runs away. > > How do I track down _where_ exactly the mem is spent? Probably that leads to > the "why" then .. The first place to look - since sometimes looking in a specific place makes memory profilers ea