[Twisted-Python] Twisted 16.3.2 Release Announcement

[Amber "Hawkie" Brown]

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release 
of Twisted 16.3.1.

This is a security fix release, and is recommended for all users of Twisted. 
The fixes contained are:

- Twisted's HTTP server, when operating over TLS, would not cleanly close 
sockets, causing it to build up CLOSE_WAIT sockets until it would eventually 
run out of file descriptors.

For more information, check the NEWS file (link provided below).

You can find the downloads at > (or alternatively 
>). The NEWS file is also 
available at >.

Many thanks to everyone who had a part in this release - the supporters of the 
Twisted Software Foundation, the developers who contributed code as well as 
documentation, and all the people building great things with Twisted!

Twisted Regards,
Amber Brown (HawkOwl)


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] How do you determine the buffer size of a transport - a use-case for not using back pressure

[Steve Morin]

Anyone know how do you determine the buffer size of a transport, to know
how much data is waiting to be transmitted from using transport.write?

Or how you would go about adding that ability to a reactor/transport?

On Wed, Aug 17, 2016 at 3:43 PM, Steve Morin  wrote:

> Twisted Community
>
> Problem: How do you determine the buffer size of a transport, to know how
> much data is waiting to be transmitted from using transport.write?
>
> Wait! You're going to say: use the Producer Consumer API (
> http://twistedmatrix.com/documents/current/core/howto/producers.html )
>
> To do what: So that instead of using back pressure I can check the buffer
> and when it's "too big/full" can decide to do something to the transport I
> am writing to:
>
> Slow transport handling options:
>
> - Buffer to disk instead of memory
> - Kill the transport
> - Decide to skip sending some data
> - Send an error or message to the transport I am writing to
> - Reduce the resolution, increase the compression (things like video or
> audio)
>
> Why not use back pressure?: Some use-cases and some protocols this doesn't
> make sense.
>
> Examples:
> - You're sending video and if the receiver can't keep up you want to
> downgrade or upgrade the quality of the video, but if you don't know if you
> can't tell how much it buffering.
> - You're receiving from one connection and then broadcasting what you
> received to multiple clients and you want to handle it by sending an error
> to a client that doesn't keep up
> - You're sending a real-time protocol and want to skip sending some data
> that's no longer relevant if the buffer is too slow.
>
> On a server what are the consequences:
> Too much buffering in many transport write buffer can cause a server to
> fail.  I don't know how to keep track of this to proactively without access
> to the buffer sizes.  Resolutions can then be to, not accept new
> connections when memory pressure is too high, kill connections with the
> weakest/slowest clients or have a protocol that can have client switch
> connections to new servers when instructed to spread out the load.
> 1) I would like to hear on how people would solve this sort of problem in
> Twisted for a server?
> 2) I would like to hear people opinions on this in general.
>
> Tobias Oberstein - BCC'ed you on this email because you seems to have
> tackled similar problems (based on the mailing list) and would really love
> to get your take on this too.
>
> Glyph and Jean-Paul, you're also big on those threads so any opinions you
> have would be appreciated as well.
>
> Some of my background research
>
> * http://twistedmatrix.com/pipermail/twisted-python/2015-
> January/029064.html
> * Later but good in the chain: http://twistedmatrix.com/
> pipermail/twisted-python/2015-January/029071.html
>   * Twisted receiving buffers swamped?
>   * Summary: Great thread but runs into a tangent for a while and picks up
> good at the end again discussing producer again and the need for
> backpressure
>   * Other:
> * http://crossbar.io/
> * http://tavendo.com/
>   * Scenario: "Twisted is reading off the TCP stack from the kernel and
> buffering in userspace faster than the echo server is pushing out stuff to
> the TCP stack into the kernel. Hence, no TCP backpressure results, netperf
> happily sends more and more, and the memory of the Twisted process runs
> away."
>   * Confirmed: Data isn't Buffered in "userspace inside Twisted, and
> before data is handled by the app in dataReceived."
> * https://twistedmatrix.com/pipermail/twisted-python/2010-
> September/022900.html
>   * How to cap the buffering size of data to be sent in Protocol class
>   * Summary: Same issue, very short no good info
> * https://twistedmatrix.com/pipermail/twisted-python/2012-
> March/025416.html
>   * Limit on transport.write
>   * Summary: Similar issue, very short no good info, glyph confirms that
> transport.write buffers everything sent to it.
> * https://twistedmatrix.com/pipermail/twisted-python/2012-
> February/025215.html
>   * pushing out same message on 100k TCPs
>   * Summary: Interesting discussion but different issue - interesting
> aside about irc spanning trees for a large broadcasts
> * http://twistedmatrix.com/pipermail/twisted-python/2008-April/017503.html
>   * Question on push/pull producers inter-workings, was : "Is there a
> simple Producer/Consumer example or tutorial?"
>   * Summary: Related and goes into what a producer is - an explanation of
> it
> * http://twistedmatrix.com/pipermail/twisted-python/2004-May/007732.html
>   * When do calls to transport.write() block ?
>   * Summary: Discusses the right issue, talks about buffer call back if
> full which would be great (if configurable)
> * https://pythonhosted.org/qbuf/qbuf.twisted_support.
> MultiBufferer-class.html
>   * https://launchpad.net/qbuf
>   * Summary: c buffer implementation that's incomplete thought might be
> promising
> * http://stackoverflow.com/questions/21821119/twisted-
> producer-

Re: [Twisted-Python] How do you determine the buffer size of a transport - a use-case for not using back pressure

[Jean-Paul Calderone]

There are at least two buffers (per direction) you might be interested in.
You can get the kernel buffer size with getsockopt - SO_SNDBUF and
SO_RCVBUF.  Then, there may also be a user-space buffer (or perhaps more
than one) managed by the transport implementation.  The details of this are
entirely up to the transport so there's no general answer apart from "read
the implementation, contribute a patch implementing a public interface for
retrieving the information".

Jean-Paul

On Fri, Aug 19, 2016 at 10:08 AM, Steve Morin  wrote:

> Anyone know how do you determine the buffer size of a transport, to know
> how much data is waiting to be transmitted from using transport.write?
>
> Or how you would go about adding that ability to a reactor/transport?
>
> On Wed, Aug 17, 2016 at 3:43 PM, Steve Morin 
> wrote:
>
>> Twisted Community
>>
>> Problem: How do you determine the buffer size of a transport, to know how
>> much data is waiting to be transmitted from using transport.write?
>>
>> Wait! You're going to say: use the Producer Consumer API (
>> http://twistedmatrix.com/documents/current/core/howto/producers.html )
>>
>> To do what: So that instead of using back pressure I can check the buffer
>> and when it's "too big/full" can decide to do something to the transport I
>> am writing to:
>>
>> Slow transport handling options:
>>
>> - Buffer to disk instead of memory
>> - Kill the transport
>> - Decide to skip sending some data
>> - Send an error or message to the transport I am writing to
>> - Reduce the resolution, increase the compression (things like video or
>> audio)
>>
>> Why not use back pressure?: Some use-cases and some protocols this
>> doesn't make sense.
>>
>> Examples:
>> - You're sending video and if the receiver can't keep up you want to
>> downgrade or upgrade the quality of the video, but if you don't know if you
>> can't tell how much it buffering.
>> - You're receiving from one connection and then broadcasting what you
>> received to multiple clients and you want to handle it by sending an error
>> to a client that doesn't keep up
>> - You're sending a real-time protocol and want to skip sending some data
>> that's no longer relevant if the buffer is too slow.
>>
>> On a server what are the consequences:
>> Too much buffering in many transport write buffer can cause a server to
>> fail.  I don't know how to keep track of this to proactively without access
>> to the buffer sizes.  Resolutions can then be to, not accept new
>> connections when memory pressure is too high, kill connections with the
>> weakest/slowest clients or have a protocol that can have client switch
>> connections to new servers when instructed to spread out the load.
>> 1) I would like to hear on how people would solve this sort of problem in
>> Twisted for a server?
>> 2) I would like to hear people opinions on this in general.
>>
>> Tobias Oberstein - BCC'ed you on this email because you seems to have
>> tackled similar problems (based on the mailing list) and would really love
>> to get your take on this too.
>>
>> Glyph and Jean-Paul, you're also big on those threads so any opinions you
>> have would be appreciated as well.
>>
>> Some of my background research
>>
>> * http://twistedmatrix.com/pipermail/twisted-python/2015-Janua
>> ry/029064.html
>> * Later but good in the chain: http://twistedmatrix.com/piper
>> mail/twisted-python/2015-January/029071.html
>>   * Twisted receiving buffers swamped?
>>   * Summary: Great thread but runs into a tangent for a while and picks
>> up good at the end again discussing producer again and the need for
>> backpressure
>>   * Other:
>> * http://crossbar.io/
>> * http://tavendo.com/
>>   * Scenario: "Twisted is reading off the TCP stack from the kernel and
>> buffering in userspace faster than the echo server is pushing out stuff to
>> the TCP stack into the kernel. Hence, no TCP backpressure results, netperf
>> happily sends more and more, and the memory of the Twisted process runs
>> away."
>>   * Confirmed: Data isn't Buffered in "userspace inside Twisted, and
>> before data is handled by the app in dataReceived."
>> * https://twistedmatrix.com/pipermail/twisted-python/2010-Sept
>> ember/022900.html
>>   * How to cap the buffering size of data to be sent in Protocol class
>>   * Summary: Same issue, very short no good info
>> * https://twistedmatrix.com/pipermail/twisted-python/2012-Marc
>> h/025416.html
>>   * Limit on transport.write
>>   * Summary: Similar issue, very short no good info, glyph confirms that
>> transport.write buffers everything sent to it.
>> * https://twistedmatrix.com/pipermail/twisted-python/2012-Febr
>> uary/025215.html
>>   * pushing out same message on 100k TCPs
>>   * Summary: Interesting discussion but different issue - interesting
>> aside about irc spanning trees for a large broadcasts
>> * http://twistedmatrix.com/pipermail/twisted-python/2008-April
>> /017503.html
>>   * Question on push/pull producers inter-workings, was : "Is there a
>>