Re: [Twisted-Python] Experimenting with tubes

[Glyph]

On Aug 19, 2014, at 3:58 PM, Jan Pobrislo  wrote:

> On Thu, 14 Aug 2014 14:29:56 -0700
> Glyph Lefkowitz  wrote:
> 
>> On Aug 11, 2014, at 6:51 AM, c...@webprojekty.cz wrote:
>> 
>>> Hello, I've been playing with the new tubes that are being
>>> implemented:
>>> http://comments.gmane.org/gmane.comp.python.twisted/27248
>>> https://twistedmatrix.com/trac/ticket/1956
>> 
>> Thanks so much for taking the time to play with it, and taking some
>> time to write feedback.
>> 
>>> Here are few things that I did with it. I won't publish the full
>>> code now, as in it's current shape it could implode eyeballs of
>>> twisted devs and possibly make them summon some of the elder gods,
>>> but I'll see if I can produce something less vile as I merge the
>>> ongoing changes to the tubes branch.
>> 
>> I'd be interested to see the code nevertheless.  If you had to do
>> eyeball-imploding antics to get Tubes to work well for your use-case,
>> being able to have a look at that would help us evaluate whether
>> those antics were required by the code, encouraged by misfeatures of
>> the API design, or just issues with lack of documentation.
> 
> It's mostly me not really documenting anything, not writing tests and
> littering it with debug statements (which will go away as soon as I
> find time to improve my debugging module so it can monkeypatch them).

You should be able to debug tubes mostly with composition to analyze flows.  If 
you can't do what you want with that, we should talk :-).

> http://wpr.cz/ccx/bzr/tubes7/ for my changes only
> http://wpr.cz/ccx/bzr/tubes7-merge-2/ for changes on top of the bzr
> mirror of the svn branch
> 
> example usage:
> http://wpr.cz/ccx/paste/2014-08-19/2/
> http://wpr.cz/ccx/paste/2014-08-19/3/
> http://wpr.cz/ccx/paste/2014-08-19/4/
> http://wpr.cz/ccx/paste/2014-08-19/5/

I think I don't understand the purpose of all of these.  Particularly, what is 
the purpose of TypedTube, since Tube already supports specification of input 
and output types?

>>> So far I wrote relatively simple app that read logfiles, parse them
>>> and insert what they got out of them into a database.
>> 
>> If it's actually reading a file, another nice to-do would be an
>> IFount provider that provides the contents of a file with appropriate
>> flow control, and maybe a thread or process in the background to do
>> the file I/O.  Another thing you could contribute to the branch,
>> possibly?  :-)  How did you implement this?
> 
> At the moment I don't mind the blockingness of the calls. I did write a
> ThreadReader and ThreadWriter though for my earlier tubes-alike with
> Queue-based loop.
> 
> What is more interesting challenge (and we discussed this earlier on
> irc) would be generic async file api. I suggested implementing 9p2000
> back then and I still think it is a good starting point... but nothing
> I have spare time for at the moment.

The point is not to use a specific implementation.  Rather, the point is to get 
a single well-documented entry-point within Twisted for asynchronously reading 
a file so that people can start using it.  Frankly this entrypoint could be a 
total lie and actually do the I/O synchronously on the main thread, as long as 
it could be transparently upgraded to being the truth without exposing the 
change to applications in the future :-).  If we make everyone implement their 
own read-a-file fount, then there's no hope that future Twisted maintenance 
could improve their performance.

>> I'm not sure I totally understand the case that you're describing
>> right now.  Can you perhaps contribute a unit test which demonstrates
>> why this line of code is necessary?
> 
> I'd love to, alas I'll be bit preocuppied with some more urgent matters
> for following week or two. The short version is "flowStopped just
> didn't get passed through the series otherwise".

I hope I'll have some time later in the week to investigate this. 

>> Are you running into ?
> 
> Most probably, as far as I can tell from the vague description.

Well, you'll be glad to know that's also on my to-do list :-).

>> That ... definitely sounds kind of gross.  As does actually setting
>> the nextFount attribute directly on the fan.Out.
> 
> Indeed. The point of the experiment was not produce nice code but to
> see if there are any major pitfalls using the tubes API.

As long as it's clear that this is not really necessary :).

>> twisted.web.client.Agent has a solution to this where there's a
>> multi-failure object that aggregates multiple errors into one thing.
>> I think we have to do something similar.  Unfortunately this is a
>> very confusing interface in addition to being poorly documented and
>> relies on private classes that expose ostensibly public attributes.
>> We need to very carefully document this within fan.In.
> 
> Some nice abstraction of multiple failures would be indeed handy. I'm
> pretty sure DeferredList could use one too.

Yeah, uh, maybe.  Als

[Twisted-Python] Twisted WebSockets Branch

[Rob Oakes]

Dear Twisted Developers,

As a quick introduction, my name is Rob Oakes. I'm the lead developer 
for a company called Guru Labs. I'm writing for two reasons:


# Reason 1: Thank You

First and foremost, I'm writing to express thanks for creating an 
excellent framework. We use Twisted extensively in our infrastructure, 
and it is typically our go-to tool for any sort of custom server.


The other two reasons have to do with some questions about the 
development of Twisted. Before I dive into those, however, let me 
provide some background.


For the better part of a year, we've been using some code out of the 
websockets (twisted.web.websockets.WebsocketsResource) branch to wrap a 
trio of custom protocols we use in one of our web based products. I 
know that the code is still pre-release, but we've generally found it 
to be stable and work very well.


# Reason Two: Websockets Development

This brings me to the second reason I'm writing. Over the past year, it 
seems as though the development on the Twisted websockets branch has 
stalled. We would like to unstall it. For this reason, Guru Labs would 
be interested in:


1.) Contracting with one of the core Twisted devs of Twisted who might 
be interested in finishing the implementation of the websockets 
wrappers (resolving the issues described in ticket 4173) or another 
party. We are happy to pay hourly rates, set a bounty, make a donation 
to the Twisted project ... whatever.


A stable implementation of websockets available in Twisted Web is a 
priority to us, and if it's a matter of money, we are happy to throw 
money where it might be needed. (If this of interest, please contact me 
off-list at roa...@gurulabs.com.)


or

2.) Assigning a Guru Labs developer (probably me) who might complete 
the work.


I've been studying the issues which are still open (specifically 4173), 
and the various branches associated with websocket development 
(websocket-4173-3 and websocket-4173-4) and it seems that most of the 
major concerns are related to the manner in which websocket connections 
are closed.


Assuming that someone else doesn't step forward ... I've already merged 
the most recent version of trunk with these branches, but I've found 
myself with several questions on how to best continue with the work.


a. Which of the two websocket branches should be used as the basis for 
new work?


On GitHub, websocket-4173-4 is marked as closed with a note telling the 
contributor to see the contribution guidelines. There are also several 
additions to the code which do not follow Twisted conventions (as I 
understand them). The last set of commits also seem to come from 
approximately the same time.


websocket-4173-4 includes code which attempts to resolve issues noted 
in 4173 that is not present in 4173-3, but there aren't really any 
comments to determine if this should be incorporated or not. At this 
point, I'm really not familiar enough with the code to draw my own 
conclusion.


b. In general, the protocol wrapper seems to work quite well. However 
when merged with the most recent version of trunk (post twisted 14), 
I've been seeing frequent  unhandled deferred errors. The most common 
is:


"twisted.internet.error.ConnectionLost: Connection to the other side 
was lost in a non-clean fashion: Connection lost"


It happens when:

* Connections are closed from the server, using transport.loseConnection
* Connections do not transmit a "close" frame prior to disconnecting
* To reiterate, this issue only started appearing after merging the 
websocket code with Twisted 14. The previous version of Twisted we were 
using (Twisted 12.3 worked flawlessly.)


No error is raised if the client correctly closes the connection or 
when using non-browser based clients (like the Python ws4py websocket 
client).


We mostly see the exception when the objects are garbage collected 
(based upon the deferred documentation at 
http://twistedmatrix.com/documents/14.0.0/core/howto/defer.html). Put 
another way, we see a whole string of errors upon stopping the reactor. 
Despite the exception, we don't see any errors in the browser client.


Also somewhat frustratingly, the traceback isn't terribly helpful. This 
is a fairly routine example:


```error
Unhandled error in Deferred:

Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionLost: Connection to the other 
side was lost in a non-clean fashion: Connection lost.

```

Technically, the error is probably appropriate, as they appear when the 
connection is lost in a non-clean fashion. I am a little concerned, 
though, in that I haven't found a good way to catch or suppress the 
error. Moreover, it doesn't seem like the defferreds are getting 
garbage collected which seems like a memory leak waiting to happen.


Can someone clarify if this is:

* intended behavior, and if so, what might be a strategy I can use for 
managing the error in my wrapped protocols
* which part of the websocket code I should