Re: [Twisted-Python] Is there pb documentation somewhere?

[Glyph]

On Aug 18, 2014, at 2:25 PM, Daniel Sank  wrote:

> It's just like the case of a GUI and a business logic object. The GUI 
> probably gets a reference to the business logic object so that eg. button 
> pushes can invoke methods on the object. However, that reference should 
> probably be weak so that the business logic object can be garbage collected 
> when it's finished with its business. There's no sense (to me) in keeping an 
> object alive because a GUI, logger, or other observer is observing it. Am I 
> just wrong?

When it comes to GUI toolkits, there are two philosophies on this.

One, embodied in toolkits like OS X's Cocoa and (if you squint at it just 
right) Qt, is that this reference should always be weak (or, you know, __unsafe 
__unretained which is like "weak" with a bit of a speech impediment) because 
something else (a window management layer, for example, or a data-access layer 
updating some data) will probably be holding the reference.  This is popular in 
C-style toolkits with an object model and reference counting because there's 
often an implicit circular reference between a view and its controller, and 
cleaning that up in C or C++ can be messy.

Another, embodied in toolkits like GTK+ and the JavaScript DOM, is that this 
reference should always be strong, because the GUI can logically manipulate the 
model object it refers to, and so it should have a strong reference - otherwise 
GUI actions might spontaneously start causing crashes when something unrelated 
forgets about that object.

I am a big fan of the latter style.  Although there is often something to hold 
that strong reference, sometimes there is actually nothing else to hold it, and 
so you have to create bizarre lifecycle shenanigans to replicate the fairly 
straightforward behavior of "the user's eyeballs are looking at the screen, 
there's a window on the screen, the window refers to my model object, therefore 
the user's eyeballs have a strong reference to my model object".  Some things 
that present GUIs are observers, some things are manipulators; the former model 
works for observers, the latter model works well for both.

So I'm inclined to say you're wrong.  However, according to the 
efficient-market hypothesis, Cocoa must be better than any of those other 
things, so I may be in a minority there.

Nevertheless in PB the distinction is even more stark: if your example is that 
you have a model object with a GUI observer, it is the GUI that would expose 
the Referenceable, because the model would need to call methods on the view to 
update it.  So this isn't about whether your model stays memory-resident while 
the GUI is up, but rather, whether the GUI itself stays memory-resident while 
the model is alive!  Obviously you wouldn't want your GUI or your logger to 
disappear while the model is still active.

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Is there pb documentation somewhere?

[Glyph]

On Aug 18, 2014, at 8:31 PM, Daniel Sank  wrote:

> Dustin,
> 
> > Adding what amounts to a use-after-free bug to the protocol seems like a 
> > really bad idea to me.
> 
> Oh goody, a sarcastic comment which doesn't actually bother to explain the 
> bug :)

Oddly this is _exactly_ how a co-worker characterized the problem as well, with 
(as far as I can see) no communication of this idea ;).

 describes the disastrous 
consequences of this in C; the Python equivalent is mild, but it's still 
annoying to get 'None has no attribute "frob"' when you do self.frobber.frob().

> Since what I describe is basically WeakReferenceable, it's not obvious to me 
> that there's a bug. I tell you when the thing to which your 
> Remote(Weak)Reference points is destroyed, just like weak references invoke 
> finalization callbacks. If you try to invoke remote methods after than 
> happens I just return you an error. What's the problem? This is exactly how 
> weak references work.

The thing about weak references is that they are almost always to objects that 
are shared, that have a well-defined lifecycle.  Your PB objects generally 
should _not_ be shared; a well-structured PB application will almost always 
create thin wrapper objects, factoring the remote-access-control logic out of 
the core model abstractions.  Exposing weak references across a network is also 
very, very tricky: distributed reference counting is hard enough without trying 
to introduce real distributed GC that involves reference tracing.

> > If your server-side app is sensitive to when objects are destroyed for any 
> > reason aside
> > from management of its memory consumption, I'd argue your app is broken.
> 
> That's a really good point. Only CPython destroys objects deterministically 
> when the ref count hits zero.

Yes. And even the CPython developers have said it's effectively a bug to depend 
on this behavior, because you can't really know when it's going to happen.  A 
debugger might be holding on to your stack frames for a little while.  The 
profiler might be keeping your locals around for a moment longer than you 
expect.  This is why we have idioms like 'with f as open(...)' now.

> > Certainly it opens you to a denial of service from a malicious client, 
> > which might hold references
> > to objects you'd rather it didn't
> 
> So my instinct to keep strong references only where they're needed is just 
> bad?

Here are a couple of ways to solve this problem without getting weak references 
involved:

Set limits on things.  PB isn't great about this, but this is an area where it 
could get better, and where all the fixes are really straightforward (find the 
place where PB does a thing, set a limit, raise an exception if the thing is 
over that limit). In this case, don't allow clients to hold unlimited numbers 
of simultaneous references.  Start throwing errors when too many live 
references exist on one connection.  A reasonable application should not need 
that many at once - if you set a limit at around 1024 and allow servers to tune 
it for particular workloads, it should be fine.  (Set it per-type maybe?)
Don't give out references to objects you can't revoke logically, at an 
application level.  If you have a chess piece that has been taken, that is not 
a NULL pointer or None.  There is a small, fixed number of chess pieces per 
game, so you don't have to worry about denials of service.  Therefore your 
ChessPiece class should have a 'taken' state associated with it; in fact, you 
could remember which move of the game the piece was taken on, and produce an 
error message which specifically reminds the player when it was taken.  Much 
like 'with f as open(...)' explicitly invokes 'f.close()' at the end of the 
block but does not force 'f' to get garbage collected, you should still be able 
to have a PB protocol-level reference to an application-level revoked object.  
Debugging distributed systems is hard enough without translating every 
revoked-permission error into some common "the distributed GC happened, I don't 
know what happened to your object, life is hard".

Is this making sense?

> Should a GUI or a logger keep a strong reference to the things they observe?


As my other message indicated - yes :).

-glyph___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Is there pb documentation somewhere?

[exarkun]

On 03:31 am, sank.dan...@gmail.com wrote:

Dustin,
Adding what amounts to a use-after-free bug to the protocol seems like 
a

really bad idea to me.

Oh goody, a sarcastic comment which doesn't actually bother to explain 
the

bug :)


Sarcastic?  What?  Dustin *was* explaining the problem.  Without 
sarcasm, so far as I can tell.


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Experimenting with tubes

[Jan Pobrislo]

On Thu, 14 Aug 2014 14:29:56 -0700
Glyph Lefkowitz  wrote:

> On Aug 11, 2014, at 6:51 AM, c...@webprojekty.cz wrote:
> 
> > Hello, I've been playing with the new tubes that are being
> > implemented:
> > http://comments.gmane.org/gmane.comp.python.twisted/27248
> > https://twistedmatrix.com/trac/ticket/1956
> 
> Thanks so much for taking the time to play with it, and taking some
> time to write feedback.
> 
> > Here are few things that I did with it. I won't publish the full
> > code now, as in it's current shape it could implode eyeballs of
> > twisted devs and possibly make them summon some of the elder gods,
> > but I'll see if I can produce something less vile as I merge the
> > ongoing changes to the tubes branch.
> 
> I'd be interested to see the code nevertheless.  If you had to do
> eyeball-imploding antics to get Tubes to work well for your use-case,
> being able to have a look at that would help us evaluate whether
> those antics were required by the code, encouraged by misfeatures of
> the API design, or just issues with lack of documentation.

It's mostly me not really documenting anything, not writing tests and
littering it with debug statements (which will go away as soon as I
find time to improve my debugging module so it can monkeypatch them).

http://wpr.cz/ccx/bzr/tubes7/ for my changes only
http://wpr.cz/ccx/bzr/tubes7-merge-2/ for changes on top of the bzr
mirror of the svn branch

example usage:
http://wpr.cz/ccx/paste/2014-08-19/2/
http://wpr.cz/ccx/paste/2014-08-19/3/
http://wpr.cz/ccx/paste/2014-08-19/4/
http://wpr.cz/ccx/paste/2014-08-19/5/


> > So far I wrote relatively simple app that read logfiles, parse them
> > and insert what they got out of them into a database.
> 
> If it's actually reading a file, another nice to-do would be an
> IFount provider that provides the contents of a file with appropriate
> flow control, and maybe a thread or process in the background to do
> the file I/O.  Another thing you could contribute to the branch,
> possibly?  :-)  How did you implement this?

At the moment I don't mind the blockingness of the calls. I did write a
ThreadReader and ThreadWriter though for my earlier tubes-alike with
Queue-based loop.

What is more interesting challenge (and we discussed this earlier on
irc) would be generic async file api. I suggested implementing 9p2000
back then and I still think it is a good starting point... but nothing
I have spare time for at the moment.


> I'm not sure I totally understand the case that you're describing
> right now.  Can you perhaps contribute a unit test which demonstrates
> why this line of code is necessary?

I'd love to, alas I'll be bit preocuppied with some more urgent matters
for following week or two. The short version is "flowStopped just
didn't get passed through the series otherwise".


> Are you running into ?

Most probably, as far as I can tell from the vague description.


> That ... definitely sounds kind of gross.  As does actually setting
> the nextFount attribute directly on the fan.Out.

Indeed. The point of the experiment was not produce nice code but to
see if there are any major pitfalls using the tubes API.


> twisted.web.client.Agent has a solution to this where there's a
> multi-failure object that aggregates multiple errors into one thing.
> I think we have to do something similar.  Unfortunately this is a
> very confusing interface in addition to being poorly documented and
> relies on private classes that expose ostensibly public attributes.
> We need to very carefully document this within fan.In.

Some nice abstraction of multiple failures would be indeed handy. I'm
pretty sure DeferredList could use one too.

 
> > As for data representation that I choose to pass between each tube
> > I've started with simple namedtuples and following that I've built
> > a simple "datatype" class somewhat reminiscent of
> > https://github.com/hynek/characteristic
> > which I learned of few moments after I finished polishing my own
> > implementation. What I have there is added layer above namedtuples
> > that autogenerate zope Interfaces (so I can have adaptation), do
> > field type and value validation/adaptation and possibly (as a
> > future extension) provide easy way to make them into AMP commands
> > so the series can be split into communicating processes as needed.
> > (What would be interesting imo is something like ampoule for tubes,
> > or perhaps a ThreadTube and SubprocessTube for performing blocking
> > operations)
> 
> I think it's likely we'll acquire a dependency on Characteristic
> sometime soon, I have promised to look at the issues on
>  and try to address
> them already :).

What makes me ponder is how to work with multiple types of messages
being passed through. Traditionally in twisted one would use different
methods for handling each one, eg. IRCClient has userJoined, userLeft,
and so on. If we keep 

Re: [Twisted-Python] Is there pb documentation somewhere?

[Glyph]

On Aug 19, 2014, at 5:05 AM, exar...@twistedmatrix.com wrote:

> On 03:31 am, sank.dan...@gmail.com wrote:
>> Dustin,
>>> Adding what amounts to a use-after-free bug to the protocol seems like a
>> really bad idea to me.
>> 
>> Oh goody, a sarcastic comment which doesn't actually bother to explain the
>> bug :)
> 
> Sarcastic?  What?  Dustin *was* explaining the problem.  Without sarcasm, so 
> far as I can tell.

My response may have come across as a little ambiguous, so let me also say - I 
think use-after-free is a pretty accurate description of the issue as well, I 
don't believe this was intended sarcastically.

-glyph


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python