Re: [Twisted-Python] Notes from Twisted BoF at PyCon 2013

[exarkun]

Thanks for taking those notes, Itamar.

I copied the summary onto the wiki and fleshed out the ideas a bit:

https://twistedmatrix.com/trac/wiki/Fellowship2013/Priorities

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Testing Twisted code without trial

[exarkun]

On 18 Mar, 07:32 pm, a...@roiban.ro wrote:
>On 22 January 2013 22:03,   wrote:
>>On 09:29 am, a...@roiban.ro wrote:
>>>On 22 January 2013 02:21,   wrote:
On 20 Jan, 02:35 pm, a...@roiban.ro wrote:
>
>I agree that this is a ugly hack and I removed the project.
>>
>>Hi Adi,
>>
>>trial does what it does by touching a lot of internal stuff as well.
>>This is still bad, but at least it's our fault if it ever breaks 
>>instead
>>of yours.  There's also a long term plan (or "plan" may be putting it
>>too strongly, perhaps I should say "hope") that this part of trial 
>>will
>>change to only use public interfaces.  This will probably require
>>reactors actually implementing restartability, or it will require
>>changing the trial feature slightly (eg, so it starts a reactor, runs
>>all tests, then stops the reactor - if it did this, I'm sure you can
>>imagine how "waiting" for a Deferred would just be adding a callback 
>>to
>>the right place, as in any other Twisted-based application).
>>
>>Are you interested in helping out with making reactors restartable? :)
>
>Sorry for the late reply.
>
>I am still clumsy when working with Twisted so I don't know if I can
>help to much here.
>
>I don't know what is expected from a restartable reactor.
>
>The way I am testing deferreds is by starting the reactor, allow for
>the deferred to execute and then stop the reactor.
>
>I don't want to pause it and then continue the execution from where it
>was stopped.
>
>To help with debugging I am also printing a snapshot of reactor state
>at a certain time.
>
>
>
>I prefer the Arrange/Act/Assert way of writing test:
>
>
>checker = mk.credentialsChecker()
>credentials = mk.credentials()
>
>deferred = checker.requestAvatarId(credentials)
>failure = self.getDeferredFailure(deferred)
>
>self.assertFailureType(AuthentiationError, failure)
>self.assertEqual(credentials.username, failure.value.username)
>
>
>I found it easier to read than this version:
>
>
>checker = mk.credentialsChecker()
>credentials = mk.credentials()
>
>def check_result(result_or_failure):
>self.assertFailureType(AuthentiationError, failure)
>self.assertEqual(credentials.username, failure.value.username)
>
>deferred = checker.requestAvatarId(credentials)
>deferred.addBoth(check_result)
>
>return deferred
>
>
>
>I have updated the code to use as many public reactor members as 
>possible.
>
>The following private member are still use:
>reactor._startedBefore, reactor._started
>
>It uses the following public methods:
>startRunning(), doIteration(), stop(), iterate()
>
>Here is the main part that blocks the execution until the deferred got 
>a result.
>It executes the deferred in the reactor loop.
>
>https://github.com/chevah/empirical/blob/master/chevah/empirical/testcase.py#L240
>
>--
>
>Maybe this is has only limited usage, but I just wanted to share this 
>work.
>For me, this makes writing test a much nicer experience.

Hi Adi,

This basically looks like an implementation of the old, now-removed 
`TestCase.wait` API.

We got rid of `wait` for several reasons:

  * It was hard to implement.  By the end, it sort of worked with most 
reactors - but not all of them.

  * It is a tool for building non-deterministic, slow tests.  If tests 
are written *not* to do real I/O and *not* to wait for real time to 
pass, then they don't need to let a real reactor spin.

We replaced these ideas:

  * with returning a `Deferred` from a test method (which works even if 
you don't use trial to run your tests - but not if you don't subclass 
trial's `TestCase`).  We eventually moved on from this idea, though many 
parts of Twisted itself are still tested using this feature, to...

  * things like `twisted.test.proto_helpers.MemoryReactor`, 
`twisted.internet.task.Clock`, and most recently 
`TestCase.successResultOf` and `TestCase.failureResultOf` (but don't 
confuse these with your `getDeferredFailure` - they are significantly 
less capable).

I'd encourage you to explore testing strategies that use 
reactor/transport/time fakes and give us feedback about where they're 
not making your job easy enough.  I think ultimately you'll be happier 
with the resulting tests, and you won't have to maintain so much hairy 
reactor manipulation code.

Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] So... Python 3.4 is getting its own async I/O system

[Peter Westlake]

On Wed, Mar 20, 2013, at 9:36, Glyph wrote:
...
> Since async file I/O is not a thing you can implement[1], ...
> 
> [1]: 

Understood that it isn't possible to use native asynchronous I/O. But an
implementation that used threads behind the scenes and returned a
Deferred (for small files) or a Protocol (for big ones) would be a nice
thing to have.

Peter.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Lore and generating reStructuredText (Lore2Sphinx)

[Kevin Horn]

On Sat, Mar 23, 2013 at 9:57 PM, Glyph  wrote:

>
> On Mar 21, 2013, at 7:17 AM, Kevin Horn  wrote:
>
> Notice that the node trees look exactly the same.  Now this is not quite
> true, as there's probably some attributes on the actual Python nodes that
> might be used to distinguish them when writing output which aren't
> displayed here...they certainly get rendered into HTML differently.  But
> the point is that the directive itself is GONE and you have no real way of
> recreating it from the node tree.
>
>
> The directive isn't "gone"; it turns into the attributes on the Python
> nodes that you're talking about.  Presumably that's what's used to render
> it into HTML.  I believe it was Doug Hellmann who indicated to Jean-Paul
> that this was possible.
>
>
While this is true for some built-in docutils directives, there is no
guarantee that this will be the case.  A directive basically says "call a
Python callable according to a certain interface, and put the returned
nodes here."  If the directive in question uses a callable that returns
nodes with attributes set in a certain way, then you have some breadcrumbs
to figure out how those nodes were created, but there's nothing that says
that the nodes will definitely be set up that way.

For example, you could have a directive that has a ReST list as content,
and changes the items in the list into some kind of link or something.
 maybe it looks something like this (not a real/valid nodetree...):


...
...
...

How can you tell that this was created by a directive?  You can't, because
it could just as easily have been a list full of links to begin with.

This is why rstgen has it's own node definitions, as it is focused on what
source constructs should be generated, rather than what the docutils output
should look like.

Of course it's possible that the docutils nodes that we would actually need
from the Twisted docs are all introspectible *enough* that you could maybe
just build a docutils doctree and make good enough guesses to create output
which included directives.  It might even be easy to make those guesses.
 But you'd still be guessing, and would fail in the general case.  Also,
you'd still need to write the code to render those nodes into valid ReST,
which is really the hard part of the process.  Also, you'd need to "parse"
everything inside the "directive node" (or whichever node you've decided
represents the directive) in order to turn it into directive arguments,
options and contents.

Reading exarkun's expansion of the notes Itamar posted [1]_ it looks like
another idea proposed was to generate Sphinx (or maybe Sphinx-looking)
output directly from Lore, which could maybe work, but I think would also
be a lot of work, for a lot less benefit.

However, if what you really want is to have a Lore plugin that generates
RestructuredText, then why not have a lore plugin that generates a rstgen
tree, which rstgen will already know how to render into ReST?  Other than
the obvious objection that rstgen isn't done yet, this seems the best
solution to me.  Of course I may be biased. :)

Even if you didn't want to use rstgen itself, though, I still think you're
better off creating some tree-like structure that is *not* a docutils
document tree, and then have that structure render itself into ReST.

BTW, Doug Hellman almost certainly knows more about the internals of
docutils than I do, so maybe he's right and there is a way to (relatively)
easily generate ReST from a docutils tree including the directives.  But I
don't think it is.


> Perhaps you mean "there's no public API for constructing the node tree
> representation of an arbitrary directive"?
>
>
This is certainly true, but I think it doesn't go far enough in describing
the issue



.. [1] https://twistedmatrix.com/trac/wiki/Fellowship2013/Priorities

--
Kevin Horn
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Twisted dinner pictures

[Hynek Schlawack]

Hi,

As you may have noticed, I'm building a little page to show off that PyCon 
wasn't about donglegate. http://thisispycon.com

I would love to post about our little dinner and would be delighted if anyone 
having pics AND quotes could forward them to me.

Same goes for sprints etc, flood me!

Cheers,

Hynek

Sent from my phone.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] So... Python 3.4 is getting its own async I/O system

[Itamar Turner-Trauring]

On Mon, Mar 25, 2013 at 11:59 AM, Peter Westlake
wrote:

> Understood that it isn't possible to use native asynchronous I/O. But an
> implementation that used threads behind the scenes and returned a
> Deferred (for small files) or a Protocol (for big ones) would be a nice
> thing to have.
>

For streaming files, you want something pausable; there are existing
producers which don't use threads, and there will presumably be a tubes
equivalent. Maybe having the tubes one use threads by default is a good
idea.

For getting a file as a string, you can just do:

d = deferToThread(lambda path: FilePath(path).getContent())

For more complex usage patterns, you'll probably want to write custom code
using deferToThread/deferToThreadPool anyway.

-- 
Itamar Turner-Trauring, Future Foundries LLC
http://futurefoundries.com/ — Twisted consulting, training and support.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] So... Python 3.4 is getting its own async I/O system

[Tobias Oberstein]

> > Will this tackle async file I/O also or only network?
> 
> 
> Since async file I/O is not a thing you can implement[1], presumably it will 
> only
> be networking.
> 

Ok, so AIO is broken on Linux both implementation- and API-wise, but 
Windows/IOCP and/or *BSD/kqueue might be a different story. I haven't tested 
myself though. Probably would give kqueue on FreeBSD/ZFS a run ..

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] So... Python 3.4 is getting its own async I/O system

[Tobias Oberstein]

> > Since async file I/O is not a thing you can implement[1], ...
> >
> > [1]: 
> 
> Understood that it isn't possible to use native asynchronous I/O. But an
> implementation that used threads behind the scenes and returned a Deferred
> (for small files) or a Protocol (for big ones) would be a nice thing to have.

Yeah, reading a single block from disk can take somewhere between roughly 10us 
(FusionIO) to 10ms (magnetic platter), and if I need only single blocks (not 
streaming .. say I access an on-disk key-value store), it would be nice if that 
would be encapsulated in a deferred, while the reactor drives other stuff. 
Ideally without a background worker thread pool on platforms that have 
sufficient/sane support. And worked around via thread pool on others.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] So... Python 3.4 is getting its own async I/O system

[Glyph]

On Mar 25, 2013, at 8:59 AM, Peter Westlake  wrote:

> On Wed, Mar 20, 2013, at 9:36, Glyph wrote:
> ...
>> Since async file I/O is not a thing you can implement[1], ...
>> 
>> [1]: 
> 
> Understood that it isn't possible to use native asynchronous I/O. But an
> implementation that used threads behind the scenes and returned a
> Deferred (for small files) or a Protocol (for big ones) would be a nice
> thing to have.

For what it's worth, I completely agree.  If someone has, or someone ever does, 
implement a nice async file I/O system, a nice abstract API for Twisted to do 
it would allow us to swap in such a thing and get a performance boost without 
anyone changing their code.

As Itamar says though, a Fount would be an ideal way to present this once I 
merge the Tubes branch.  (It'll be up for review any day now, thanks to all the 
help I got at PyCon...)

-glyph___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] So... Python 3.4 is getting its own async I/O system

[exarkun]

On 07:47 pm, ita...@futurefoundries.com wrote:
>On Mon, Mar 25, 2013 at 11:59 AM, Peter Westlake
>wrote:
>>Understood that it isn't possible to use native asynchronous I/O. But 
>>an
>>implementation that used threads behind the scenes and returned a
>>Deferred (for small files) or a Protocol (for big ones) would be a 
>>nice
>>thing to have.
>
>For streaming files, you want something pausable; there are existing
>producers which don't use threads, and there will presumably be a tubes
>equivalent. Maybe having the tubes one use threads by default is a good
>idea.
>
>For getting a file as a string, you can just do:
>
>d = deferToThread(lambda path: FilePath(path).getContent())

Not that `FilePath` is guaranteed to be thread-safe...

Jean-Paul
>For more complex usage patterns, you'll probably want to write custom 
>code
>using deferToThread/deferToThreadPool anyway.
>
>--
>Itamar Turner-Trauring, Future Foundries LLC
>http://futurefoundries.com/ — Twisted consulting, training and support.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Notes from Twisted BoF at PyCon 2013

[Glyph]

On Mar 25, 2013, at 6:38 AM, exar...@twistedmatrix.com wrote:

> Thanks for taking those notes, Itamar.
> 
> I copied the summary onto the wiki and fleshed out the ideas a bit:
> 
> https://twistedmatrix.com/trac/wiki/Fellowship2013/Priorities
> 
> Jean-Paul

Thanks to both of you for getting this published.

I hope that everyone will read it and ask for clarification if anything is not 
obvious to those who could not attend :).

-glyph


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Testing Twisted code without trial

[Glyph]

On Mar 25, 2013, at 6:48 AM, exar...@twistedmatrix.com wrote:

>  * with returning a `Deferred` from a test method (which works even if 
> you don't use trial to run your tests - but not if you don't subclass 
> trial's `TestCase`).  We eventually moved on from this idea, though many 
> parts of Twisted itself are still tested using this feature, to...

I have a minor quibble with saying we have "moved on" from this idea.

Trial is useful both for writing unit tests (which use MemoryReactor, Clock, 
etc) and for writing integration tests (which return Deferreds and do real I/O).

Twisted *itself* is mostly tested with unit tests that test smaller chunks of 
functionality, and there are rarely (perhaps never) good reasons to return a 
Deferred from a test case within Twisted's own test suite, but I maintain lots 
of test code that depends intimately on the return-a-Deferred functionality, 
and other, higher-level projects do too.

We have no plans to remove or deprecate this functionality, but it's important 
to know that you should not use it unless you really need it.

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Lore and generating reStructuredText (Lore2Sphinx)

[Glyph]

On Mar 25, 2013, at 9:16 AM, Kevin Horn  wrote:

> How can you tell that this was created by a directive?  You can't, because it 
> could just as easily have been a list full of links to begin with.

But, I don't care if it was created by a directive or not.

I think we're talking about two different things.

What you seem to be talking about is using Sphinx to do source-to-source 
Lore-to-ReST transformation.  In that case, you're (sort of) right, in that 
information is lost when you invoke directives.  If we did this, and it worked, 
it would just be a slightly better way to implement lore2sphinx; we'd still 
need to manage the transition in largely the same way.

What *I'm* talking about is just using Lore source as an input to Sphinx, and 
going straight to the output HTML.  In order to do this, we just need to 
construct the right tree and actually *invoke* the directive callables at the 
right time.  They produce whatever output they want to produce, and we hand 
that back to Sphinx, and it outputs some docs.  With this strategy, we just 
switch to sphinx by switching our build process; we don't switch input formats. 
 Then, if someone wants to use Lore they can, if they want to use ReST they 
can, and we can migrate on an as-needed basis; there's no need for a single big 
format migration for us to start using Sphinx.

-glyph___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python