Re: [Twisted-Python] Streaming Requests

[Cory Benfield]

> On 8 Jan 2017, at 06:44, Mark Williams  wrote:
> 
> * Comments?
>  Will this approach break a public API?  Does it sound terrible?  Or
>  good?  Please share your thoughts!
> 
> Let's hope 2017 is the year of the streaming request!
> 
> -Mark

I’m very excited to see someone tackling this again: I began taking a swing at 
it last year and abandoned it in favour of some of the other HTTP/2 work. 
However, I think it’s a vital addition and I really look forward to seeing it 
land.

For my part, I should note that HTTPChannel has a twin in HTTP/2 land, which is 
the combination of t.w._http2.H2Connection and t.w._http2.H2Stream. These will 
also want updating. However, once you’ve defined the shape of the patch for 
HTTPChannel I am happy to help out with reviewing that and making corresponding 
changes to the HTTP/2 side of things too.

Cory
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Testing Strategies (was Re: Streaming Requests)

[Glyph Lefkowitz]

On Jan 8, 2017, at 4:34 PM, Jean-Paul Calderone  
wrote:
> 
> Here's one example I know of off the top of my head, 
>   
> >.
>   This one isn't a from-scratch re-implementation of the implicit Request 
> interface but a Request subclass, instead.  However, it still interacts with 
> a lot of important pieces of Request which aren't part of IRequest.

Mark already addressed how he won't be breaking this use-case (which is hugely 
important, and core to the whole idea of a compatibility policy, so that is as 
it should be).

However, this kind of test-mocking is, I think, ultimately done at the wrong 
layer.  It's trying to override some very vaguely-specified internals in the 
middle of an implementation.

Really, twisted.web should provide its own testing tools, of course.  But if 
you're going to implement something that does this sort of overriding, I think 
the idiom to follow would be treq.testing: 
https://github.com/twisted/treq/blob/fcf5deb976c955ca6ef6484f414d25839932940e/src/treq/testing.py
 
,
 rather than any of the various implementations of DummyRequest (including more 
than a few I'm sure I've written).

Internally, treq.testing uses MemoryReactor and the 
(not-technically-public-but-it-really-should-be) iosim module.  Therefore the 
layer it's plugging in at is at the very well-defined interface between IAgent 
and IStreamClientEndpoint, and between HTTPChannel and Transport.  This gives 
the determinism and (most of the) speed of an in-memory implementation, but a 
great deal of the realism of a full-blown integration test.  In particular, it 
binds to the appropriate objects within Twisted and will throw deprecation 
warnings if their interfaces change.  (Also of note; since what it's building a 
fake for is really 99% Agent, the bulk of this could probably move up another 
level into Twisted with very little effort.)

I think following this idiom of pushing I/O through a fake reactor and having a 
back-end that can be controlled via test-specific APIs is probably what we 
should all be striving for, and in Twisted we should be working to make the 
memory-reactor stuff more complete and convenient, rather than adding dummy 
implementations at each layer of the stack.

This is a set of ideas that I've been gradually arriving at over a long period 
of time, but it's probably high time for some public discussion by now, even if 
it's just everybody saying "yeah, that sounds good" :-).

On a related note, 'proto_helpers' is in a super awkward place.  
'twisted.testing', anyone? :)

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Testing Strategies (was Re: Streaming Requests)

[Jean-Paul Calderone]

On Mon, Jan 9, 2017 at 4:52 AM, Glyph Lefkowitz 
wrote:

> On Jan 8, 2017, at 4:34 PM, Jean-Paul Calderone 
> wrote:
>
>
> Here's one example I know of off the top of my head, <
> https://github.com/ScatterHQ/flocker/blob/master/flocker/
> restapi/testtools.py#L316>.  This one isn't a from-scratch
> re-implementation of the implicit Request interface but a Request subclass,
> instead.  However, it still interacts with a lot of important pieces of
> Request which aren't part of IRequest.
>
>
> Mark already addressed how he won't be breaking this use-case (which is
> hugely important, and core to the whole idea of a compatibility policy, so
> that is as it should be).
>
> However, this kind of test-mocking is, I think, ultimately done at the
> wrong layer.  It's trying to override some very vaguely-specified internals
> in the middle of an implementation.
>
>
Absolutely.



> Really, twisted.web should provide its own testing tools, of course.  But
> if you're going to implement something that does this sort of overriding, I
> think the idiom to follow would be treq.testing: https://github.
> com/twisted/treq/blob/fcf5deb976c955ca6ef6484f414d25
> 839932940e/src/treq/testing.py, rather than any of the various
> implementations of DummyRequest (including more than a few I'm sure I've
> written).
>

Though, note, the link I gave above was support code for something very
similar to this treq code:
https://github.com/ScatterHQ/flocker/blob/master/flocker/restapi/testtools.py#L460

To clarify your point a bit (I think):


   - MemoryAgent (from Flocker) provides a testing IAgent by implementing
   an IRequest that does in-memory resource traversal to dispatch requests and
   generate responses.
   - RequestTraversalAgent (from treq) provides a testing IAgent by
   implementing (using) iosim to do in-memory protocol/transport interacts to
   drive an in-memory HTTP conversation that runs all of the regular Twisted
   Web HTTP processing machinery.

RequestTraversalAgent's approach is better because the protocol/transport
interface is better defined.  Because it's better
defined, RequestTraversalAgent doesn't have to touch pieces that we might
want to consider implementation details (whether they're _-prefixed or
not).  It also invokes a larger portion of the real implementation code
making it more likely to be a realistic simulation of real-world use of the
code.

Having spelled this out, what occurs to me now is
that RequestTraversalAgent is really just a step or so up the ladder and
there's further to go.  For example, RequestTraversalAgent only needs to
exist at all because `iosim` has a distinct interface.  This distinct
interface means you need a RequestTraversalAgent-like thing for each
reactor-using thing for which you want to provide a test double.  If this
adaption behavior were bundled up differently then I think we'd get a lot
more in-memory test doubles for free (or closer to it - we'd be another
rung up the ladder, at least).  That seems like it would be a big win given
how *few* of these testing helpers Twisted has historically managed to
provide (suggesting it's just too hard to do so given the current tools).


>
> This is a set of ideas that I've been gradually arriving at over a long
> period of time, but it's probably high time for some public discussion by
> now, even if it's just everybody saying "yeah, that sounds good" :-).
>

So, it pretty much sounds good, though with the above refinements. :)


>
> On a related note, 'proto_helpers' is in a super awkward place.
>  'twisted.testing', anyone? :)
>

Yes.  I almost suggested this about a week ago when I was preparing to
contribute some testing code but then realized I couldn't contribute the
code after all. :(

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Announcing txAWS 0.2.3.1

[Jean-Paul Calderone]

I've just release txAWS 0.2.3.1.  txAWS is a library for interacting with
Amazon Web Services (AWS) using Twisted.

AWSServiceEndpoint's ssl_hostname_verification's parameter now defaults to
True instead of False.  This affects all txAWS APIs which issue requests to
AWS endpoints.  For any application which uses the default
AWSServiceEndpoints, the server's TLS certificate will now be verified.

This resolves a security issue in which txAWS applications were vulnerable
to man-in-the-middle attacks which could either steal sensitive information
or, possibly, alter the AWS operation requested.

The new release is available on PyPI in source and wheel forms.  You can
also find txAWS at its new home on github, .

Special thanks to Least Authority Enterprises
() for
sponsoring the work to find and fix this issue and to publish this new
release.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] understanding twisted, better error handling

[steven meiers]

hi,



i use twisted to send a web request over a proxy, this works over
treq.get(url, agent=myagent) without problems.

but in the process (started out with agent.request) i could not figure
out from what line of twisted code this error:

[Failure instance: Traceback (failure with no frames): :
[]

came from.

btw, is there a way to have a debugger attached to twisted code that
shows what code is currently executed and has a step forward option?


the error does come up when you give agent.request a "GET" instead of a
b"GET".


heres the test code:
from twisted.python.log import err
from twisted.web.client import ProxyAgent
from twisted.internet import reactor, defer
from twisted.internet.endpoints import TCP4ClientEndpoint
import treq

def display(response):
print("Received response")
print(response.content)


def err(failure):
print(failure)

def main():
endpoint = TCP4ClientEndpoint(reactor, "223.25.102.186", 8080)
agent = ProxyAgent(endpoint)
d = agent.request("GET", bytes("http://somedomain.de";, 'utf-8'))
d.addCallbacks(display)
d.addErrback(err)

if __name__ == "__main__":
main()
reactor.run()


this code will produce this error:
[Failure instance: Traceback (failure with no frames): :
[]

and some .py files with line numbers. none of which really are the
problem.



as it turns out, the "GET" as a string gets problematic here:
https://github.com/twisted/twisted/blob/twisted-16.6.0/src/twisted/web/
_newclient.py#L651

requestLines.append(b' '.join([self.method, self.uri,
b'HTTP/1.1\r\n']))


wouldnt it be better to have something like this:


try:
requestLines.append(b' '.join([self.method, self.uri,
b'HTTP/1.1\r\n']))
except TypeError as e:
raise TypeError("could not join: " + str(self.method) + " "
+ str(self.uri) + ' ' + 'HTTP/1.1\r\n' )




which will produce:
 [Failure instance: Traceback:
: could not join: GET b'http://somedomain.de'
HTTP/1.1

to give the user a chance to find out what he did wrong?
...when he uses agent instead of treq for whatever reason.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] understanding twisted, better error handling

[Glyph Lefkowitz]

> On Jan 9, 2017, at 5:20 PM, steven meiers  wrote:
> 
> the error does come up when you give agent.request a "GET" instead of a
> b"GET".

That's expected; you do have to pass the method as `bytes`, as documented here: 
https://twistedmatrix.com/documents/16.6.0/api/twisted.web.iweb.IAgent.html#request
 


-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python