Re: [Twisted-Python] web.client.readbody

[Glyph]

> On Apr 3, 2016, at 7:06 AM, John Aherne  wrote:
> 
> I was obviously half-asleep when I posted the previous email.
> 
> Doing more tests, it would seem that only googlemaps is giving me this 
> problem.
> 

Is the request in question one you expect no data from?
-g


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Running trial on windows

[Glyph]

> On Apr 3, 2016, at 6:45 AM, John Aherne  wrote:
> 
> When I run trial in a virtualenv it seems to miss some imports and skips some 
> tests.
> 
> It would seem to be mainly wincertstore that it can't import.
> 
> To make it work I have to include wincertstore in my main python setup and 
> then it starts to test the wincertstore tests.
> 
> Anyone else seen this happening.
> 
> This is Windows 10 and python2.7.11 32 bit, virtualenvwrapper-win 1.2.1
> 

How exactly are you running `trial´ ?  I believe that, since we don't use 
console-scripts entry points, we don't get .exe files generated on Windows, and 
so the wrong thing may be happening when you type 'trial'.

IIRC the way to ask cmd.exe this is 'where trial'.

-glyph
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] SSL wincertstore Ticket 6371

[Glyph]

> On Apr 3, 2016, at 6:29 AM, John Aherne  wrote:
> 
> I have picked up on this again.
> 
> There would appear to be a failing test in test_agent.py.

You said 'test_agent' but the error below is coming from test_tap; which are 
you referring to?

> I have added a new test to check that certificateOptions.trustRoot is an 
> instance of  OpensSSLWindowsCertificateAuthorities if wincertstore is 
> available
> 
> And I have modified the existing test  
> test_setsTrustRootOnContextDefaultTrustRoot to be skipped if wincertstore is 
> available.
> 
> I have run some simple tests using Agent against googlemaps to confirm the 
> host verification works and fails and a similar test with treq. So looks good.
> 
> But before I send up the patch I need to be happy that trial twisted 
> completes with out error. And I keep getting an error on one test but I can't 
> work out what the problem is. Hopefully someone can fathom it from the 
> traceback and point me in the right direction.
> 
> Thanks
> 
> [ERROR]
> Traceback (most recent call last):
> Failure: twisted.trial.util.DirtyReactorAggregateError: Reactor was unclean.
> DelayedCalls: (set twisted.internet.base.DelayedCall.debug = True to debug)
>  ThreadedResolver._cleanup('d.root-servers.net ', 
> )>
>  ThreadedResolver._cleanup('e.root-servers.net ', 
> )>
>  ThreadedResolver._cleanup('f.root-servers.net ', 
> )>
>  ThreadedResolver._cleanup('h.root-servers.net ', 
> )>
>  ThreadedResolver._cleanup('l.root-servers.net ', 
> )>
>  ThreadedResolver._cleanup('a.root-servers.net ', 
> )>
>  ThreadedResolver._cleanup('b.root-servers.net ', 
> )>
>  ThreadedResolver._cleanup('c.root-servers.net ', 
> )>
>  ThreadedResolver._cleanup('g.root-servers.net ', 
> )>
>  ThreadedResolver._cleanup('i.root-servers.net ', 
> )>
>  ThreadedResolver._cleanup('j.root-servers.net ', 
> )>
>  ThreadedResolver._cleanup('k.root-servers.net ', 
> )>
>  ThreadedResolver._cleanup('m.root-servers.net ', 
> )>
> 
> twisted.names.test.test_tap.OptionsTests.test_recursiveConfiguration
> ---
> Ran 11184 tests in 84.952s
> 
> FAILED (skips=2946, errors=1, successes=8237)
> 

I am not sure what the issue is here, but does this test pass or fail for you 
on a clean trunk@HEAD without your changes?

It might be best to just submit your patch for review, even if it's failing, 
and the reviewer might be able to shed some light.  They'll also be able to run 
the tests on our build farm, which might shed some light on whether the problem 
is related to your patch or just related to your system.

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] web.client.readBody

[Cory Benfield]

> On 3 Apr 2016, at 14:35, John Aherne  wrote:
> 
> I have been using deliverBody to process in responses but decided that 
> readBody might be a better fit.
> 
> So far it works with http, but as soon as I switch to https it fails.
> 
> I get a response http code 200 OK but no data.
> 
> I've puzzled over this but can't see what the problem is.
> 
> Has anyone else seen this problem and found out what the solution is.
> 
> Thanks

The problem is an interaction in readBody. Your logs actually contain the key 
clue:

> LENGTH twisted.web.iweb.UNKNOWN_LENGTH

This indicates that the remote server (which, while your logs don’t outright 
say it, is clearly a Google server) is doing something particularly stupid: 
that is, they’re sending a response that is neither chunked nor 
content-length-delimited. This means that message completion can only be 
signalled by the closing of the connection once the response is complete.

Twisted, correctly, gets a bit nervous about this: it’s very difficult to tell 
the actual completion of the response from any number of error conditions where 
the connection gets abruptly torn down. For this reason, the docstring of 
IResponse.deliverBody says that in a case like this: "The protocol's 
connectionLost method will be called with: PotentialDataLoss, which indicates 
that it cannot be determined if the entire response body has been delivered.”

When readBody’s protocol connectionLost method is called with 
PotentialDataLoss, it calls the errback with PartialDownloadError, which is 
what you’re seeing. The body of the response is available on the 
PartialDownloadError as PartialDownloadError.response, so if you’re interested 
in continuing to use readBody you’ll probably want to register an errback on 
the deferred that checks for this error and handles it appropriately: in your 
case, probably by converting the error to a safe response and then calling the 
callback!

In this instance, I’d *also* recommend that you reach out to whatever Google 
service is sending this response. RFC 7230 says that a server SHOULD send a 
Content-Length header if not sending a Transfer-Encoding: chunked header, and 
points out that not doing so exists primarily for backward compatibility with 
HTTP/1.0.

I hope that all helps!

Cory



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] web.client.readbody

[John Aherne]

To Glyph

This works fine if I use response.deliverBody

I get a json response back.

I'll look at what Cory says and see where I get.

Thanks

John

-- 
*John Aherne*




*www.rocs.co.uk *
020 7223 7567
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] web.client.readBody

[John Aherne]

Cory

Thanks for the detailed reply.

I'll see what I can make of it.

I must admit I don't see Google paying much attention to this.

It does work if I use response.deliverBody, but I thought getting the
complete response in one hit would be better.

John

-- 
*John Aherne*




*www.rocs.co.uk *
020 7223 7567
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] running trial on windows

[John Aherne]

I have to specify a really long path to pick up the copy of trial.py for
the virtualenv I am working in.

I specify the virtualenv python I want to run with and then the path to
trial.py then the test I want to run.

It would appear to be only the wincertstore module that is missed.

Perhaps I'll send a report off to virtualenvwrapper about this after some
more messing around.



*John Aherne*




*www.rocs.co.uk *
020 7223 7567
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] SSl wincertstore Ticket 6371

[John Aherne]

The failure is in test_agent

 test_setsTrustRootOnContextDefaultTrustRoot

This fails if wincertstore is available with this failure:-

twisted.trial.unittest.FailTest:
 is not an instance of 


This is because the patch from aaron which I have applied
has_sslverify.platformTrust check for wincertstore and does not return
OpenSSLDefaultPaths but returns
_sslverify.OpenSSLWindowsCertificateAuthorities

The solution I followed was to  add a test to specifically check for a
return of OpenSSLWindowscertificateAuthorities if wincertstore was
available otherwise to skip it.

The other part was to skip the test for
test_setsTrustRootOnContextDefaultTrustRoot
if wincertstore was available

The error message comes from the tests for the root name servers.

I just need to fix that so I have a clean test run before making any
changes.

I just can't see what the problem is with the error

Thanks




-- 
*John Aherne*




*www.rocs.co.uk *
020 7223 7567
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] SSl wincertstore Ticket 6371

[John Aherne]

I have done a clean download of twisted and run the tests without any
changes.

I still get the same error on:--

twisted.names.test.test_tap.OptionsTests.test_recursiveConfiguration

[ERROR]
Traceback (most recent call last):
Failure: twisted.trial.util.DirtyReactorAggregateError: Reactor was unclean.
DelayedCalls: (set twisted.internet.base.DelayedCall.debug = True to debug)
)>
)>
)>
)>
)>
)>
)>
)>
)>
)>
)>
)>
)>

twisted.names.test.test_tap.OptionsTests.test_recursiveConfiguration
---
Ran 11183 tests in 109.669s

FAILED (skips=2945, errors=1, successes=8237)


-- 
*John Aherne*




*www.rocs.co.uk *
020 7223 7567
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] web.client.readBody

[Cory Benfield]

> On 4 Apr 2016, at 09:32, John Aherne  wrote:
> 
> Cory
> 
> Thanks for the detailed reply.
> 
> I'll see what I can make of it.
> 
> I must admit I don't see Google paying much attention to this.
> 
> It does work if I use response.deliverBody, but I thought getting the 
> complete response in one hit would be better.
> 
> John

Yeah, so the difference is that with response.deliverBody you’re not checking 
the code in connectionLost. If you were, you’d be seeing PotentialDataLoss as 
well. ;)

There’s nothing wrong with using readBody, you just need to adjust your 
errback/callback chain to transform the response appropriately in this case.

And yeah, I doubt Google will listen. =P

Cory



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Twisted 16.1 Release Announcement

[Amber "Hawkie" Brown]

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release 
of Twisted 16.1!

This release is hot off the heels of 16.0 released last month, including some 
nice little tidbits. The highlights include:

- twisted.application.internet.ClientService, a service that maintains a 
persistent outgoing endpoint-based connection -- a replacement for 
ReconnectingClientFactory that uses modern APIs;
- A large (77% on one benchmark) performance improvement when using 
twisted.web's client on PyPy;
- A few conch modules have been ported to Python 3, in preparation for further 
porting of the SSH functionality;
- Full support for OpenSSL 1.0.2f and above;
- t.web.http.Request.addCookie now accepts Unicode and bytes keys/values;
- `twistd manhole` no longer uses a hard-coded SSH host key, and will generate 
one for you on the fly (this adds a 'appdirs' PyPI dependency, installing with 
[conch] will add it automatically);
- Over eighteen tickets overall closed since 16.0.

For more information, check the NEWS file (link provided below).

You can find the downloads at  (or 
alternatively ). The NEWS file is 
also available at .

Many thanks to everyone who had a part in this release - the supporters of the 
Twisted Software Foundation, the developers who contributed code as well as 
documentation, and all the people building great things with Twisted!

Twisted Regards,
Amber Brown (HawkOwl)


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] SSl wincertstore Ticket 6371

[Glyph]

> On Apr 4, 2016, at 2:56 AM, John Aherne  wrote:
> 
> I have done a clean download of twisted and run the tests without any changes.

OK.  This is a bug we should investigate, but I think we can safely say it's a 
separate bug and not one which will cause your patch to fail review.

-glyph___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] running trial on windows

[Glyph]

> On Apr 4, 2016, at 1:39 AM, John Aherne  wrote:
> 
> I have to specify a really long path to pick up the copy of trial.py for the 
> virtualenv I am working in.
> 
> I specify the virtualenv python I want to run with and then the path to 
> trial.py then the test I want to run.
> 
> It would appear to be only the wincertstore module that is missed.
> 
> Perhaps I'll send a report off to virtualenvwrapper about this after some 
> more messing around.
> 

How'd you install the wincertstore module?

Does running \path\to\venv\python.exe -c 'import wincertstore' work?

-g


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python