Re: [Twisted-Python] T9217 / PR1052: Wheels, wheels, and more wheels

[Glyph]

> On Aug 29, 2018, at 10:13 AM, Kyle Altendorf  > wrote:
> 
> Hi All,

Hi Kyle!

> I am working on ticket #9217 / PR #1051 to add lots more wheel generation to 
> the Twisted CI.  I decided to give the cibuildwheel package a try and it made 
> this process almost too easy (well... sort of :] ).  I've got 
> AppVeyor-Windows, Travis-Linux, and Circle-OSX all building a variety of 
> wheels for the supported Python versions and bit depths.  Travis doesn't save 
> artifacts 'easily' so I went ahead and doubled up on Linux on Circle for now, 
> though it's having some Docker issues at the moment and hasn't been 
> successful yet.  For some reason in this one case the project directory isn't 
> getting mounted into the container as expected.

Thanks for doing this!

> wheel links:
> https://github.com/twisted/twisted/pull/1051#issuecomment-416743261 
>  (and 
> next comment)
> 
> Now that I've got the wheel builds happening I figured it'd be good to try 
> them out on 'real' machines.  Turns out we get a failure on 
> twisted.cred.test.test_strcred.SSHCheckerTests.test_isChecker for at least 
> the two checks I've done so far (Windows and OSX).  I haven't done more than 
> a cursory look at that yet, but it's on the list to understand and resolve.  
> More testing would of course be welcome.  Real world, just trial Twisted's 
> own tests, whatever would be appreciated if you are interested.
> 
> failures with wheels:
> https://github.com/twisted/twisted/pull/1051#issuecomment-416977723 
> 

It's not clear to me that we have any test environments that accurately test 
what happens after you install from PyPI.  If we could build these wheels and 
then actually run tests on installing them that would be a big step forward in 
test fidelity!  But given the current state of the art, I'd be happy if they 
just got built from the same code that passed our tests in existing 
configurations.

> Overall, it's a bit unclear what the intended use of the various CI hosts are 
> for Twisted.

Ideally, Travis would handle it all.

But Travis can't run multiple kernels, so we have buildbots for lots of 
alternate kernels, just to make sure.  Including BSD kernels, since those are 
actually substantially different where sockets are concerned.

And Travis macOS support is so slow that it was a serious blocker for a while, 
so the buildbot is filling in until Circle can take over.

And Travis can't do Windows, so that's where appveyor comes in.

There might still be various non-special-kernel testing things that buildbot 
does, but those are probably holdovers that, if they still have a reason to 
exist, it's exactly because Travis doesn't have artifact hosting.

Hopefully that's comprehensive?

> I hear that Travis OSX builds were really slow, but from what I can see 
> Circle isn't doing any OSX (other than what I added).

The issue for addressing this is https://twistedmatrix.com/trac/ticket/9445 
 if you'd like to pick up where Adi 
left off.  It's probably just resolving some conflicts, resubmitting for 
review, and bugging people to get it through the process :).

> There wasn't any artifact storage being used on Circle either.  So, I'm not 
> sure if there was a reason to use Circle that went away, or...  But, not 
> having to hook Travis up to S3 or somesuch for storage is quite nice so 
> Circle wins at least in that category.

I think the big problem on Circle is capacity limits, but I don't know if 
that's just a macOS thing, or what.

> Now that I've got something rough in place, are there any opinions about how 
> this should work?  I don't know the present release workflow so I don't know 
> if we'd want an automatic push to PyPI on tags (probably not)

I definitely want this.  This is really the whole point, what we're trying to 
get to :).  Why don't you think we want this?

> or just artifacts on the build server to grab manually (would need some S3 or 
> such for Travis, or Circle for Linux builds as well).  Anything else?  Do we 
> want automated tests against the wheels?  cibuildwheel does have a feature 
> for that though I haven't done anything with it yet.

Ideally we'd push the wheels on tag once the tests passed, but I'd be OK with 
just unconditionally just building / pushing on tag for starters.

> Anyways...  hello, thanks for Twisted, and I hope this work ends up saving 
> some people some time.

I can't wait for fully automated releases :).___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Keeping logger from snatching stdout?

[Glyph]

On Aug 29, 2018, at 3:48 PM, Jeff Grimmett mailto:grimmto...@gmail.com>> wrote:

> I'm working on a Flask project using the flask_twisted package from PyPI and 
> have encountered a mystery.  I don't *have* to solve it to move on, but darn 
> it, I can't let it go :)
> 
> https://pypi.org/project/Flask-Twisted/ 
> 
> https://github.com/cravler/flask-twisted/ 
> 
> 
> So here's the breakdown: when use the standard logging module to output to a 
> logfile + stdout, everything is fine.  I do see some logging output from 
> Twisted as well (the web server part) but for the most part life is groovy.
> 
> I'm going to be integrating in a wxPython windows app to this (previously 
> it's been a shell app) and step 1 was to make my own stdout handler that 
> eventually would pipe all that stuff to a window in the wx app.  The first 
> step of THAT (step 1.a) was to replace all the logging stuff with print() 
> placeholders.

Why would you do this rather than simply handle logging events? Both twisted 
and the stdlib have mechanisms for adding observers that will give you access 
to more structured information - particularly useful if you are trying to 
display something in a GUI.

> At that point, things got confusing, as now ALL of my print()s are being 
> handled by Twisted's logging handler.

This (sort of) makes sense...

> I brought flask-twisted in local to my code so I could monkey around with it. 
> First thing I noticed was that it was using twisted.python.log. I messed 
> around with that - commented it out, and now I get no stdout output at all.  
> Set the parameter setStdout to False, same thing.

So, this is the old, legacy API, and what you want to use is this: 
https://twistedmatrix.com/documents/current/api/twisted.logger.LogBeginner.beginLoggingTo.html
 


But it doesn’t make sense to me that you would see no output at all, or that 
setStdout=False would swallow your print statements.

> The adapter code uses twisted.internet.reactor,  threads, 
> twisted.web.server.Site, twisted.web.wsgi.WSGIResource, and 
> twisted.web.resource.Resource, any one which might be responsible. I spent 
> yesterday evening digging around but haven't found anything yet.

The only thing starting logging here appears to be flask-twisted; we only start 
logging in twist & twistd, within twisted itself.

> Any guidance / ideas?  

I'm very curious what is going on here, and whether it's a logging system bug.  
Could you whittle this down to a minimal reproducer?

-glyph___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] T9217 / PR1052: Wheels, wheels, and more wheels

[Kyle Altendorf]

On 2018-09-05 03:25, Glyph wrote:

On Aug 29, 2018, at 10:13 AM, Kyle Altendorf  wrote:
wheel links:
https://github.com/twisted/twisted/pull/1051#issuecomment-416743261 
(and next comment)


Now that I've got the wheel builds happening I figured it'd be good to 
try them out on 'real' machines.  Turns out we get a failure on 
twisted.cred.test.test_strcred.SSHCheckerTests.test_isChecker for at 
least the two checks I've done so far (Windows and OSX).  I haven't 
done more than a cursory look at that yet, but it's on the list to 
understand and resolve.  More testing would of course be welcome.  
Real world, just trial Twisted's own tests, whatever would be 
appreciated if you are interested.


failures with wheels:
https://github.com/twisted/twisted/pull/1051#issuecomment-416977723


It's not clear to me that we have any test environments that accurately 
test what happens after you install from PyPI.  If we could build these 
wheels and then actually run tests on installing them that would be a 
big step forward in test fidelity!  But given the current state of the 
art, I'd be happy if they just got built from the same code that passed 
our tests in existing configurations.


cibuildwheel does have an option to provide a test command.  If we did 
that would we replace the existing tests or supplement?  If 
supplementing that will presumably be a significant increase in build 
times.  Especially if we did it for all wheels.


So, how much of a build time hit do we want to take?

I hear that Travis OSX builds were really slow, but from what I can 
see Circle isn't doing any OSX (other than what I added).


The issue for addressing this is 
https://twistedmatrix.com/trac/ticket/9445 if you'd like to pick up 
where Adi left off.  It's probably just resolving some conflicts, 
resubmitting for review, and bugging people to get it through the 
process :).


https://github.com/twisted/twisted/pull/1056/files

It looks like the builds I added in my wheel PR, for whatever that is 
worth.  Well, other than picking xcode v9 vs me picking v10 which 
neither of the PRs use anyways.  I added the requires over in the wheel 
PR as well in case it ends up saving a useful amount of our OS X quota.


Now that I've got something rough in place, are there any opinions 
about how this should work?  I don't know the present release workflow 
so I don't know if we'd want an automatic push to PyPI on tags 
(probably not)


I definitely want this.  This is really the whole point, what we're 
trying to get to :).  Why don't you think we want this?


Uhm...  hmm...  *shrug*  I figured someone might want to do a final spot 
check or have manual intervention for a real release.  You can't replace 
a file on PyPI so...  but sure, I didn't mean to argue against 
automation.


Here's one spot I did automatic deployment off of Travis to PyPI.

https://github.com/altendky/gitignoreio/blob/aeeb9546b6f27f411b191699d3625b4d0a08e7cf/.travis.yml#L11-L19

Pretty straightforward though I think it does the wheel build which 
isn't what we'd really want.  I'd have to check on how to deploy 
existing wheels as well as those from OS X and Windows.  I'd suggest a 
separate PR?  I mean maybe it'd be easier to get one PR reviewed than 
two but that doesn't seem like a great reason to mash things together.


Cheers,
-kyle

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Issue with TLS failure modes

[Robert DiFalco]

Is the timeout the only way to handle this kind of TLS handshake error?

Sent from my iPhone

> On Aug 30, 2018, at 10:14 AM, Robert DiFalco  wrote:
> 
> I'm having an issue properly handling TLS failure modes. For example consider 
> the EchoServer and EchoClient code. If I use a TLS client with a TCP4 server, 
> I do not get a handshake exception until I abort the connection. But I don't 
> want to abort the connection unless I get a handshake error. 
> 
> What I'd like to do is to check the handshake status in my protocol before my 
> client sends bytes to the server. I'd like my send message to be able to 
> raise the : [('SSL routines', 'ssl23_read', 'ssl 
> handshake failure'). But for some reason it seems to get lost until I abort 
> the connection. Does this sound familiar to anyone?
> 
> What I've done for now is setup a Timeout mixin so that after my client.send, 
> if I do not get an ACK back (which my particular protocol does) within two 
> minutes, I just abort the connection. This then calls connectionLost with the 
> correct SSL.Error. But if it's in the error queue (and found during the 
> course of abortConnection then isn't there a way to find it sooner? Like 
> before my timeout and before I call send on the client protocol?
> 
> Thanks!

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Issue with TLS failure modes

[Jean-Paul Calderone]

On Wed, Sep 5, 2018 at 9:40 AM Robert DiFalco 
wrote:

> Is the timeout the only way to handle this kind of TLS handshake error?
>

It might be easier to answer your original question if you include a Short,
Self Contained, Correct (Compilable), Example .

Jean-Paul


> Sent from my iPhone
>
> > On Aug 30, 2018, at 10:14 AM, Robert DiFalco 
> wrote:
> >
> > I'm having an issue properly handling TLS failure modes. For example
> consider the EchoServer and EchoClient code. If I use a TLS client with a
> TCP4 server, I do not get a handshake exception until I abort the
> connection. But I don't want to abort the connection unless I get a
> handshake error.
> >
> > What I'd like to do is to check the handshake status in my protocol
> before my client sends bytes to the server. I'd like my send message to be
> able to raise the : [('SSL routines',
> 'ssl23_read', 'ssl handshake failure'). But for some reason it seems to get
> lost until I abort the connection. Does this sound familiar to anyone?
> >
> > What I've done for now is setup a Timeout mixin so that after my
> client.send, if I do not get an ACK back (which my particular protocol
> does) within two minutes, I just abort the connection. This then calls
> connectionLost with the correct SSL.Error. But if it's in the error queue
> (and found during the course of abortConnection then isn't there a way to
> find it sooner? Like before my timeout and before I call send on the client
> protocol?
> >
> > Thanks!
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python