Re: [Python-Dev] Sad status of Python 3.x buildbots
On 4 September 2014 16:59, Shorya Raj wrote: >> The limiting factor is generally time - various aspects of the current >> system are clumsy, but they work, so unless someone is particularly >> keen and willing to work through all the factors that led to the >> current setup and propose changes (including at least a rough concept >> of how ongoing maintenance will work), the status quo wins by default. > > If you could outline some of these, I would be willing to have a look at > what I could do. One that seems promising to me would be to experiment with dynamic testing in OpenStack: http://docs.buildbot.net/latest/manual/cfg-buildslaves.html#openstack Creating test VMs on demand avoids a lot of the security issues otherwise incurred by someone setting up a persistent build slave and provides a more consistent test environment. https://wiki.python.org/moin/BuildBot gives details of the test system configuration details. Due to the overlap with the infrastructure team (to make use of the Rackspace VM resources), [email protected] may be a better list to ask for more details on what OpenStack resources we have available (this idea really sits somewhere between infrastructure and python-dev, as Antoine does most of the BuildBot master maintenance). If we could get something like that running, then we might be able to trim the preconfigured buildbot fleet back to just the non-x86 machines. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sad status of Python 3.x buildbots
Yes there are "details" indeed. But not show stoppers. A prototype can be seen here: http://cclimited.webfactional.com The nice bit is testing will be executed in a total fresh instance, an added bonus On 4 September 2014 05:16:01 GMT+01:00, Nick Coghlan wrote: >On 4 September 2014 11:07, Antonio Cavallo >wrote: >> I wonder if there is any interest in starting to use the opensuse >build >> servers for continuous build and testing on redhat, fedora suse and >(I >> think) debian: that will solve once for all the maintenance issues on >those >> platforms (and provide a reliable build). > >It sounds like a good idea in principle, but I suspect there would be >a lot of devils in those details. That doesn't make it a bad idea, >just something that would need a volunteer to investigate further >before we could form a real opinion. > >Cheers, >Nick. > >-- >Nick Coghlan | [email protected] | Brisbane, Australia -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sad status of Python 3.x buildbots
On 4 September 2014 18:50, A. Cavallo wrote: > Yes there are "details" indeed. But not show stoppers. A prototype can be > seen here: http://cclimited.webfactional.com > The nice bit is testing will be executed in a total fresh instance, an added > bonus Oh, very nice! What is needed to get that up and running? Regards, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Thu, 4 Sep 2014 13:11:38 +1000 Nick Coghlan wrote: > That leaves Python 2.7, and I have to say I'm now persuaded that a > backport (including any required httplib and urllib features) is the > right way to go. One of the tasks I'd been dreading as a follow-on > from PEP 466 was organising the code audit to make sure our existing > Python 2 applications are properly configuring SSL. If we instead > change Python 2.7.9 to validate certificates by default, then the need > to do that audit *goes away*, replaced by the far more mundane tasking > of doing integration testing on 2.7.9, which we'd have to do *anyway*. What are "our existing Python 2 applications"? Is it a Red Hat-specific statement? What is the "code audit" you are talking about? Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 4 September 2014 22:39, Antoine Pitrou wrote: > On Thu, 4 Sep 2014 13:11:38 +1000 > Nick Coghlan wrote: >> That leaves Python 2.7, and I have to say I'm now persuaded that a >> backport (including any required httplib and urllib features) is the >> right way to go. One of the tasks I'd been dreading as a follow-on >> from PEP 466 was organising the code audit to make sure our existing >> Python 2 applications are properly configuring SSL. If we instead >> change Python 2.7.9 to validate certificates by default, then the need >> to do that audit *goes away*, replaced by the far more mundane tasking >> of doing integration testing on 2.7.9, which we'd have to do *anyway*. > > What are "our existing Python 2 applications"? Is it a Red Hat-specific > statement? What is the "code audit" you are talking about? Yes, that's a Red Hat specific statement. In addition to OpenStack, many of our other products include Python components. We mostly rely on pycurl, pyopenssl, python-nss, etc for SSL/TLS for historical reasons (since ssl wasn't available in the Python standard library until RHEL 6), which was why these concerns got picked up via the OpenStack community rather than the Fedora community. However, being aware of the problem creates that element of doubt as to whether there are *other* cases where someone used the stdlib ssl module rather than one of the alternatives that verifies HTTPS by default. I'd previously been looking at the question while wearing my upstream core developer & CPython redistributor hats. Glyph et al finally prompted me to look at it while wearing my "automated quality assurance toolchain architect for an application vendor" hat (i.e. my actual day job), and that's a *radically* different perspective when it comes to a language runtime that fails to verify HTTPS connections correctly by default. That's the core reason I changed my mind: this conversation forced me to think through how I could ensure correct HTTPS handling in all our Python code, and the only workable solution I now see is to fix it at the standard library level. The alternative is to prepare for the likelihood of future CVEs in Python based tools based on failure to properly verify SSL certificates, as I don't see a practical way of automatically detecting "tried to use HTTPS without verifying the SSL certificate properly". Regards, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sad status of Python 3.x buildbots
Also, have you considered using Travis? It would allow us to add to the configurations, at least generally... On Thu, Sep 4, 2014 at 4:16 PM, Nick Coghlan wrote: > On 4 September 2014 11:07, Antonio Cavallo > wrote: > > I wonder if there is any interest in starting to use the opensuse build > > servers for continuous build and testing on redhat, fedora suse and (I > > think) debian: that will solve once for all the maintenance issues on > those > > platforms (and provide a reliable build). > > It sounds like a good idea in principle, but I suspect there would be > a lot of devils in those details. That doesn't make it a bad idea, > just something that would need a volunteer to investigate further > before we could form a real opinion. > > Cheers, > Nick. > > -- > Nick Coghlan | [email protected] | Brisbane, Australia > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rajshorya%40gmail.com > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sad status of Python 3.x buildbots
> > The limiting factor is generally time - various aspects of the current > system are clumsy, but they work, so unless someone is particularly > keen and willing to work through all the factors that led to the > current setup and propose changes (including at least a rough concept > of how ongoing maintenance will work), the status quo wins by default. If you could outline some of these, I would be willing to have a look at what I could do. On Thu, Sep 4, 2014 at 6:06 PM, Nick Coghlan wrote: > On 4 September 2014 15:45, Shorya Raj wrote: > > Also, have you considered using Travis? It would allow us to add to the > > configurations, at least generally... > > Many SaaS solution assume the use of specific centralised identity > providers, which isn't a function we're prepared to outsource (we know > we need to fix our current proliferation of identity silos, but that's > a big project and there are other tasks ahead of it on the todo list > for the existing infrastructure team). > > The limiting factor is generally time - various aspects of the current > system are clumsy, but they work, so unless someone is particularly > keen and willing to work through all the factors that led to the > current setup and propose changes (including at least a rough concept > of how ongoing maintenance will work), the status quo wins by default. > > Cheers, > Nick. > > -- > Nick Coghlan | [email protected] | Brisbane, Australia > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sad status of Python 3.x buildbots
And so what would need to be done to rectify this problem? If there are no objections, I would like to resolve this particular point, at least until better people can be found to do so. On Thu, Sep 4, 2014 at 8:50 PM, A. Cavallo wrote: > Yes there are "details" indeed. But not show stoppers. A prototype can be > seen here: http://cclimited.webfactional.com > The nice bit is testing will be executed in a total fresh instance, an > added bonus > > On 4 September 2014 05:16:01 GMT+01:00, Nick Coghlan > wrote: >> >> On 4 September 2014 11:07, Antonio Cavallo wrote: >> >>> >>> I wonder if there is any interest in starting to use the opensuse build >>> servers for continuous build and testing on redhat, fedora suse and (I >>> think) debian: that will solve once for all the maintenance issues on those >>> >>> platforms (and provide a reliable build). >>> >> >> It sounds like a good idea in principle, but I suspect there would be >> a lot of devils in those details. That doesn't make it a bad idea, >> just something that would need a volunteer to investigate further >> >> before we could form a real opinion. >> >> Cheers, >> Nick. >> >> > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rajshorya%40gmail.com > > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython and parallel make
Hi, Thanks for your comments! I've done more tests on a new machine and, for better or worse, was no longer able to reproduce the issue. Parallel compilation now works fine; I'll keep collecting output from make, and would post it here if the problem appears again. One additional issue appeared, though: it seems that extension modules are always built sequentially no matter what the value of MAKEFLAGS is. I've seen that these are being built by a custom setup.py script. Do you think it would be possible to parallelize this script? Would people be interested in having a parallel version? Cheers, Jonas ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython and parallel make
On Thu, 4 Sep 2014 12:06:25 +0200 Jonas Wagner wrote: > > One additional issue appeared, though: it seems that extension modules are > always built sequentially no matter what the value of MAKEFLAGS is. I've > seen that these are being built by a custom setup.py script. Do you think > it would be possible to parallelize this script? Would people be interested > in having a parallel version? See http://bugs.python.org/issue5309 Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sad status of Python 3.x buildbots
Setting up a repo with the code and cleaning a bit here and there. Over the weekend I can put something useable. Nick Coghlan wrote: On 4 September 2014 18:50, A. Cavallo wrote: Yes there are "details" indeed. But not show stoppers. A prototype can be seen here: http://cclimited.webfactional.com The nice bit is testing will be executed in a total fresh instance, an added bonus Oh, very nice! What is needed to get that up and running? Regards, Nick. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
