> On Dec 29, 2016, at 4:20 PM, Tim Allen <screwt...@froup.com> wrote:
> 
> On Thu, Dec 29, 2016 at 05:27:44AM -0800, Glyph Lefkowitz wrote:
>> Increasingly, we are assuming a pip-like packaging toolchain for
>> dealing with Twisted's dependencies, so getting familiar with this
>> stuff - pip, requirements.txt, pinning, virtualenv - is worthwhile.
>> (It'll make your life easier in more ways than one.)
> 
> I wasn't going to mention it (because we found a workaround), but since
> the topic has come up...
> 
> My current employer has a fully pip-based packaging toolchain for all
> our internal Python apps, where we build wheels for our apps and all
> their dependencies, and then deploy them into a virtualenv in
> production. For reproducability reasons, we have a PyPI mirror inside
> the corporate firewall, and our wheel-building Docker image has
> a `pip.conf` configured to point at it.
> 
> Twisted 16.6 broke this system when it added a package named
> "incremental" to the `setup_requires` list in setup.py. Because
> `setup_requires` is a feature provided by setuptools, packages listed
> there are installed by (the same machinery as) setuptools' `ez_install`
> command, which does *not* respect `pip.conf`, and so it tried to
> download the package from upstream PyPI and timed out banging its head
> against the firewall.

Thanks, this is very useful information.

If you wanted to have this verified for the future, a network-isolated buildbot 
would be a good way to ensure we don't mess this up in other ways.  Based on 
this thread it seems that multiple users have this concern, so it's certainly 
not specific to you, either.

It would also force us to fix the dang documentation-generator test cases that 
keep failing when a certain someone forgets to update their SSL cert :-).

> As a workaround, we messed with our build-scripts to manually "pip
> install" incremental before installing Twisted, or anything that depends
> on it, so everything's fine again.
> 
> The Python Packaging Authority already has plans for replacing
> `setup_requires` with something more Pip-friendly (see PEP 518), so over
> time this won't be an issue. For the short term, though, I hope there
> won't be too many more things added to `setup_requires` without
> announcement.

Incremental is a bit of a special case, since "version definition" is one of 
the few things that needs synchronization between setup-time and run-time.  I 
was pretty nervous about adding `setup_requires`, but the value of getting our 
versioning logic outside of Twisted itself seemed worth the potential packaging 
fallout.

So, while you can expect to see many more things showing up in install_requires 
unannounced, as we split out low-level portions of Twisted and adopt libraries 
like h11, I doubt that you'll see another setup_requires arrive, especially 
before PyPA gets together a better approach than the literal setup.py 
setup_requires.  Given that you've specifically asked for notice, any project 
member submitting or reviewing a patch that adds one should give a shout here.

-glyph
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Reply via email to