Re: Python 3.4 and ensurepip (rehashed, long)

[Donald Stufft]

Hey there,

Pip/virtualenv maintainer here and also one of the authors of PEP453. I had
someone complaining to me that Python 3.4’s venv module was broken today and
through investigating that I found this discussion. I'm signed onto the list
now :)

I think I wrote most of those recommendations, I'm not a downstream packager
but I had hoped that if they were wrong someone would have spoke up. I know
the PEP is accepted already but personally I don't have a problem with updating
the recommendations if something else makes sense and still fulfills the
spirit of the PEP.

Also yea pip in the system Python currently kind of sucks. I want to make this
better eventually! I just don't know how yet or have the cycles to spend
investigating it.

-----
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Python 3.4 and ensurepip (rehashed, long)

[Donald Stufft]

On Mar 21, 2014, at 3:24 PM, Scott Kitterman  wrote:

> On Wednesday, March 19, 2014 17:40:51 Barry Warsaw wrote:
>> Signed by ba...@warsaw.us.   Show Details
>> TL;DR: Let's re-enable the ensurepip module in Python 3.4, and possibly
>>   address some usability issues.  We should descend en masse on Montreal
>> and stage a revolt at Pycon. :)
>> 
>> Python 3.4 has an `ensurepip` module[1] which implements the specification
>> in PEP 453 regarding the explicit bootstrapping of pip in Python
>> installations[2]. This is promoted as a boon to users, especially on
>> platforms without OS provided package managers, i.e. not Debian.
>> 
>> The PEP makes some recommendations for downstreams[3], which we do not
>> currently adopt, and maybe shouldn't fully.  Our current Python 3.4 package
>> disables ensurepip at build time.
>> 
>> The previous discussion on this subject[4] was extensive, and it may not be
>> worth rehashing that.  I just reviewed the thread via Gmane, and AFAICT
>> there is some general consensus:
>> 
>> * pip should not be used to install packages "globally", i.e. in the system
>>  Python's dist-packages.
> 
> Until installing in the user's space is the default,

I’m ok (as a pip maintainer) with that happening in concept. There’s stuff that
needs to be done and figured out related to how it’ll work in practice, 
backwards
compatibility, consistency (especially with Windows which we unfortunately do
have to care about).

> I don't see it as being 
> suitable for installation by default.  I liked Piotr's idea about an option 
> along the lines of -i-know-if-a-break-it-i-keep-both-halves to override that.
> 
> Scott K

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Python 3.4 and ensurepip (rehashed, long)

[Donald Stufft]

On Mar 21, 2014, at 3:39 PM, Donald Stufft  wrote:

> 
> On Mar 21, 2014, at 3:24 PM, Scott Kitterman  wrote:
> 
>> On Wednesday, March 19, 2014 17:40:51 Barry Warsaw wrote:
>>> Signed by ba...@warsaw.us.  Show Details
>>> TL;DR: Let's re-enable the ensurepip module in Python 3.4, and possibly
>>>  address some usability issues.  We should descend en masse on Montreal
>>> and stage a revolt at Pycon. :)
>>> 
>>> Python 3.4 has an `ensurepip` module[1] which implements the specification
>>> in PEP 453 regarding the explicit bootstrapping of pip in Python
>>> installations[2]. This is promoted as a boon to users, especially on
>>> platforms without OS provided package managers, i.e. not Debian.
>>> 
>>> The PEP makes some recommendations for downstreams[3], which we do not
>>> currently adopt, and maybe shouldn't fully.  Our current Python 3.4 package
>>> disables ensurepip at build time.
>>> 
>>> The previous discussion on this subject[4] was extensive, and it may not be
>>> worth rehashing that.  I just reviewed the thread via Gmane, and AFAICT
>>> there is some general consensus:
>>> 
>>> * pip should not be used to install packages "globally", i.e. in the system
>>> Python's dist-packages.
>> 
>> Until installing in the user's space is the default,
> 
> I’m ok (as a pip maintainer) with that happening in concept. There’s stuff 
> that
> needs to be done and figured out related to how it’ll work in practice, 
> backwards
> compatibility, consistency (especially with Windows which we unfortunately do
> have to care about).
> 
>> I don't see it as being 
>> suitable for installation by default.  I liked Piotr's idea about an option 
>> along the lines of -i-know-if-a-break-it-i-keep-both-halves to override that.
>> 
>> Scott K
> 
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

https://github.com/pypa/pip/issues/1668

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Python 3.4 and ensurepip (rehashed, long)

[Donald Stufft]

On Mar 25, 2014, at 3:11 PM, Barry Warsaw  wrote:

> On Mar 21, 2014, at 04:30 AM, Bohuslav Kabrda wrote:
> 
>> I'm on python-dev but not pypa-dev. But if you want to take the discussion
>> there, I guess I'm ok with following yet another mailing list.
> 
> Gmane makes life on a zillion mailing lists so much more manageable. ;)
> 
>> Yep, they do that. That's actually another thing that we're working on right
>> now - creating an upstream-acceptable patch that would make all system-wide
>> pip installations go into /usr/local/lib/..., while system-wide RPM installed
>> packages would install into /usr/lib/... We know that Debian already has some
>> downstream patches for this and there is also a bug opened for this upstream
>> [1].  But yeah, currently users just "sudo pip install foo" into system
>> Python packages path.
> 
> Many Pycons ago, Toshio, myself, Tarek, and several others discussed a scheme
> where by (through more indirection of course), various packaging tools could
> query configuration files to determine operating system specific paths.  I
> thought it was quite a nice scheme which would have let us easily drop a conf
> file to declare where Debian wanted certain files installed, and Fedora or OS
> X, etc could declare others.  Mostly this was in the context of the (now
> abandoned) successor to setup.py, but I think tools like pip and such could
> have taken advantage of that.
> 
> I'd still like something like that.  On Debian, Python adds a /usr/local
> directory to sys.path, but it's not /usr/local/lib/pythonX.Y/site-packages,
> it's .../dist-packages for reasons we've long hashed out and I don't want to
> re-open.  Suffice to say that we do have a directory where `sudo pip install`
> could install to that would play nice with system Python but wouldn't
> interfere with distro packages.  While I don't much like `sudo pip install` as
> a general recommendation, in principle we support that.
> 
> Just to keep in mind when proposing this upstream that distros will have
> similar goals, but different concrete paths.

I have a half formed idea in my head about a PEP that adds more directories
on sys.path and the relevant APIs to query them (pip doesn’t decide where to
install stuff, it just asks distutils where to install stuff).

> 
>>> What will rewheeling prefer when:
>>> 
>>> * bundled ones are newer than system ones?
>> 
>> That's not currently implemented in our patch, since we'll be removing
>> bundled wheels and relying on system versions only. But to solve this
>> situation for the upstream patch, we'll do it like this
>> - if user runs just "python3 -m ensurepip", nothing will happen
>> - if user runs "python3 -m ensurepip --upgrade", his setuptools/pip will get 
>> upgraded
>> Note, that this corresponds with current ensurepip documentation [2] of such
>> situations.
> 
> I read the patch, and I think I get it, but just to be clear, rewheeling
> only covers pip and setuptools, not arbitrary other packages, right?

As far as I understand it yes, also Fedora’s python-pip still bundles
pip._vendor.* 

> 
> I still don't like introducing the circular dependency in Debian, so Piotr's
> suggestion is about where I think Debian should head.   That is:
> `python3 -m ensurepip` would suggest (but not execute) to install python3-pip
> (our binary package providing pip for Python 3) if it's not already
> installed.  The key thing is to make sure that pyvenv works in both cases.
> 
> I'm not exactly sure about —upgrade.

I assume once someone has installed pip with apt-get they’d still be able
to run pip install —upgrade pip if they wanted too?

> 
>>> I'm not sure what we would do if we wanted avoided the dependency cycle, and
>>> pip/setuptools wasn't yet installed system wide.
>> 
>> Yeah, that's a tough one and it's actually one of the reasons why we decided
>> to introduce the dependency cycle. The obvious (and IMO wrong) solution would
>> be subprocess.Popen(['sudo', 'yum', 'install', 'python-pip']), but that just
>> isn't something that I'd want to do.
> 
> Right, we wouldn't either I think.
> 
> Cheers,
> -Barry
> 


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Python 3.4 and ensurepip (rehashed, long)

[Donald Stufft]

On Mar 25, 2014, at 3:18 PM, Barry Warsaw  wrote:

> On Mar 21, 2014, at 02:28 PM, Matthias Klose wrote:
> 
>> - whether to use python3-pip or a freshly downloaded pip.  When using
>>   pip to install for the system python3, maybe try to use the shipped
>>   python3-pip.  I have no opinion if another pip is downloaded and
>>   installed into for environments created by venv or virtualenv.  Some
>>   package builds use virtual environments, but the population of these
>>   has to be satisfied by package build dependencies anyway.
> 
> In general, I'd like to use the Debian package pip/setuptools for installation
> into system locations, and I am okay with using bundled pip/setuptools for
> installation into virtualenvs.  I think that's pretty much the current state
> of affairs anyway, except for pyvenv.

Doing what you do for system locations and bundling a Python package is
the status quo for virtualenv, which the ensurepip/venv combo copied that
technique.

> 
> Cheers,
> -Barry
> 
> 
> -- 
> To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
> Archive: https://lists.debian.org/20140325151821.438fa...@limelight.wooz.org
> 


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Python 3.4 and ensurepip (rehashed, long)

[Donald Stufft]

On Mar 26, 2014, at 9:12 AM, Piotr Ożarowski  wrote:

> [Barry Warsaw, 2014-03-25]
>> One of the things I'd like to see, in addition to supporting
>> platform-specified system-level installation directories
>> (i.e. /usr/local/lib/pythonX.Y/dist-packages on Debian), is an upstream 
>> switch
>> to tell Python to ignore this directory, mirroring e.g. -s.  That way, if
>> people say "Well I just pip installed foo into /usr/local and it broke my
>> system" we'd be able to respond "you better use pythonX.Y --dont-blame-us".
> 
> I like the idea (where --dont-blame-us simply disables support for
> .egg/.zip/.whl in the interpreter, assuming pip/easy_install installs
> zips instead of unpacking them)
> -- 
> Piotr Ożarowski Debian GNU/Linux Developer
> www.ozarowski.pl  www.griffith.cc   www.debian.org
> GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
> 
> 
> -- 
> To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
> Archive: https://lists.debian.org/20140326131210.gd30...@sts0.p1otr.com
> 

Pip does not install as an .egg or a .whl. It doesn’t use .egg at all and 
regardless
of which format it downloads it unpacks it and installs it. There is no 
difference in
format between what pip installs and any debian package I’ve ever looked at.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Python 3.4 and ensurepip (rehashed, long)

[Donald Stufft]

On Mar 26, 2014, at 9:16 AM, Piotr Ożarowski  wrote:

> [Donald Stufft, 2014-03-26]
>> On Mar 26, 2014, at 9:12 AM, Piotr Ożarowski  wrote:
>>> [Barry Warsaw, 2014-03-25]
>>>> One of the things I'd like to see, in addition to supporting
>>>> platform-specified system-level installation directories
>>>> (i.e. /usr/local/lib/pythonX.Y/dist-packages on Debian), is an upstream 
>>>> switch
>>>> to tell Python to ignore this directory, mirroring e.g. -s.  That way, if
>>>> people say "Well I just pip installed foo into /usr/local and it broke my
>>>> system" we'd be able to respond "you better use pythonX.Y --dont-blame-us".
>>> 
>>> I like the idea (where --dont-blame-us simply disables support for
>>> .egg/.zip/.whl in the interpreter, assuming pip/easy_install installs
>>> zips instead of unpacking them)
>> 
>> Pip does not install as an .egg or a .whl. It doesn’t use .egg at all and 
>> regardless
>> of which format it downloads it unpacks it and installs it. There is no 
>> difference in
>> format between what pip installs and any debian package I’ve ever looked at.
> 
> then --dont-blame-us has to mean removing /usr/local from sys.path
> -- 
> Piotr Ożarowski Debian GNU/Linux Developer
> www.ozarowski.pl  www.griffith.cc   www.debian.org
> GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
> 
> 
> -- 
> To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
> Archive: https://lists.debian.org/20140326131639.ge30...@sts0.p1otr.com
> 

In my half formed idea in my head the way it’d work is there’d be a 
vendor-packages
directory where downstream can install things to, and a flag to the interpreter 
to
remove the typical site-packages. So then you’d get something like:

python -I —no-site-packages -m something

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Python 3.4 and ensurepip (rehashed, long)

[Donald Stufft]

On Mar 26, 2014, at 10:35 AM, Barry Warsaw  wrote:

> On Mar 26, 2014, at 09:24 AM, Donald Stufft wrote:
> 
>> In my half formed idea in my head the way it’d work is there’d be a
>> vendor-packages directory where downstream can install things to, and a flag
>> to the interpreter to remove the typical site-packages. So then you’d get
>> something like:
>> 
>>   python -I —no-site-packages -m something
> 
> There has to be a short option for --no-site-packages (or whatever) so that it
> will work with shebang lines, where we already recommend -Es.

Yea I just didn’t feel like thinking up a short option :)

>  But also, -I
> should imply this new option for full isolation.

Not sure about this, I don’t think I saw the original discussion but it looks 
like
-I is to prevent the user from injecting malicious code (so it removes env vars,
the user site packages, the current dir, etc). I don’t think that something
installed by pip by the system administrator falls under that.

> 
> Which means for Python 3.4 and beyond we should be recommending system
> services and scripts add -I to the shebang line instead of -Es.  Then we'd get
> this new /usr/local isolation switch for free.
> 
> -Barry


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Python 3.4 and ensurepip (rehashed, long)

[Donald Stufft]

\o/

> On Apr 4, 2014, at 7:12 PM, Barry Warsaw  wrote:
> 
>> On Mar 19, 2014, at 05:40 PM, Barry Warsaw wrote:
>> 
>> The current situation in the Python 3.4 package is suboptimal because:
>> 
>> % pyvenv-3.4 /tmp/zz
>> Error: Command '['/tmp/zz/bin/python3.4', '-Im', 'ensurepip', '--upgrade', 
>> '--default-pip']' returned non-zero exit status 1
> 
> Please see issue #732703.  I have a patch that restores usable functionality
> to pyvenv.  Hopefully Matthias will approve and apply the patch.
> 
> Cheers,
> -Barry


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/010b1ae2-0d89-4b3d-9ba7-1c35f9d9b...@stufft.io

Re: Bug #732703 and fixing ensurepip/pyvenv

[Donald Stufft]

On May 6, 2014, at 6:28 PM, Barry Warsaw  wrote:

> PEP 427 does discourage direct import from wheels[1] but for the things we
> care about immediately, the two conditions should not affect us (i.e. no C
> extensions, and no bad use of __file__) so I think we could get away with it.
> I'm open to other good arguments for or against using the wheels directly.

For what it’s worth, I’m the one who pushed hard against ZipImport capabilities
being codified in PEP427 and I can say that we are *incredibly* unlikely to
remove that ability without introducing a better way of achieving that. So even
if the current method breaks, there is almost certainly going to be a path 
forward.

---------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Bug #732703 and fixing ensurepip/pyvenv

[Donald Stufft]

On May 7, 2014, at 11:35 AM, Barry Warsaw  wrote:

>  The stuff vendorized inside there don't have to have the
> dist-info directories, so perhaps they don't need to be wheeled at all, but
> they *do* have to be vendorized because of technicalities in the way pip works
> within a virtual environment.

To be specific, they cannot appear as installed to a ``pip list`` command,
where they are imported from doesn’t particular matter.

Acceptable:

1. Putting them unpacked in pip/_vendor/ and modify the imports to import things
   from pip._vendor (aka what pip normally does)
2. Putting them unpacked in pip/_vendor/ (or anywhere really) *without*
   dist-info/egg-info and modifying sys.path to add that path.
3. Putting them packed in a zip file (whl or plain zip) in pip/_vendor
   (or anywhere really) and modifying sys.path to add those zips.

Unacceptable:

4. Installing them directly into the virtual environment
5. Putting them unpacked in pip/_vendor/ (or anywhere really) *with*
   dist-info/egg-info and modifying sys.path to add that path.

Basically if pkg_resources can find a dist-info/egg-info for it, then it'll
see it installed and it'll be available for ``pip uninstall`` (explicitly or
implicitly through an upgrade/downgrade).

Also to be explicit, I've never actually tested what pip will do if there are
*two* copies of requests (for example), one installed into site-packages with a
dist-info/egg-info and one being just the ``requests`` directory that had been
added into sys.path (basically options 2 and 3). I *think* it will work but
that should be tested.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Bug #732703 and fixing ensurepip/pyvenv

[Donald Stufft]

On May 9, 2014, at 6:16 PM, Barry Warsaw  wrote:

> A follow up.
> 
> I've done a fair bit of prototyping a solution, and with Donald's gracious
> help and feedback, I think I have a plan that will work and should be
> compliant with policy.  I'm beginning to make changes to various DPMT packages
> to build the whole stack, but I'm only uploading the simpler changes for now
> until I have a working local testbed.
> 
> The basic approach does involve building a bunch of wheels as explained
> earlier.  We have to build the pip and setuptools wheels from our devendorized
> packages, and install those into the venv's site-packages directory.
> Installation is done by ensurepip, and I have a branch of python3.4 that
> re-enables this.
> 
> Now that python3-wheel has cleared NEW, I need to modify the d/rules files to
> build the wheels.  I have this working for pip in svn (along with significant
> packaging changes to use pybuild and update to 1.5.5), but I won't upload this
> just yet.  It installs the wheels into /usr/share/python-wheels.
> 
> The tricky bit is what to do about the recursively vendorized packages.  Here
> I think the best course of action is as outlined previously - build wheels of
> these from quilt-patched Debian source at package build time, and install
> those too into /usr/share/python-wheels.  When pyvenv is called, the
> appropriate wheels will be copied into /usr/share/python-wheels and the
> venv's pip will know to add those to sys.path.  We cannot unpack these
> dependent wheels into the venv's site-packages so this is to me the simplest
> and best approach.  I think Donald agrees.  The one ugly bit is what to do
> about the venv wheels when you `pip upgrade pip` since this will pull the PyPI
> pip into the venv.  Our unvendorized wheels won't get deleted from the venv,
> but they also won't be used (because PyPI's pip will bundle, and won't contain
> the patch to use the unvendored versions).  IOW, our wheels will stick around
> as unused trash.  Oh well - this only happens if you `pip upgrade pip` in the
> venv.  (We can try to fix that later if we really care.)

Yes I agree. 

For the record I don’t think getting pip to uninstall them during an upgrade 
will
be very difficult but if you’re OK with them sticking around us unused turds
then there’s no real harm to not doing it.

> 
> Other than extensive testing , what needs to happen now is that I need
> to modify the packaging of the dependencies to build the wheels.  This is
> fairly easy for html5lib, requests, python3-chardet, and urllib3, although
> some of these need their setup.py patched to use setuptools instead of
> distutils.core (because only the former supports the bdist_wheel command
> provided by python3-wheel).  These packages are all team maintained.
> 
> It's trickier for distlib, colorama, and six because those are not team
> maintained, so I will generate patches and pester the maintainers once I've
> proven that the whole stack works.  I've reached out to the colorama
> maintainer to see if we can move that into DPMT, since it's only ever seen
> NMUs since the initial upload.  Colin, Matthias, keep an eye out for patches
> to six and distlib respectively.
> 
> I won't finish this up this weekend, but will continue working on it early
> next week.
> 
> Stay tuned, and best to ping me on IRC if you have questions.
> 
> Cheers,
> -Barry


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: wheel support for Debian?

[Donald Stufft]

On May 15, 2014, at 6:32 PM, Barry Warsaw  wrote:

> My thoughts...
> 
> On May 16, 2014, at 12:07 AM, Matthias Klose wrote:
> 
>> - should we add wheels everywhere? I don't think we should,
>>  but I'd like to state this somewhere, like in the python policy.
> 
> Agreed, we should not add wheels everywhere.  I would like to keep it very
> limited to exactly the (small) set of packages we need to devendorize
> ensurepip, recursively.  If some other devendorizing task in the future
> requires the use of wheels, then we have a framework in place, but I would
> like to actively discourage their use.
> 
> I do plan to propose an update to policy stating this, but I haven't gotten to
> that yet.  I will of course post the proposed update here first.
> 
>> - where to put wheels?  /usr/share/python-wheels is an ad-hoc
>>  decision which was never proposed. I'm aware about "universal"
>>  wheels but I'd like to clarify where wheels should be located.
>>  Do we need /usr/share/python/wheels, and/or /usr/share/python3/wheels?
> 
> I proposed /usr/share/python-wheels here:
> 
> https://lists.debian.org/debian-python/2014/05/msg00025.html
> 
> but it's a detail that was probably easily lost in the wall of text.  I didn't
> see any objections to that specifically though.  We could change it if
> something clearly better is proposed, although it would necessitate some new
> uploads and updated quilt patches.
> 
> For the current use case, we only need pure-Python wheels, and in fact Python
> can't currently import extension modules from zips, so architecture dependent
> wheels wouldn't work anyway.  Universal wheels (Python 2 and 3 compatible) are
> used because that's what the ensurepip machinery already uses.  it's just as
> easy to create universal wheels, and all the packages we currently care about
> *are* bilingual, so using them here reduces the upstream delta.  Since I don't
> view the building of wheel packages as general purpose, I think it's fine to
> just put them in a shared directory.
> 
> In other words, non-universal wheels YAGNI.

For the record since it’s pip that this is for, we have no plans to ever 
include something
that won’t work with a universal Wheel. Our requirements for us to depend on 
something
is 2.6, 2.7, 3.2+ with single source and pure Python.

> 
>> - naming of wheel packages.  It's good to see wheels packaged
>>  in a separate binary package. However there is no proposal
>>  how to name these packages.
> 
> That was also proposed in the above referenced message.  Suggestions welcome,
> but I think python-foo-wheels is as good as anything (it's pretty
> self-descriptive ;).
> 
> Cheers,
> -Barry
> 
> 
> -- 
> To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
> Archive: https://lists.debian.org/20140515183201.6acc4...@anarchist.wooz.org
> 


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: wheel support for Debian?

[Donald Stufft]

Unpacking the Wheel files will trigger pip to see them as installed and it
will attempt to uninstall them if it needs to. If they are going to be unpacked
it’ll need to have the .dist-info directory removed from the unpacked locations
in order to prevent that from happening.

On May 19, 2014, at 7:19 AM, Jakub Wilk  wrote:

> * Matthias Klose , 2014-05-19, 12:44:
>> we don't unpack jar files either.
> 
> Fortunately, we don't have to mimic all the Java misfeatures.
> 
> -- 
> Jakub Wilk
> 
> 
> -- 
> To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
> Archive: https://lists.debian.org/20140519111953.gc8...@jwilk.net
> 


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Proposed changes to python-virtualenv

[Donald Stufft]

On May 29, 2014, at 8:15 PM, Scott Kitterman  wrote:

> On May 29, 2014 7:54:53 PM EDT, Barry Warsaw  wrote:
>> I'm looking again at updating tox to the latest upstream 1.7.1.  Along
>> the
>> way, I'd like to make /usr/bin/tox a Python 3 script.
>> 
>> This requires that virtualenv be importable, e.g. `$python -m
>> virtualenv`.  It
>> is today in Python 2 since /usr/bin/virtualenv is a Python 2 script and
>> we
>> only build it for one version of Python.  I want to change that too, so
>> that
>> at least we build the virtualenv modules for both Python 2 and 3.  I'd
>> like to
>> switch /usr/bin/virtualenv to be Python 3 as well, and I'd like to
>> tackle the
>> vendorizing issue similar to how we fixed it with ensurepip, since we
>> have all
>> the wheels we need now.
> ...
>> Thoughts?
>> -Barry
> 
> I'd rather remove the wheels we have and give up on ensurepip than start down 
> this slippery slope. 
> 
> Wheels are an ugly solution for us to work around upstream doing things that 
> can most charitably be described as fixing problems we don't have.  No more. 
> 
> Scott K

Eh, it’s not true that users of Debian do not have the problems that ensurepip
solves. Perhaps *you* don’t personally have them but anyone whose ever needed
to install a set of Python packages that Debian either doesn’t package, or
packages the wrong version of them have the problems that virtualenv/venv
solves and virtualenv/venv without pip is practically worthless.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Proposed changes to python-virtualenv

[Donald Stufft]

On May 29, 2014, at 7:54 PM, Barry Warsaw  wrote:

> I'm looking again at updating tox to the latest upstream 1.7.1.  Along the
> way, I'd like to make /usr/bin/tox a Python 3 script.
> 
> This requires that virtualenv be importable, e.g. `$python -m virtualenv`.  It
> is today in Python 2 since /usr/bin/virtualenv is a Python 2 script and we
> only build it for one version of Python.  I want to change that too, so that
> at least we build the virtualenv modules for both Python 2 and 3.  I'd like to
> switch /usr/bin/virtualenv to be Python 3 as well, and I'd like to tackle the
> vendorizing issue similar to how we fixed it with ensurepip, since we have all
> the wheels we need now.
> 
> This means however that I need to rejigger the binary packages in the
> python-virtualenv source package.  Right now, python-virtualenv contains both
> the modules and /usr/bin/virtualenv.  I need to at least add a
> python3-virtualenv package to contain the Python 3 modules.  But it doesn't
> make sense to have a /usr/bin/virtualenv with both a Python 2 and Python 3
> shebang line, and thus it doesn't make sense to put Python 3
> /usr/bin/virtualenv in the python-virtualenv binary package.  However, I still
> want to make it obvious and easy which package someone needs to install to get
> the command line script.
> 
> My thinking is that I would add a `virtualenv` binary package which would
> contain just the /usr/bin script (with a Python3 shebang) and any other common
> files as makes sense (e.g. the manpage).  `virtualenv` would Replaces/Breaks
> `python-virtualenv` and Depends on python3-virtualenv.
> 
> I thought about using a python-virtualenv-common for that but since this will
> be the most likely installed package for end users, I thought `apt-get install
> virtualenv` looked nicer.
> 
> Thoughts?
> -Barry

Does anything other than tox depend on virtualenv? Unless something python 2.x
depends on virtualenv the only real benefit to having virtualenv installed in
both 2.x and 3.x is what the default interpreter is whenever you create a
virtual environment. IOW virtualenv is perfectly capable of creating virtual
environments in interpreters other than the one it's installed in.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Dependency of virtual environments on Python wheel (was: Proposed changes to python-virtualenv)

[Donald Stufft]

On May 29, 2014, at 9:18 PM, Ben Finney  wrote:

> Barry Warsaw  writes:
> 
>> On May 29, 2014, at 08:15 PM, Scott Kitterman wrote:
>> 
>>> I'd rather [Debian] remove the wheels we have and give up on
>>> ensurepip than start down this slippery slope.
>> 
>> That means we give up on pyvenv, and given that virtualenv will
>> eventually be a wrapper around pyvenv, that means we give up on
>> virtual environments.
> 
> Could you expand on that? What is it about pyvenv, or wheel, or both,
> that leads from Debian removing wheel packages to Debian “giving up on
> pyvenv”?
> 
> Or, if you prefer, what is it about pyvenv that makes it infeasible to
> do without wheel packages?

pyvenv doesn't strictly need Wheels themselves, but it needs something that
is capable of interrogating Python to ask it where it should install the
packages as well as which is capable of understanding virtual environments at
all.

Historically virtualenv has done this by embedding sdists, manually unpacking
them, and then executing setup.py. With the most recent version of virtualenv,
and with pyenv (which uses ensurepip) this has been switched to wheels since
they are really just fancy zip files which means that Python is able to import
them directly. This allows pyenv (again via ensurepip) and virtualenv to use
pip to install both itself and setuptools instead of directly executing
setup.py. This has made the bootstrapping of setuptools/pip into these virtual
environments cleaner, faster, and simpler.

Strictly speaking as well, pyvenv itself doesn't require ensurepip (where the
wheels are being used). However pyvenv does default to bootstrapping pip into
the virtual environment which that particular feature *does* require ensurepip
and the Wheel files. In Python 3.3 there is no ensurepip and pyvenv does not
include pip, but feedback has overwhelmingly been that 3.3's pyvenv is useless
because it does not provide any means of installing things into it like
virtualenv does. This has lead people to continue to use virtualenv instead
of pyvenv which can hopefully be rectified in 3.4+ now.

Additionally it's important to remember that the only reason virtualenv itself
works on Debian is because it violates the Debian policy and ships the Wheel
files that I (as the person whose released the upstream virtualenv) included
within it. The solution that Barry has with pyvenv is more complex than simply
including those files as python-virtualenv does, however it is my understanding
that is allows pyvenv to be compliant with Debian policy.

> 
>> Wheels are a solution to a problem that is *explicitly* Debian, given
>> the policy and DFSG violations vendorizing entails. Upstream doesn't
>> have the same concern.
> 
> (I think by “to vendorize” you mean “to bundle”. Why the neologism
> “vendorize”, what is the distinction?)

I think the general distinction is that bundling is more generic than vendoring.
Vendoring typically refers to including the code of another package as part of
the code of the primary package. For instance, pip vendors requests (and others)
as pip._vendor.requests. It is effectively a part of the pip namespace now
instead of simply bundling the two packages together so that installing pip
also installs requests.

> 
> What do wheels do which address Debian in particular? Why is it that
> Debian can't ignore wheels as superfluous?

Wheels don't address a problem that is specific to Debian, as I explained above
we need a package to install into the virtual environment. As far as I am aware
dpkg is not aware of virtual environments and I cannot install a package (such
as python-pip) multiple times into multiple virtual environments on a single
machine. So more accurately this solves a generic problem for which the
typical Debian machinery does not have a solution for.

> 
> -- 
> \  “It is well to remember that the entire universe, with one |
>  `\   trifling exception, is composed of others.” —John Andrew Holmes |
> _o__)  |
> Ben Finney
> 
> 
> -- 
> To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
> Archive: https://lists.debian.org/85iooo2hez.fsf...@benfinney.id.au
> 


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: python-pip and python-virtualenv broken in sid after upgrade to python 2.7.7rc1

[Donald Stufft]

On Jun 2, 2014, at 8:31 AM, Mário Duarte Cruz  wrote:

> Hi,
> 
> Since the upgrade to python 2.7.7rc1 in sid, pip and virtualenv's pip have 
> been broken, most likely due to https://bugs.python.org/issue7776 .
> 
> Updating pip to 1.5.6 and virtualenv to 1.11.6 should fix the issues (current 
> versions in sid are 1.5.5 and 1.11.4 respectively).
> 
> Apologies for not using the bug tracker, too many hoops to jump through. I'd 
> usually just wait it out - sid being sid - but this has been irking me for 
> over a week.
> 
> Cheers, 
> 
> Mario Cruz
> 


pip 1.5.6 just upgrades the bundled requests FWIW.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Proposed changes to python-virtualenv

[Donald Stufft]

On Jun 2, 2014, at 4:35 PM, Barry Warsaw  wrote:

> On May 30, 2014, at 11:47 AM, Barry Warsaw wrote:
> 
>> So let's say pbundler is the only thing other than tox that does this.
> 
> Well, now pbundler has been removed from the archive, so I don't care about
> it anymore. ;)  I can't see any reason not to make virtualenv a Python 3
> application and Python 3-only library.  Therefore, I won't change the binary
> packages built by this, but just retarget them to Python 3.
> 
> Cheers,
> -Barry

Sounds reasonable to me, the only “downside” is that virutalenv will default
to Python 3, which is probably not what most people want (however they
can do virtualenv -p python2).

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: python-pip and python-virtualenv broken in sid after upgrade to python 2.7.7rc1

[Donald Stufft]

On Jun 3, 2014, at 5:28 AM, Mário Duarte Cruz  wrote:

> On Mon, Jun 2, 2014 at 2:51 PM, Donald Stufft  wrote:
> 
> 
> pip 1.5.6 just upgrades the bundled requests FWIW.
> 
> 
> 
> I forgot to mention that it breaks if you're sitting behind a proxy (might 
> not be obvious from a cursory look at the linked bug):
> 
> [...]
> Getting page https://pypi.python.org/simple/pip/
> Could not fetch URL https://pypi.python.org/simple/pip/: connection error: 
> hostname 'proxy.company.com' doesn't match either of '*.c.ssl.fastly.net', 
> 'c.ssl.fastly.net', [...]
> 
> 
> @Donald: the currently available get-pip.py seems to also be affected by the 
> same bug
> 

Ah yea, I forgot to rebuild that with 1.5.6, I’ll get it fixed in a bit. Thanks.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: python-pip and python-virtualenv broken in sid after upgrade to python 2.7.7rc1

[Donald Stufft]

The fix in 1.5.6 was updating requests. That's the only change. I think Debian 
needs to update urllib3. That's where the actual fix was. 

> On Jun 3, 2014, at 4:07 PM, Barry Warsaw  wrote:
> 
>> On Jun 03, 2014, at 11:28 AM, Mário Duarte Cruz wrote:
>> 
>> I forgot to mention that it breaks if you're sitting behind a proxy (might
>> not be obvious from a cursory look at the linked bug):
> 
> I'm about to upload pip 1.5.6 to unstable.  Please double check that this is
> fixed there, and if not, please open a bug on the python-pip package.
> Remember though that we devendorize all those bundled dependencies.
> 
> Cheers,
> -Barry
> 
> 
> -- 
> To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
> Archive: https://lists.debian.org/20140603160757.53458...@anarchist.wooz.org
> 


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/b799b798-7c22-4f5b-8e5a-6128151a7...@stufft.io

Re: python-pip and python-virtualenv broken in sid after upgrade to python 2.7.7rc1

[Donald Stufft]

On Jun 3, 2014, at 6:55 PM, Barry Warsaw  wrote:

> On Jun 03, 2014, at 04:31 PM, Donald Stufft wrote:
> 
>> The fix in 1.5.6 was updating requests. That's the only change. I think
>> Debian needs to update urllib3. That's where the actual fix was.
> 
> I think you mean we need to update the requests package.  That's on my list.
> 
> -Barry

Do you unbundle urllib3 from requests? The fix that broke proxying was a
thing in urllib3, which requests bundles, which pip bundles requests.

Here’s the actual fix for that issue https://github.com/shazow/urllib3/pull/385


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Proposed changes to python-virtualenv

[Donald Stufft]

On Jun 4, 2014, at 4:20 PM, Stefano Rivera  wrote:

> Hi Barry (2014.06.02_22:51:24_+0200)
>>> Sounds reasonable to me, the only “downside” is that virutalenv will default
>>> to Python 3, which is probably not what most people want (however they
>>> can do virtualenv -p python2).
>> 
>> But it's what most people *should* want .
> 
> Won't python 3 users be using python3 -m venv?
> 
> The only real advantage of virtualenv over venv is bundled pip, but
> stdlib python has that too, now…

Maybe, virtualenv has other advantages, some of which are additional tooling
written around it (virtualenvwrapper etc). It also has the benefit that it works
across multiple versions of Python, so if someone wants to create Python 2
and Python <3.4 virtual environments, they are likely to be using virtualenv
for consistency sake.


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Bug#750638: ITP: ndg-httpsclient -- enhanced HTTPS support for httplib and urllib2 using PyOpenSSL

[Donald Stufft]

On Jun 5, 2014, at 7:09 AM, Daniele Tricoli  wrote:

> Hello Julien,
> thanks for packaging ndg-httpsclient!
> 
> On Thursday 05 June 2014 12:26:22 Julien Cristau wrote:
>> My main interest is to be able to talk to websites using SNI with
>> scripts using python-requests.
> 
> Once in the archive I will also add ndg-httpsclient into python-requests' 
> Suggests.
> 
> Kind regards,
> 
> P.S. I'll do the same for python-urllib3:
> https://github.com/shazow/urllib3/pull/156
> 
> -- 
> Daniele Tricoli 'Eriol'
> http://mornie.org

You need pyasn1, pyopenssl, and ndg-httpsclient in order for the 
requests/urllib3 stuff to kick in.

It’d probably be a sane idea to use recommends, at least on Python 2.x since 
using that also
prevents CRIME and the like which Python 2.x is vulnerable to else wise IIRC.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Bug#750638: ITP: ndg-httpsclient -- enhanced HTTPS support for httplib and urllib2 using PyOpenSSL

[Donald Stufft]

On Jun 5, 2014, at 11:47 AM, Daniele Tricoli  wrote:

> Hello Donald,
> 
> On Thursday 05 June 2014 10:24:48 Donald Stufft wrote:
>> You need pyasn1, pyopenssl, and ndg-httpsclient in order for the
>> requests/urllib3 stuff to kick in.
> 
> Yes, of course: I was keeping an eye on all the needed packages.
> 
>> It’d probably be a sane idea to use recommends, at least on Python 2.x since
>> using that also prevents CRIME and the like which Python 2.x is vulnerable
>> to else wise IIRC.
> 
> Thanks for pointing this: for python-requests I will add to Recommends all of 
> the needed packages to ensure that SNI works as expected and to prevent CRIME.
> 
> For python3-requests do you think it's needed to also add them to Reccomends?
> Upstream issue 20994[¹] is still open, but Python3 support SNI, and ssl 
> compression can be disabled, as reported on the issue, using 
> OP_NO_COMPRESSION 
> (on python3 >= 3.3, but we have 3.4). I think use Suggests is fine in this 
> case.
> I will add a README.Debian to explain clearly all of it.
> 
> Cheers,
> 
> 
> [¹] http://bugs.python.org/issue20994
> 
> -- 
> Daniele Tricoli 'Eriol'
> http://mornie.org

Yea it shouldn’t matter on Python 3.x as the SSLContext stuff urllib3 will use 
to give good defaults there already.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Bug#750638: ITP: ndg-httpsclient -- enhanced HTTPS support for httplib and urllib2 using PyOpenSSL

[Donald Stufft]

On Jun 5, 2014, at 12:02 PM, Barry Warsaw  wrote:

> On Jun 05, 2014, at 11:52 AM, Donald Stufft wrote:
> 
>> Yea it shouldn’t matter on Python 3.x as the SSLContext stuff urllib3 will
>> use to give good defaults there already.
> 
> Does any of this impact our wheels for virtualenv/pyvenv (Py2 and Py3)?
> 
> -Barry

Not really. requests will opportunistically use those three libraries in order 
to
have better TLS on Python 2.x. ensurepip doesn’t include them since It’s
for Python 3.x only, and virtualenv doesn’t because they require a compiler
(well pyopenssl does).

If Debian wanted to make pip in virtualenv safer they could create wheels for
those 3 and install them into python 2.x virtualenvs by default (they can be
installed normally, they don’t require any magic). This would be Debian going
above the “standard” for what upstream or any other OS does afaik.

However I don’t think it’s really a big deal, most of the attacks on TLS affect
the confidentiality portions, however pip doesn’t really care too much about
that and currently relies on TLS mostly for the authenticity portion.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Packaging of suds-jurko (was: suds)

[Donald Stufft]

On Jul 1, 2014, at 5:41 PM, Barry Warsaw  wrote:

> On Jul 01, 2014, at 01:10 PM, Mathias Behrle wrote:
> 
>> The first tests on suds-jurko are looking very promising. I built the package
>> succesfully as a drop-in replacement for the current python-suds package. It
>> builds correctly for python2 and python3 with all tests. I tested part of the
>> functionality for python2, all was working well. The maintainer of suds-jurko
>> is very active and responsive.
> 
> Will a Python 3 compatible suds library allow us to make progress on #732644
> without rewriting bts to use REST+JSON ?
> 
>> 1) Can I drop in the suds-jurko fork into the current suds package as
>> proposed by Jordan?
> 
> Given that suds on PyPI hasn't been updated in almost 4 years, I think we can
> reasonably assume its upstream is defunct.  We had a sort of analogous
> situation with setuptools, but the distribute and setuptools upstreams did
> eventually merge back together.
> 
> A counter example might be oauth which was also abandoned upstream and for
> which a new upstream called oauthlib was released.  However, in that case, the
> replacement was *not* API compatible, so it made sense to make it a different
> Debian package.
> 
> I don't really have a strong opinion, as I can see both sides of the coin.
> You're *probably* safe just taking over the source package, but if you woke up
> tomorrow with an extra dose of paranoia, then you might favor a new source
> package, which also wouldn't be objectionable, albeit more work to transition
> dependencies.
> 
>> 2) If not 1) what would be the best alternative?
>> 
>> In this case I would plan
>> 
>> - a new python-suds-jurko package, conflicting with python-suds
>> - filing bugs to rdepends to use the new package
>> - removing the old package as soon as possible
> 
> Yep.  It's a bit ugly though (I don't like the -jurko blarg).  Oh well, do
> what you think is right.
> 
> -Barry

*Puts on PyPI Admin Hat*

Probably if suds-jurko or whatever is the unofficial “suds” that people should
be using then there is a good chance that PyPI would be willing to transfer
the name of suds to one of the forks. I’d have to talk to Richard to be sure
about that but it’s potentially an option.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Packaging of suds-jurko (was: suds)

[Donald Stufft]

On Jul 2, 2014, at 5:09 AM, Mathias Behrle  wrote:

>> *Puts on PyPI Admin Hat*
>> 
>> Probably if suds-jurko or whatever is the unofficial “suds” that people 
>> should
>> be using then there is a good chance that PyPI would be willing to transfer
>> the name of suds to one of the forks. I’d have to talk to Richard to be sure
>> about that but it’s potentially an option.
> 
> That indeed would be really great. Jurko just had the same idea and it would 
> be
> the cleanest solution.
> 
> What steps should be done to achieve this? Is it enough to point Jurko to post
> a request on http://sourceforge.net/p/pypi/support-requests/ ?

That or send an email to don...@python.org and rich...@python.org asking for
it, mention that I suggested it and that it’s been dead for 4 years and what 
have
you.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: 'deviations from upstream' wiki.debian.org/Python docs

[Donald Stufft]

I hope not, it’s way behind setuptools at this point.

On Jul 2, 2014, at 11:19 AM, Jurko Gospodnetić  wrote:

>  Hi all.
> 
>  Just wandering if the information from https://wiki.debian.org/Python about 
> 'Deviations from upstream' up to date. Not a Debian user myself, but the 
> following notes seem weird:
> 
>> The python-setuptools package installs the Distribute fork instead
>> of the standard setuptools.
>> 
>> The python-virtualenv also uses distribute by default, but can
>> enable classic setuptools with an optional switch.
> 
>  Do debian distributions still actually package `distribute` instead of 
> `setuptools`?
> 
>  Best regards,
>Jurko Gospodnetić
> 
> 
> -- 
> To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
> Archive: https://lists.debian.org/53b422f1.9080...@pke.hr
> 


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Building Python without SSLv3 breaks requests

[Donald Stufft]

Upstream would probably like the patch to fix things when sslv3 is disabled. I 
think shazow would merge it easily. 


> On Nov 19, 2014, at 6:20 PM, Barry Warsaw  wrote:
> 
>> On Nov 19, 2014, at 08:14 AM, Matthias Klose wrote:
>> 
>> I'll wait for the -12 results. I think it would be better to test for this
>> attribute first before using it.
> 
> I think this can be combined with Daniele's suggestion, such that the
> _openssl_versions dictionary is only populated with keys from the ssl module
> that exist.  It's a little more work, but istm the right thing to do.
> 
> Cheers,
> -Barry


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/0d0d8b0b-506d-4edb-9fed-35edd7574...@stufft.io

Re: Resources/best practices for making a .deb out of a virtualenv

[Donald Stufft]

> On Dec 31, 2014, at 2:21 PM, Tim Chase  wrote:
> 
> I've got a Python project where the dependencies are
> isolated in a virtualenv and would like to package it up as a .deb
> the purposes of sharing.  Normally I'd include the additional
> libraries as straight-forward dependencies of the Debian package
> itself, but some of the required package-versions conflict with the
> stock Stable or Installation packages (e.g. Django 1.4 in Stable vs.
> 1.6 or 1.7 which are more current versions).
> 
> Are there any good resources on how to package up a virtualenv for
> distribution so that it can use more recent libraries?
> 
> (I'm not currently subscribed to the mailing list, but will if that's
> easer than just CC'ing me or Reply-All'ing me).
> 

I’ve used https://github.com/spotify/dh-virtualenv in the past and it worked 
well.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/a9ef950f-ca8b-40e0-9c24-948cdbe2c...@stufft.io

Re: PyPI and debian/watch

[Donald Stufft]

> On Feb 4, 2015, at 3:05 AM, Ben Finney  wrote:
> 
> Tristan Seligmann  writes:
> 
>> The debian/watch file I wrote for python-nacl (which also verifies the
>> PGP signature) seems to work.
> 
> I can't get PGP signature retrieval to rowk (“uscan warning:
> pgpsigurlmangle option exists, but the upstream keyring does not exist”)
> even with your suggested pattern.
> 
> But I have also written a working uscan configuration::
> 
> opts="filenamemangle=s/\S+\/([^\/]+\.tar\.gz)#md5=[[:alnum:]]+$/$1/" \
>https://pypi.python.org/simple/python-daemon/ \
>\S+/python-daemon-(\S+)\.tar\.gz#md5=[[:alnum:]]+ \
>debian
> 
> 
> Barry Warsaw  writes:
> 
>> I'd love to be able to have something as simple as:
>> 
>> version=3
>> https://pypi.python.org/simple/mypkg/mypkg-(.*).tar.gz
>> 
>> which is close to what most packages probably use today, modulo the
>> base url path.
> 
> That would be great. But remember that the uscan documentation
> recommends a tighter matching pattern, so that would be::
> 
>version=3
>https://pypi.python.org/simple/mypkg/mypkg-(.+).tar.gz
> 
>> I filed a bug against pypa/warehouse so hopefully we can get something
>> better before Jessie is released (which is when I think there will be
>> more pressure for a better solution, since most packages won't be
>> updated during the freeze).
>> 
>> https://github.com/pypa/warehouse/issues/358
> 
> Thanks very much!
> 
> I'm not a fan of having it live at “…/uscan/” though. This is not
> specific to Debian, it's a sensible API design for all.
> 

If it gets implemented it'll live at /uscan/ because it exists primarily to
work around the deficiencies that exist in uscan (Particularly the dificulty
in ignoring url fragments). Everyone else should just use the URLs at /simple/
which most systems use with no problem because they can parse the URLs and
ignore the URL fragments (or use them for verifying the hash if need be).

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/fa9f43c5-64c5-4cff-a972-30d162099...@stufft.io

Re: PyPI and debian/watch

[Donald Stufft]

> On Feb 4, 2015, at 10:07 AM, Barry Warsaw  wrote:
> 
> On Feb 04, 2015, at 08:08 AM, Donald Stufft wrote:
> 
>> If it gets implemented it'll live at /uscan/ because it exists primarily to
>> work around the deficiencies that exist in uscan (Particularly the dificulty
>> in ignoring url fragments). Everyone else should just use the URLs at 
>> /simple/
>> which most systems use with no problem because they can parse the URLs and
>> ignore the URL fragments (or use them for verifying the hash if need be).
> 
> I'll just note that I've found the fragments inconvenient in other settings
> too.  They aren't very user friendly since (IMHO) they add noise that users
> cutting and pasting urls generally don't care about.  They also "feel" odd in
> that the md5 checksum doesn't fit what I think as a typical fragment.
> Traditionally, they are used to point to an anchor (sub-resource) within the
> parent resource.  That's not the case here.
> 
> http://en.wikipedia.org/wiki/Fragment_identifier
> 
> has this to say:
> 
> """
> Several proposals have been made for fragment identifiers for use with plain
> text documents (which cannot store anchor metadata), or to refer to locations
> within HTML documents in which the author has not used anchor tags:
> 
> As of September 2012 the Media Fragments URI 1.0 (basic) is a W3C
> Recommendation.[12]
> 
> The Python Package Index appends the MD5 hash of a file to the URL as a
> fragment identifier.[13] If MD5 were unbroken (it is a broken hash function),
> it could be used to ensure the integrity of the package.
> 
> https://pypi.python.org ... 
> zodbbrowser-0.3.1.tar.gz#md5=38dc89f294b24691d3f0d893ed3c119c
> """
> 
> So even without the uscan incompatibility (which is just one of the two
> factors leading to noisy d/watch file), I think there's some value in
> fragment-less URLs.  I understand the checksum isn't being used
> cryptographically here, but maybe thinking ahead to the use of more secure
> algorithms in the future can lead to a more flexible design:
> 
> Legacy (if it indeed needs to be kept for backward compatibility):
> 
> /simple/foo-x.y.z#md5=blah
> 
> then:
> 
> /simple/plain/foo-x.y.z
> /simple/sha256/foo-x.y.z#sha256=blah
> 

Long term PyPI is going to move away from trying to cram a bunch of information
into a hyperlink and relying on HTML parsing and instead is going to move the
installer APIs over to using something more suited to the task, most likely
JSON. At that point we'll be able to design the API to be more extendable in
this regards since we'll be able to do something like:

{
...,
hashes: {
"md5": "...",
"sha256": "...",
},
...
}

and the client can simply select which hash it wants to use. Long term the
/simple/ API on PyPI will exist only for legacy purposes so people still using
versions of pip, easy_install, etc that only support /simple/ will still be
able to access PyPI.

That doesn't really help uscan at all though since as far as I know uscan has
no ability to parse JSON.

As far as copy/pasting goes, the /simple/ API is an API, it's not designed to
be human consumable but consumable by software. The UI centric pages at /pypi
are the ones designed to be consumable by humans (Although currently PyPI puts
the hash there as well, however Warehouse (aka PyPI 2.0) does not).

The problem here really lies within uscan making assumptions about the
structure of URLs and the content of the HTML on those pages. From looking at
https://wiki.debian.org/debian/watch I'm guessing that it inherited those
assumptions from when FTP was the more common way to distribute files instead
of HTTP(S). That same page also mentions that qa.debian.org runs a number of
"redirectors" for sites like SourceForge and GitHub so perhaps a better answer
is for Debian QA to run a redirector for PyPI instead of PyPI implementing a
redundant API endpoint with a slightly different layout and HTML primarily for
Debian.

One note in that regard is that the /simple/ indexes don't include the .asc
files if someone has uploaded them however the old URLs that debian/watch used
did. If that is something that is needed we could easily add them to the
/simple/ pages.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/246c48b2-9f15-4827-aab4-b574f95a2...@stufft.io

Re: PyPI and debian/watch

[Donald Stufft]

> On Feb 4, 2015, at 11:20 AM, Barry Warsaw  wrote:
> 
> On Feb 04, 2015, at 10:53 AM, Donald Stufft wrote:
> 
>> That same page also mentions that qa.debian.org runs a number of
>> "redirectors" for sites like SourceForge and GitHub so perhaps a better
>> answer is for Debian QA to run a redirector for PyPI instead of PyPI
>> implementing a redundant API endpoint with a slightly different layout and
>> HTML primarily for Debian.
> 
> +1
> 
> Cheers,
> -Barry

Dunno the best way to give this to y'all but I wrote a thing:

https://github.com/dstufft/pypi-debian

I can transfer it on github or release it on PyPI or whatever. It shouldn't
really need any maintenance or anything but I'm probably not going to pay
attention to it if it does need any so someone else might want to actually
own it.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/363795e0-f96f-423f-b6a6-d3f3fbf76...@stufft.io

Re: PyPI and debian/watch

[Donald Stufft]

> On Feb 4, 2015, at 3:02 PM, Tianon Gravi  wrote:
> 
> On 4 February 2015 at 06:08, Donald Stufft  wrote:
>> If it gets implemented it'll live at /uscan/ because it exists primarily to
>> work around the deficiencies that exist in uscan (Particularly the dificulty
>> in ignoring url fragments).
> 
> This seems like we're building a workaround to a tool we could
> theoretically change. :(
> 
> "debian/watch" has a "version=3", which is presumably so that there
> can be a "version=4" when deficiencies are discovered -- wouldn't it
> be worthwhile to consider revbumping the watch format and updating
> uscan to have some improved support for edge cases like this?  I know
> uscan has some other open bugs too that could use some thought towards
> a more flexible format to handle cases like this.

We talked about this in #debian-python and there was concern that a new version
of uscan wouldn’t be in Jessie and then wouldn’t cover the people who need it
the most.

I don’t know if that’s true or not but I certainly think that uscan _should_
ignore anything that comes after a # (similarly to how it ignores anything that
comes after a ?). That would solve the largest problem, that the URL fragment
is hard to remove from the d/watch file. The other problem is that 
/simple//
has files located at /packages/ but I believe that’s not very hard to 
work
around.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/bc9910d4-1c34-4856-9f0f-b46a99da1...@stufft.io

Re: PyPI and debian/watch

[Donald Stufft]

> On Feb 4, 2015, at 4:32 PM, Stefano Rivera  wrote:
> 
> Hi Donald (2015.02.04_22:06:25_+0200)
>>> On 4 February 2015 at 06:08, Donald Stufft  wrote:
>>>> If it gets implemented it'll live at /uscan/ because it exists primarily to
>>>> work around the deficiencies that exist in uscan (Particularly the 
>>>> dificulty
>>>> in ignoring url fragments).
> 
> Would it be that hard to have fake directory listings on /simple/?
> I mean, surely keeping compatibility there is simpler than having a
> second endpoint just for Debian.

All the data that uscan needs is already on /simple/, you can make uscan work 
with
it. There is one major problem and one small problem:

1. Major: The /simple/ URLs all have a #md5= and it’s non trivial to 
write a
   d/watch file that ignores them and uscan doesn’t by default. You can do it 
but
   it’s ugly and prone to copy/paste bugs.

2. The URLs on /simple/ point to /packages/, so it requires the 2 arg form of
   d/watch instead of the single arg form.

So you can make uscan work right now with /simple/ (and a few people have) but 
#1
means that a few of the #debian-python people were not very happy with that 
solution.
I can’t remove/modify that hash without causing issues with pip/easy_install 
though.
Originally I was going to just make a /uscan/ that was /simple/ without the 
hash,
but instead I suggested to #debian-python that a redirector might be better and 
there
is now one at pypi.debian.net.


> 
>> We talked about this in #debian-python and there was concern that a new 
>> version
>> of uscan wouldn’t be in Jessie and then wouldn’t cover the people who need it
>> the most.
> 
> Who needs it the most? We could fix it in unstable and backports. The
> DEHS data on tracker.debian.org comes from quantz.debian.org. which is
> currently using devscripts from back ports.

No idea, I’m just repeating what folks said in #debian-python, I have no idea 
who
runs uscan and on what platforms. Between fixing uscan and having a redirector I
don’t have an opinion since neither one of those have an impact on what PyPI
does.

> 
> 
>> I don’t know if that’s true or not but I certainly think that uscan _should_
>> ignore anything that comes after a # (similarly to how it ignores anything 
>> that
>> comes after a ?).
> 
> Agreed.
> 
> SR
> 
> -- 
> Stefano Rivera
>  http://tumbleweed.org.za/
>  +1 415 683 3272
> 
> 
> -- 
> To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
> Archive: https://lists.debian.org/20150204213208.ga3...@bach.rivera.co.za
> 

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/c118b6b9-f56e-46be-9a80-c5e934c1f...@stufft.io

Re: Debian uscan redirector for PyPI (was: PyPI and debian/watch)

[Donald Stufft]

I'm on my phone currently but I think Barry is using it in the wheel package 
now. 


> On Feb 4, 2015, at 5:18 PM, Ben Finney  wrote:
> 
> Donald Stufft  writes:
> 
>> I suggested to #debian-python that a redirector might be better and
>> there is now one at pypi.debian.net.
> 
> Great! Can we please have (from you, or whoerver is best position to
> write it) a reference on how to use this redirector?
> 
> I don't know a good location; the Debian wiki doesn't have an obvious
> location, redirectors only seem to be mentioned briefly at
> https://wiki.debian.org/debian/watch#Uncommon_sites>.
> 
> -- 
> \   Moriarty: “Forty thousand million billion dollars? That money |
>  `\must be worth a fortune!” —The Goon Show, _The Sale of |
> _o__)   Manhattan_ |
> Ben Finney
> 
> 
> -- 
> To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
> Archive: https://lists.debian.org/8561bhgw2k.fsf...@benfinney.id.au
> 


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/dd57c9f2-bab9-41b3-ab51-70bd03fde...@stufft.io

Re: Debian uscan redirector for PyPI

[Donald Stufft]

> On Feb 5, 2015, at 6:14 PM, Ben Finney  wrote:
> 
> Ben Finney  writes:
> 
>> Donald Stufft  writes:
>> 
>>> I suggested to #debian-python that a redirector might be better and
>>> there is now one at pypi.debian.net.
> 
> Whe is the VCS for the code running that redirector?
> 
> I would like to suggest some patches. To whom should I submit those
> patches?
> 

I think Piotr.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/9bd7ebdd-612e-40a8-90e7-08b727e00...@stufft.io

Re: Fixing #744145, `pip install --user --upgrade requests` breaks pip

[Donald Stufft]

> On Feb 25, 2015, at 5:54 PM, Thomas Kluyver  wrote:
> 
> On 25 February 2015 at 14:40, Scott Kitterman  <mailto:deb...@kitterman.com>> wrote:
> Having re-read the bug, I don't see how this is anything other
> than user error in Debian.
> 
> I hope you don't mean that installing any of the packages pip relies on under 
> my home directory is 'user error'. The user shouldn't even need to know which 
> packages those are.
> 
> Another potential fix might be to run pip with Python's -s flag [1], so the 
> user site packages directory is not on sys.path. I don't know if that breaks 
> pip installing to that location, though. It needs to be on sys.path when pip 
> runs 'setup.py install', because that may rely on things installed to satisfy 
> setup_requires.
> 
> [1] https://docs.python.org/3/using/cmdline.html#cmdoption-s 
> <https://docs.python.org/3/using/cmdline.html#cmdoption-s>
> 
> Thomas


The -s flag would mean that pip would not be able to determine that something 
was installed already in the ``—user`` directory, because pip inspects the 
sys.path to determine what is installed already.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: PyCon BoF: Stretch goals for cPython, PyPy & CFFI

[Donald Stufft]

> On Apr 13, 2015, at 4:17 PM, Stefano Rivera  wrote:
> 
> Matthias and I are planning to have a Debian Python BoF at PyCon,
> tomorrow afternoon. I think lunch is 2pm, so 3pm?
> 
> Meet outside the cPython sprint room?
> 
> Matthias wants to discuss general stretch goals for Python in Debian.
> I want to make concrete plans for py3k packages that are compatible with
> multiple interpreters.
> 
> 

Does this mean PyPy too? It’d be great to have a (not just specific to Debian)
standard for how to mark a binary for a particular Python that gracefully 
handles
other interpreters too. Right now we have the de facto standard of binary, 
binaryX,
and binaryX.Y but that really only sanely handles CPython.


---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Bug#785275: ITP: python-ipaddress -- Backport of the ipaddress module from Python 3.3

[Donald Stufft]

> On May 21, 2015, at 9:50 PM, Scott Kitterman  wrote:
> 
> On Friday, May 22, 2015 12:34:02 AM Brian May wrote:
>> On Fri, 22 May 2015 at 07:14 Scott Kitterman  wrote:
>>> I was considering the idea of porting things from ipaddr to ipaddress for
>>> python2, but there's a lot more of that then there is for ipaddress (which
>>> is
>>> up to only two packages we know about).
>> 
>> As it is a goal to have everything Python3 compliant, and as Jessie no
>> longer has python3-ipaddr (it was in wheezy), everything is going to have
>> to support ipaddress for python3 anyway.
>> 
>> I don't see why we don't drop python-ipaddr, and replace it with
>> python-ipaddress.
> 
> I dropped python3-ipaddr once python3.3 was default since it comes with
> ipaddress.  For python3, everthing should use ipaddress.
> 
> At some point it might make sense to switch.  There's about 20 packages in the
> archive that use python-ipaddr versus none with one or two candidates using
> python-ipaddress in python2.  Porting two is easier than porting 20.
> 
> I'm also concerned that because of type processing differences (strings versus
> bytes and UTF-8 by default versus not) there is potential for subtle
> incompatibilities in the backported ipaddress.  I had complaints on an
> upstream project that uses ipaddress with python3 when it was available
> failing with ipaddress in python2.
> 
> Scott K
> 
> 
> --
> To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
> Archive: https://lists.debian.org/2512839.pZrC1BOzKj@kitterma-e6430
> 

I released pip 7 last night, as part of that I looked at switching from
ipaddress to ipaddr as a favor to Barry to try and help him get the new pip
into Debian. I've never used the ipaddr library before so I was flying blind
with it.

I'm not sure if I was looking at the wrong ipaddr or something but the
documentation I found was pretty crappy and looking at it I didn't see how to
actually use ipaddr the same way I was using ipaddress in pip. In addition to
that, the API doesn't really look the same as the ipaddress to me so it's going
to require some sort of compatability shim maintained in pip in order to be
able to run one module on Python 2 and another module on Python 3.

Given all of that and the fact I didn't feel like spending a bunch of time
going down a rathole to switch dependencies when we already had a perfectly
reasonable dependency that handled it I went ahead and just released pip 7 with
the ipaddress dependency still.

I'm not entirely sure why that's an unreasonable dependency here, it has a
different name than ipaddr does, at least in the PyPI modules, so it should be
entirely possible to have them both installed side by side without any
conflicts.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Dealing with flit -- a simplified packaging of python modules

[Donald Stufft]

On September 25, 2015 at 7:24:30 PM, Paul Wise (p...@debian.org) wrote:
> On Thu, Sep 24, 2015 at 1:17 PM, Thomas Kluyver wrote:
> 
> > That's my point ;-). From our upstream point of view, it's not a bug
> > that the distributions we put on PyPI contain generated/bundled files -
> > we do it that way deliberately, so that end users can install without
> > needing Javascript developer tools to build those files. If you want a
> > pure source tarball without generated files, that's available from
> > Github.
> 
> Why are end users using source packages instead of binary packages and
> then complaining that the source tarballs aren't ready-to-run binary
> packages?
> 

Because the way Python packaging currently is and historically has been, binary 
packages are not something that is widely available or viable.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Re: Dealing with flit -- a simplified packaging of python modules

[Donald Stufft]

On September 25, 2015 at 8:21:39 PM, Ben Finney (ben+deb...@benfinney.id.au) 
wrote:
> Donald Stufft writes:
>  
> > On September 25, 2015 at 7:24:30 PM, Paul Wise (p...@debian.org) wrote:
> > > Why are end users using source packages instead of binary packages
> > > and then complaining that the source tarballs aren't ready-to-run
> > > binary packages?
> >
> > Because the way Python packaging currently is and historically has
> > been, binary packages are not something that is widely available or
> > viable.
>  
> So, commendable effort has gone toward making that problem reduce,
> perhaps eventually to go away. Congratulations are deserved for getting
> us this far, and thanks for continuing to push for feasible binary
> distributions of Python packages on all supported OSen.
>  
> It seems reasonable, then, to also put effort toward making source
> distributions more targeted toward being bundles of the released
> *source* for the project version, and reducing the justification for
> bundling non-source files in the source distribution.

I think distutils/setuptools is to blame for a lot of this, they don't make it
easy to have generated files that aren't either compiled C code or py(c|o).
Ideally in the future distutils/setuptools will no longer be special (except as
a historical and legacy fallback) and we'll have something far more pluggable
in the future. This will ideally make it a lot easier and sane to have custom
build steps that create other generated files as part of the building of a
package.

>  
> Can we expect the value of “but people expect non-source stuff bundled
> in the source distrubution” to diminish, as some kind of objection to
> making proper source distributions?

It's certainly a goal of mine that we move as much of the upstream Python
packaging onto binary packages so that most people are not installing from
sdists or really, "installing" from an sdist being an outdated concept and
long term, the only thing we do with an sdist is turn it into a binary package,
even if that build step is taken on the end user's computer. I of course, can't
speak to everyone so it would not surprise me if some people simply will never
be willing to not distributed generated files as part of their sdist.

I suspect this to be a long road though, and certainly the nature of *nix makes
it harder on us, since it's hard to make a binary package that works for say,
all of Linux due to ABI problems and such. The more cases where ``pip install``
has end users pulling from a sdist, the more resistence I suspect you'll find
from authors to making sdist have more complicated build steps.

I suspect you'll have an easier time convinving authors of pure Python things
that include some generated files (translations, js, whatever) to rely entirely
on wheels than you will have convincing authors of Python things that include
some C or other compiled code that precludes universal(ish) Wheels.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Re: Dealing with flit -- a simplified packaging of python modules

[Donald Stufft]

On September 26, 2015 at 5:30:35 AM, Paul Wise (p...@debian.org) wrote:
> On Fri, 2015-09-25 at 19:25 -0400, Donald Stufft wrote:
> 
> > Because the way Python packaging currently is and historically has
> > been, binary packages are not something that is widely available or
> > viable.
> 
> Hmm, I thought eggs have been around for ages (seems about 8 years)?
> 

Eggs are problematic and hardly anyone publishes them.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Amend Debian Python Proposal to Include More Python Metadata?

[Donald Stufft]

Hello!

I'd like to suggest a change to the Debian Policy around Python packages that
will help enable the world of Python packaging to continue to progress forward.

First, a little bit of background:

At the Python level there are three metadata formats for Python packaging:

* The original, setuptools style .egg-info directories.
* The distutils style .egg-info *file* added to distutils at some point.
* The new and improved, wheel based .dist-info directories.

The presence of any of these files will signal to Python tools that a
particular distribution has been installed, however there are two fairly major
and important differences between the distutils style, and the other two.

1. The distutils style has no provisions to record what files on the system
   belong to the installed distribution, making it appear to Python tooling
   that there *are* no files other than the metadata file itself.

2. The distutils style has no provisions to include additional metadata files
   in the metadata, making it impossible to extend the python level metadata
   with additional files.

I have a series of improvements that I'd like to make to the packaging
toolchain that will sort of build on one another, but which is not going to
function correctly with the distutils style metadata and I'm hoping that I can
convince y'all to make it policy to default to generating one of the other two
kinds (with varying methods, more on that later).

Concretely the thing that this is blocking right now, is that with the newly
released pip 8.0 I tried to make it so that pip will refuse to uninstall a
project that is installed with distutils style metadata. This is because we do
not have any way to associate the actual .py (and others) files on disk with
the installed metadata, so all we have ever done is just simply remove the
metadata file, making it appear as if the item is uninstalled but leaving
behind all of the actual files. However I'm going to be reverting this in a
pip 8.0.1 release because it caused a decent amount of breakage amongst pip's
users, almost all of them people who are attempting to upgrade OS provided
packages using pip.

Now, I know that upgrading OS provided packages using pip is less than optimal
and I would greatly prefer that people did not do it (and I'm generally in
agreement) however if we don't enable people to do it, they'll just continue to
use an old version of pip and file bugs. It's a non starter for pip to make it
impossible to do.

In addition to the uninstall bit, it also means that things like pip show -f
return junk information for packages installed in this way.

Beyond just (eventually) enabling pip to disable uninstallations of distutils
based installs this will start to allow some other future changes that I think
will be more interesting to Debian. The uninstallation of distutils based
installs comes hand in hand with pip stomping all over already existing files
willy nilly because the way upgrading a project like that works is pip
uninstalls the metadata file that says X is installed, then it just overwrites
over any of the files that happen to be in it's way when it installs the newer
version. If we can remove the need for pip to gleefully overwrite files to
support these types of installed packages, then we can make it so pip will
hard fail if it attempts to overwrite an already existing file on disk.

An additional benefit here is that by switching to using the directory based
options, we can add additional metadata files to the installed projects, much
like the INSTALLER file from PEP376 (IIRC). This file will likely be the path
to having pip refuse to touch OS owned files all together without some sort of
--force flag to override the safety switch.

As far as compatibility goes, pip has always forced everything to be installed
using setuptools and as far as I am aware, there's no real fallout from doing
so. I think in 2016 it's pretty reasonable to assume that a Python project is
capable of being installed using setuptools instead of distutils.

So without getting into the actual *method* of doing this (of which there are
several different options with different trade offs) does this sound like
something at all that Debian would be interested in?

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Amend Debian Python Proposal to Include More Python Metadata?

[Donald Stufft]

> On Jan 21, 2016, at 9:32 AM, Paul Tagliamonte  wrote:
> 
> Hey Donald!
> 
> As far as using pip to do stuff system-wide, I wrote thoughts on 
> http://notes.pault.ag/debian-python <http://notes.pault.ag/debian-python>

I just want to be clear, that I totally agree with this. The reason I want to
ensure we can enable this isn't because I think doing so is a great idea and
everyone should be doing it. People *are* doing it and we (pip) have to be
careful about what we make impossible to do because a possible result of going
too break happy is people just never upgrade and things start to stagnate
again.

So for pip at the system level, my goal is try and push people away from it as
much as we can without flat out making it impossible *and* try to make it as
"safe" to do so as possible when people decide to do it anyways.

> 
> As for the rest of it, distutils is actually concretely shitty, and replacing 
> it with setuptools
> sounds sane. Sounds like a solid idea.
> 
> Perhaps we can add a Lintian warning for using distutils; it's really nasty 
> cruft,
> and it's going to bite the archive sooner rather than later.
> 

As far as options for how to implement this goes, I'm not sure what best fits
with Debian so I'll leave that up to y'all, but here are a few options that I
can think of.

A lintian check is probably a pretty good thing, I would guess that it would
specifically check for the existence of a .egg-info or .dist-info directory
(or check to the non-existence of a .egg-info file).

In pip the way we make this work is with a bit of a gross little thunk that
ensures setuptools is imported prior to setup.py being invoked which forces all
setup.py calls to use setuptools instead of distutils even if they import from
distutils.

Of course, many setup.py already unconditionally use setuptools, and many more
conditionally use it (typically falling back to distutils) so for many projects
it may be enough to just add python-setuptools to build-depends. For projects
that unconditionally use distutils, either the pip way or patching setup.py
might be a reasonable way forward.

A more controversial way that comes with possibly some extra benefits (which
Debian may not care about) is to use ``pip`` itself as the build tool rather
than directly invoking setup.py. In this pip would be responsible for mucking
around with the distutils/setuptools mess instead of that needing to be handled
by Debian specific tooling. You could even leverage this to remove some of the
runtime dependencies on python-pkg-resources and speed up some python using
software if you let pip build a wheel as part of the buld process [1][2][3]. I
suspect pip would need to grow some additional options to make it suitable for
you in this role, but I would be more than willing to help if this is desired
(or not, this proposal isn't about trying to ram pip down anyones throat!).


[1] To be clear, this isn't about adding wheels for everything to the archive,
it would only be using Wheel as an intermediate step in the build process.
[2] Console scripts generated by setuptools entry wrappers when installed by
setuptools have a dependency on pkg_resources at runtime, however those
same scripts, when installed by pip from a wheel, do not. This would enable
removing the runtime dependency on pkg_resources for anything that only has
it for console scripts.
[3] Import pkg_resources is not the fasest thing in the world, it has to scan
every directory on sys.path looking for things to activate and it comes
with a bunch of side effects. This happens implicitly for any project using
console scripts.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Amend Debian Python Proposal to Include More Python Metadata?

[Donald Stufft]

> On Jan 22, 2016, at 10:36 AM, Piotr Ożarowski  wrote:
> 
> to be honest, I still don't know what you're asking for. What do you
> want us to do? Patch 2.7's distutils?

Essentially, ensure that setuptools not distutils is used in a setup.py. There 
are generally three kinds of setup.py files:

1) Ones that use setuptools unconditionally - These ones you just leave alone, 
they are already correct and you should already have a build depends on 
python-setuptools.
2) Ones that conditionally use setuptools - These ones you just need to satisfy 
whatever condition the setup.py uses to enable setuptools. Typically this is 
just checking if setuptools is importable but sometimes they use environment 
variables or similar.
3) Ones that use distutils unconditionally - These ones you switch to making 
them use setuptools instead of distutils.

Now, that’s the high level overview, there’s an easier, more automatic way that 
could maybe just be added to pybuild (Not sure exactly how pybuild works) where 
instead of invoking the setup.py as:

python setup.py install (or whatever commands/args you’re passing)

You do it as (taken from pip):

python -c "import setuptools, 
tokenize;__file__='$PWD/setup.py';exec(compile(getattr(tokenize, 'open', 
open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))” install (or 
whatever commands/args you’re passing).

The thing is kind of ugly, but that will install things using setuptools (just 
like pip does) regardless of if it imports setuptools or distutils in it’s 
setup.py file.


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Amend Debian Python Proposal to Include More Python Metadata?

[Donald Stufft]

> On Jan 22, 2016, at 11:18 AM, Piotr Ożarowski  wrote:
> 
> let's make a deal. If you will make sure pip doesn't touch system files
> (and others will not crucify me for this) - I will make sure pybuild
> uses above line (if setuptools is not detected in setup.py but is listed
> in Build-Depends).

Regardless of what happens in this thread, pip is going to stop mucking around
in files that are owned by some other tool without some sort of opt-in/--force
style flag *and* we're going to be restructuring things to try and guide people
away from using pip outside of a virtual environment or through the user of
--user as well.

The main reason why that hasn't happened yet isn't because I'm holding those
things hostage, it's because I'm one person and my time is spread amongst
several really important tools and I have to prioritize things in accordance to
how bad it's on fire. In addition, pip is a cross platform program and we have
to ensure that whatever we do works in a cross platform manner. You mentioned
/usr/local/.../site-packages/ before, but plenty of systems don't have a
/usr/local/.../site-packages/ that their Python respects because that is a
Debian specific patch, so if we refused to install to /usr/.../site-packages/
we'd break installing at the system level on all platforms but Debian derived
ones.

What we need is a cross platform way to determine if a particular installed
Python package is "owned" by the system or if it's something we're allowed to
modify. That's going to require some effort to work out the exact best way,
but in my head the way forward on that is to use the metadata directory (the
one I'm asking y'all to start using) and add an additional piece of metadata
inside of it that just says "Hey, This is a system install" that we can inspect
and then take additional logic (like refusing to touch it without a --force)
based on that. The change I'm asking for here helps make that possible (among
other things).

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Amend Debian Python Proposal to Include More Python Metadata?

[Donald Stufft]

> On Jan 22, 2016, at 11:51 AM, Scott Kitterman  wrote:
> 
> On Friday, January 22, 2016 10:54:54 AM Donald Stufft wrote:
>>> On Jan 22, 2016, at 10:36 AM, Piotr Ożarowski  wrote:
>>> 
>>> to be honest, I still don't know what you're asking for. What do you
>>> want us to do? Patch 2.7's distutils?
>> 
>> Essentially, ensure that setuptools not distutils is used in a setup.py.
>> There are generally three kinds of setup.py files:
>> 
>> 1) Ones that use setuptools unconditionally - These ones you just leave
>> alone, they are already correct and you should already have a build depends
>> on python-setuptools. 2) Ones that conditionally use setuptools - These
>> ones you just need to satisfy whatever condition the setup.py uses to
>> enable setuptools. Typically this is just checking if setuptools is
>> importable but sometimes they use environment variables or similar. 3) Ones
>> that use distutils unconditionally - These ones you switch to making them
>> use setuptools instead of distutils.
>> 
>> Now, that’s the high level overview, there’s an easier, more automatic way
>> that could maybe just be added to pybuild (Not sure exactly how pybuild
>> works) where instead of invoking the setup.py as:
>> 
>>python setup.py install (or whatever commands/args you’re passing)
>> 
>> You do it as (taken from pip):
>> 
>>python -c "import setuptools,
>> tokenize;__file__='$PWD/setup.py';exec(compile(getattr(tokenize, 'open',
>> open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))” install
>> (or whatever commands/args you’re passing).
>> 
>> The thing is kind of ugly, but that will install things using setuptools
>> (just like pip does) regardless of if it imports setuptools or distutils in
>> it’s setup.py file.
> 
> I tried this and it works, but what's the big deal?
> 
> It provides a PKG-INFO file that has identical content to the old egg-info 
> file,
> an empty dependency links file, and a top_level.txt file with the one line in
> it.
> 
> Why is this better (I'm not a huge upstream developer of Python stuff, but I 
> do
> do some and I don't see what this gets me)?
> 
> Scott K


This a fair question:

1) distutils is effectively frozen and all improvements to the build system
   are either going to be inside of setuptools, or will be opt in build systems
   that (probably) won't rely on setup.py at all. This includes newer metadata
   formats and the like as well.

2) That top_level.txt is an important file, without it we have no idea that
   $SITE-PACKAGES/requests is in any way associated with the metadata file
   requests-2.9.1.egg-info, that top_level.txt tells us what the top level
   import names are for a particular project.

3) It slipped my mind that you have to pass an additional flag to setuptools
   right now to get the full file list (pip passes that flag unconditionally)
   however I'm going to poke setuptools to see about getting them to add the
   record file unconditionally to the .egg-info directory so it doesn't
   require the --record flag. (Although Debian could add it earlier if they
   wanted, but it's fine to wait for setuptools to change here).

4) Switching it to a directory allows us to add additional files/information to
   the Python level metadata, like the sort of divergent thread about getting
   pip to stop mucking with OS owned files, the most likely path forward is to
   drop an INSTALLER file inside the .egg-info directory that has some special
   value ("system"? "os"? Not sure) which pip will take as a signal that it
   really should not touch.

5) In pip, because of #2 and #3 a regular distutils installed package (which
   gives you an .egg-info file, not directory) we have now way when we're told
   to install a package named "foo" to associate that with any files on disk.
   This means that we're currently lying when we tell a user we've uninstalled
   a distutils installed package, we just remove the .egg-info file but leave
   all the other files behind. We've deprecated the ability to do this, and we
   tried to remove it all together in the recent pip 8.0 release, but we had to
   roll it back because it caused too much breakage, largely due to OS packages
   that were using distutils installed metadata.

   On the surface, it may seem counterproductive for Debian to help pip get
   better metadata about what files belong to a package so that we can
   uninstall it, but we're going to solve the "people modify system files with
   pip when they shouldn't do that" more directly and there exist use cases
   wh

Re: Amend Debian Python Proposal to Include More Python Metadata?

[Donald Stufft]

> On Jan 22, 2016, at 5:30 PM, Barry Warsaw  wrote:
> 
> Hey Donald, thanks for starting this conversation.  I for one am super
> appreciative of all the consideration you give for Debian's little slice of
> the world.
> 
> There's a lot to unpack in this thread, and I'm a little under the weather[1],
> so hopefully this makes sense.
> 
> Big +1 for recording the files that get installed via the .egg-info/RECORD
> file.  As you know, I've been working on dirtbike, which is a "rewheeling"
> tool to turn an installed package into a .whl.  While Debian tightly controls
> via policy the set of wheels we'll allow into the archive, dirtbike has code
> for parsing the RECORD file.  Unfortunately this is never exercised in
> practice because we don't have RECORD files - at least not for the packages we
> care about[2].
> 
> Big +1 for using setuptools everywhere.  By my count, of the packages that I
> happen to randomly have installed on my Ubuntu 16.04 system, I have 67
> .egg-info files and 113 .egg-info directories.  I'd rather have .egg-info
> directories everywhere.
> 
> +1 for a lintian warning if distutils is used.  I guess I'm +0 on forcing that
> through pybuild because it'll be unobvious and mysterious, and kind of lets
> upstreams off the hook.  I'd mildly prefer to patch packages that use
> distutils because that's much more discoverable, but I can appreciate that
> that's a lot of work we'd be imposing on maintainers, so I won't argue this
> too much (other than to say that if pybuild forces it, let's definitely
> document this in its manpage!).
> 
> On Jan 22, 2016, at 12:40 PM, Scott Kitterman wrote:
> 
>> Currently --record includes the .pyc files which is both unneeded and bad.
>> Before this gets added either in setuptools or by us, this needed to be
>> fixed.
> 
> +1 for (I think) another reason than has already been discussed.  We won't be
> generating .egg-info directories on the end-user's machine, but instead the
> machine the package is built on.  That could be a maintainer's own system or a
> central build machine depending on various conditions.  But because the pycs
> are generated on the end-user's machine, we actually don't know what pycs will
> be generated when the debs are installed, so the egg-info/RECORD file *can't*
> contain them, at least not accurately.

FWIW It appears that if you do setup.py install —no-compile to tell distutils
not to compile the .pyc during .deb build time, I believe that as is it will
just omit the .pyc files. Not sure if that’s workable for Debian or not.

> 
> On Jan 22, 2016, at 11:54 AM, Donald Stufft wrote:
> 
>> Regardless of what happens in this thread, pip is going to stop mucking
>> around in files that are owned by some other tool without some sort of
>> opt-in/--force style flag *and* we're going to be restructuring things to try
>> and guide people away from using pip outside of a virtual environment or
>> through the user of --user as well.
> 
> Of course, I'd still like --user to be the default[3].  I think that's still
> the eventual goal for pip, but isn't yet implemented because $priorities.

Yes, still a goal (it’s not as simple as —user will be the default, but that’s
the outcome for a non root account in a system Python).

> 
>> A more controversial way that comes with possibly some extra benefits (which
>> Debian may not care about) is to use ``pip`` itself as the build tool rather
>> than directly invoking setup.py. In this pip would be responsible for mucking
>> around with the distutils/setuptools mess instead of that needing to be
>> handled by Debian specific tooling.
> 
> I'd like to better understand how this would work.  IIUC, the Fedora ecosystem
> is making or already has made this switch, but I don't know how it works.
> Obviously, we don't want to install wheels into 
> /usr/lib/python3/dist-packages.

I haven’t actually personally seen if they’ve done it, but the Fedora maintainer
of the python-pip told me they were doing it, and they had a patch to pip to
add a feature they needed for it.

Forget that pip can fetch files from PyPI and install them for a moment and
consider the command ``pip install .``. Fundamentally this is similar to the
command ``make install`` right? It builds the files and then installs them
into the final location. As I understand it, the standard way for a project
like Debian to install a project using a “regular” Makefile install like that
is to do ``make DESTDIR=/tmp/some-tempdir install`` so that it invokes the
build system, sets all the paths in the resulting build system to the final
destination, but then right b

Re: Amend Debian Python Proposal to Include More Python Metadata?

[Donald Stufft]

> On Jan 22, 2016, at 6:02 PM, Barry Warsaw  wrote:
> 
> On Jan 22, 2016, at 05:50 PM, Donald Stufft wrote:
> 
>> Forget that pip can fetch files from PyPI and install them for a moment and
>> consider the command ``pip install .``. Fundamentally this is similar to the
>> command ``make install`` right?
> 
> Please remind me what the long term plan for this is.  Let's say it's 2021 and
> you and your amazing team of dozens of distutil-sig developers have been
> cranking away the whole time.
> 
> What tool will be used to build Python packages?

Whatever people want! At least, that’s my hope. Ideally we get away from
the idea that any particular build system is the right build system (because
the needs of a pure python thing like six.py is vastly different from Numpy
which needs Fortran and other crazy things).

My end goal is, as long as your build system can consume a sdist (or maybe
a repository too, still up in the air, but some kind of source input) and
produce a wheel (whether this is actually a zipped up wheel file or a properly
formatted directory is also still up in the air). Then you have a good enough
build system and that is all it needs to do, it doesn’t need to also implement
installing and what not, that’s some other tool’s job.

> 
> What tool will be used to upload (built/source?) Python packages to
> WarehouseNG?

Either twine, or we’ll fold that functionality into pip.

> 
> What tool will be used to download Python packages from WarehouseNG?

Whatever tool you want! We’ll have defined formats so as long as your tool
speaks those formats it can install them. In the case of already “compiled”
(pure Python or not) binary .whl files it can just download them, unzip,
move into the correct location + install deps and such. For source files
(in a not-yet thought out or done sdist 2.0 thing) it can download the
sdist 2.0 file, install the build dependencies, invoke the build tool for
this package (from above), get the wheel it produced, and then install
that as if it had downloaded it from PyPI.

> 
> What tool will be used to install packages from WarehouseNG into an importable
> location?

Same answer here.

> 
> Cheers,
> -Barry


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Amend Debian Python Proposal to Include More Python Metadata?

[Donald Stufft]

> On Jan 22, 2016, at 6:04 PM, Scott Kitterman  wrote:
> 
> On Friday, January 22, 2016 05:50:13 PM Donald Stufft wrote:
> ...
>> We already have an option like this, the —root option which will just append
>> a different prefix to all of the installation paths. So essentially instead
>> of invoking ``python setup.py install —root /tmp/something/`` which is what
>> I think you’re doing now, you’d do ``pip install —root /tmp/something/
>> —no-deps .`` and it’d just work similarly to what you have now, except pip
>> would be responsible for interacting with the Python build system.
> ...
> 
> Why would pip interacting with the Python build system instead of setuptools
> be better?
> 
> Scott K

setuptools *is* the build system right now (or distutils if you’re still using
that). So the benefit to Debian right now would be: We maintain the
distutils/setuptools hacks to make them suck less so you don’t have to!

Longer term, we want to enable people to go hog wild and invent their own
build systems instead of trying to do it all one size fits all with everyone
miserable because it doesn’t solve anyones problems nicely. At that point the
benefit to Debian becomes that instead of having to implement the standard
build tool interface that these tools implement (Which will most likely be
geared towards produces wheels, not towards putting files on disk in an
installed location) Debian can just let us maintain that bit of code too.

Of course, if Debian would prefer to interact directly with these tools, it’s
totally fine to do that. The whole point of us doing this work and trying to
define formats and APIs and not bless implementations is so anyone who wants
to can slot in their own tool in the process instead of being forced to use
some blessed tool.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Amend Debian Python Proposal to Include More Python Metadata?

[Donald Stufft]

> On Jan 22, 2016, at 7:18 PM, Scott Kitterman  wrote:
> 
> The Zen of Python says, among other things, "There should be one-- and 
> preferably only one --obvious way to do it".  Build systems seem to me like a 
> great place to apply that.


We have a sliding scale of complexity in what a project needs from it’s build 
system that generally breaks down into a few major classes of projects:

1) Projects that are pure python, single source, whose only real “compilation” 
is shuffling files to the correct location.
2) Projects that are pure python, but require some sort of generation step as 
part of the build process (2to3, etc).
3) Projects that have some basic C code that needs to be compiled, but which 
doesn’t link against anything special besides Python itself.
4) Projects that have some basic C code that needs to be compiled, but which 
links against other libraries like OpenSSL, libpq, etc.
5) Projects that have some basic C code, that needs to be compiled, that links 
against other libraries, and needs to be able to conditionally link against 
different libraries based on the capabilities and what is available in the 
system.
6) Extremely complex projects that need to link against many different 
libraries, possibly hard to build, possibly not C (e.g., Numpy with it’s blas 
libraries, Fortran, etc).

The problem with a one size fits all solution is that it’s very difficult to 
actually cover all of these cases in a way that is not horrible to actually use 
for each particular case. For an example, there is currently a build system 
called flit which doesn’t support anything but building straight to wheels 
(because we don’t have any sort of sdist 2.0 or anything yet). It doesn’t 
attempt to solve anything but the first class of users up there, and because of 
that it is able to create a very simple and easy to use packaging experience 
for authors, you just add a __version__ = “the version” to the top level of 
your package, and then drop in a flit.ini that looks something like:

[metadata]
module=foobar
author=Sir Robin
author-email=ro...@camelot.uk
home-page=http://github.com/sirrobin/foobar

# If you want command line scripts, this is how to declare them.
# If not, you can leave this section out completely.
[scripts]
# foobar:main means the script will do: from foobar import main; main()
foobar=foobar:main

And that’s it. You’re done. From there flit can do the rest of the work for you 
because it didn’t need to concern itself with trying to work on anything 
complex.

In addition to the above types of problems, you also have other things like 
what the “source of truth” is for your metadata. A common thing that people 
want is to be able to not have to duplicate their version in multiple locations 
(sometimes that even extends to the tags in the version control), however it’s 
not currently possible to do that in a particularly easy way. You have systems 
like pbr, setuptools_scm, versioner, etc that all do it but which all rely on 
some level of terribleness to deal with the sort of weird inverted control flow 
we have.

Once you get to all of the possible options for things people reasonably want 
to do, you quickly end up in a place where the only reasonable solution is the 
full complexity of a programming language like what we have in setup.py. 
However, that has it’s own problems which we’ve discovered over two decades of 
doing that :) It tends to end up with people doing badly tested adhoc code that 
they cargo cult from project to project and when there’s a problem there ends 
up being very little understanding why some code did that since it had been 
copy/pasted around so much. Pip has to go to a lot of effort to try and work 
around common mistakes people make in their setup.py files (like, depending on 
it being invoked with the project root as CURDIR) which most people will just 
never notice their slightly broken setup.py.

So basically, by allowing multiple build systems we will enable a world where 
your bog standard pure python project can get an extremely simple tool/ux and 
projects with crazy hard requirements like Numpy can get something more 
complex, without the two groups of users fighting each other over simplicity + 
limited vs complexity + powerful.

In the end, I think it’s likely we’ll end up with 2-3 really popular build 
tools, one that is for the complex projects, one that is for the simpler 
projects with some basic C needs, and *maybe* one that is for pure python (but 
that may be able to be handled by the basic C needs one) though there will be a 
long tail I’m sure.

-----
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: New, updated pip coming

[Donald Stufft]

> On Jan 29, 2016, at 5:33 PM, Robert Collins  wrote:
> 
> I am confused. Here's my understanding of things...
> 
> - Pip doesn't need wheels at all - its vendoring technique doesn't use wheels.
> - Virtualenv needs to install pip, wheel, setuptools when it makes a
> new environment, and it does that by some oogly code
> - venv might do the same thing, but I haven't check its actual implementation
> - the pip-whl package was used to give virtualenv a thing to use to
> install pip when making a virtualenv
> 
> If that aligns with your understanding, I'll put it down to the prose
> you wrote rather than us actually having a different worldview; OTOH
> if it doesn't, I'd like to find out where I'm wrong, so as to improve
> my understanding.
> 
> With my understanding in mind - Can I just check something?
> 
> If a new requests package is built, uploaded to the archive and
> apt-get installed on my machine, and I then do:
> virtualenv test
> . test/bin/activate
> pip install foobar
> ^--- what version of requests will be used by this in-virtualenv
> invocation of pip?

Debian doesn’t want to utilize the bundled libraries in pip for various
reasons (some I agree with, some I don't, but suffice it to say it's a
requirement) but there is recognition that pip is a bit special here and can't
just be debundled in the standard way.

In conjunction with Barry, pip has some basic support for debundling by
deleting the contents of pip/_vendor/ except for pip/_vendor/__init__.py,
creating wheel files for all of pip's dependencies, and then making a small
patch to pip/_vendor/__init__.py to turn on the debundling feature and tell it
where those wheel files live. Pip will then add all of those wheel files to
the front of sys.path.

This gives us a few things:

* Pip will *always* use the versions of those libraries from the wheels,
  regardless of what is installed. This makes it so you can't break yourself
  with ``pip install requests`` in any environment (virtual or otherwise).

* Debian gets to have all of the code in pip come from the correct source
  packages.

These wheel files that represent pip's dependencies are generated when
python-pip is being built from the versions of the packages in Debian's
repositories and the python-pip package has a Built-Using header to indicate
which dependencies have been rebundled inside of pip through this process. This
has essentially recreated static linking but for Python. It is my understanding
that all of the wheels for pip's dependencies will exist inside of the
python-pip-whl .deb.

On top of that, virtualenv and venv will both need wheels that it needs to
install for pip, setuptools, and in the case of virtualenv, wheel. I'm not sure
what the plan is for those.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: /usr/bin/python2 shebangs

[Donald Stufft]

> On Nov 1, 2016, at 6:03 PM, Scott Kitterman  wrote:
> 
> Even after python2.7 is removed from Debian, there will still be users who 
> keep a local copy because they couldn't migrate things due to $REASONS.  Re-
> using /usr/bin/python for a python3 version won't accomplish anything more 
> than gratuitously break such setups.


/usr/bin/python3 being Python 4.x is a bit weird though, and it’s likely that 
Python 4.x is not going to be another break the world release.

—
Donald Stufft

Re: pip for stretch

[Donald Stufft]

> 
> On Nov 21, 2016, at 6:33 PM, Barry Warsaw  wrote:
> 
> I have not started to look at what if anything needs to be done to transition
> to pip 9, but if you have a strong opinion one way or the other, please weigh
> in.
> 


As one might expect, I would prefer it if folks got 9.0.1 as quickly as 
possible. In particular the feature that makes it easier for upstreams to drop
Python 2 support is one that is really only effective when people can consider
pip 9 a "minimum" version of pip to support. Getting it into the hands of folks
as quickly as possible would be a big boon to that.

—
Donald Stufft

Re: GnuPG signatures on PyPI: why so few?

[Donald Stufft]

> On Mar 11, 2017, at 9:23 PM, Brian May  wrote:
> 
> Ben Finney  writes:
> 
>> However, this only works if upstream releases are actually accompanied
>> by a valid GnuPG signature each time. The PyPI infrastructure supports
>> this; why isn't it more widely encouraged?
> 
> One reason I have found for myself: I can forget to sign the package
> when uploading to PyPI, and PyPI doesn't let you make any changes after
> the package is uploaded without changing the version (including adding
> signature file). There is a long running bug report on this, it is not
> going to get fixed (TLDR it is not a bug, it is a design feature to
> allow for caching).

I think there is no reason we *couldn’t* allow uploads of signatures after the 
fact, although it sort of makes an already tenuous security claim even more 
tenuous I think. Since distutils/setuptools don’t have reproducible “builds” of 
sdists if you wanted to after the fact upload a signature your choices are:

* Hope you still have the original files you uploaded still laying around, sign 
*them* and then upload just the signature.
* Download the files, open them up and inspect all of the files to ensure they 
match the correct state.
* Download the files, blindly sign them.

Of those options, the first one is no more or less “secure” than uploading 
alongside the original file, but the problem is that it’s impossible to 
differentiate that from the second two, and that could genuinely be worse (for 
example, compromised PyPI gives author a bad file, they sign it and upload the 
signature thus making everyone feel it is a “safe” file). Both the second 
options are kind of bad because it is incredibly easy to smuggle something bad 
into a tarball that a human being would miss.

The larger reason why I personally haven’t implemented it is because I believe 
the GPG signatures on PyPI are largely useless from a practical standpoint and 
I’m not really interested in doing anything to improve them and I would 
personally much rather just disable them completely.


> 
> Maybe there is some way of turning signatures on by default, so I don't
> have to remember for every upload, if so, I haven't been able to work it
> out yet however.

Um, I think you can set something in setup.cfg for ``setup.py upload``. I don’t 
think there is anything like that for Twine. I can’t speak for Ian but I don’t 
personally see anything inherently wrong with adding a environment variable or 
config option to twine that allows it to always sign by default. Ian may feel 
differently though!

—
Donald Stufft

Re: GnuPG signatures on PyPI: why so few?

[Donald Stufft]

> On Mar 12, 2017, at 10:35 PM, Paul Wise  wrote:
> 
> On Mon, Mar 13, 2017 at 4:28 AM, Jeremy Stanley wrote:
> 
>> upload them to PyPI since the authors of the coming Warehouse
>> replacement for the current CheeseShop PyPI have already indicated
>> that they intend to drop support for signatures entirely.
> 
> Did they give any reasoning for this decision?
> 


https://mail.python.org/pipermail/distutils-sig/2016-May/028933.html 
<https://mail.python.org/pipermail/distutils-sig/2016-May/028933.html>

—
Donald Stufft

Re: the new PyPI, coming next month

[Donald Stufft]

> On Mar 31, 2018, at 11:23 PM, Scott Kitterman  wrote:
> 
> What replaces gpg for ensuring integrity of the uploaded code?


To be clear, PGP signatures can still be uploaded and they are still available 
for download, they just don’t appear in the UI anymore. Longer term I’d *like* 
to get rid of PGP signatures, because I think their value here is actually 
pretty low. In that case they’d be replaced with TUF, but that’s a longer term 
project.

Re: the new PyPI, coming next month

[Donald Stufft]

> On Apr 1, 2018, at 2:27 AM, Dominik George  wrote:
> 
> Hi,
> 
>> To be clear, PGP signatures can still be uploaded and they are still
>> available for download, they just don’t appear in the UI anymore.
> 
> So, what does the pypi.debian.net redirector use for uscan?  I imagine it
> used to scrape the website.  Can it be changed to use the JSON API?

The original PoC I wrote used the JSON API, but I don’t think what’s being 
deployed is descendant from my PoC so I have no idea what it uses, but if it’s 
not using the JSON API then yes it can be.

> 
>> Longer term I’d *like* to get rid of PGP signatures, because I think
>> their value here is actually pretty low.
> 
> I partially share this opinion, but that's a question to be discusses with
> the Debian policy people in general.  While checking a GPG signature on the
> source tarball in general is a good idea, I am afraid some developers just
> drop any key they find on first glance into the package and are done with
> it, which actually provides nothing but a false sense of safety.
> 
>> In that case they’d be replaced with TUF, but that’s a longer term
>> project.
> 
> That one?: https://github.com/theupdateframework/tuf 
> 


Yes.


> 
> Well, I can only say *please* do not remove the possibility to upload signed
> source tarballs, but leave that to the developers!
> 
> -nik
> 
> --
> PGP-Fingerprint: 3C9D 54A4 7575 C026 FB17  FD26 B79A 3C16 A0C4 F296
> 
> Dominik George · Hundeshagenstr. 26 · 53225 Bonn
> Phone: +49 228 92934581 · https://www.dominik-george.de/
> 
> Teckids e.V. · FrOSCon e.V. · Debian Developer
> 
> LPIC-3 Linux Enterprise Professional (Security)



signature.asc
Description: Message signed with OpenPGP

Re: [RFC] DPT Policy: Canonise recommendation against PyPi-provided upstream source tarballs

[Donald Stufft]

File names on PyPI are write once. Once a specific file name has been used it 
can never be used again (even if the entire project was deleted and recreated). 

Projects can delete uploaded files (and as mentioned they can be yanked, but 
yanking is just extra metadata beside the file), but file content can never 
change, only be removed. 

Sent from my iPhone

> On Jun 25, 2021, at 11:47 PM, Brian Thompson  wrote:
> 
> On Fri, Jun 25, 2021 at 07:01:39PM -0400, Nicholas D Steeves wrote:
>> Does PyPi provide immutable releases?
> 
> From experience, I can tell you that yes, releases cannot be overwritten,
> but they can be "yanked".  Pypi states that a yanked release is:
> 
>  "A release that is always ignored by an installer, unless it is the
>  only release that matches a version specifier (using either '==' or
>  '===)."
> 
> -- 
> Best regards,
> 
> Brian T

Re: can pip be made using local Debian packages for any dependencies

[Donald Stufft]

I'm pretty sure that most if not all debian packages already ship the required 
information for pip to see them as installed, and if they are installed and 
they satisfy the dependency constraints that pip has for those projects, then 
they'll be used.

The question of having pip automatically install a debian package instead of 
using PyPI basically comes down to a few problems:

- pip doesn't know how to invoke apt (and yum, and Conda, and everything else).
- how do we translate "foo" on PyPI to the equivalent package in Debian (and 
Fedora, and Conda, etc)
- how do we handle virtual environments?

Nobody has ever come up with a particularly good answer to those problems, so 
pip upstream hasn't been able to even consider doing something like that.

Of course Debian could patch it's copy of pip to do that, that would make some 
of those questions easier, but not all of them, and would drastically diverge 
it's behavior from the "expected" behavior, which makes it a rough idea in 
general for Debian to diverge that drastically.
On 2/12/2023 3:18:55 AM, Ian Norton  wrote:
https://packaging.python.org/en/latest/specifications/recording-installed-packages/
defines the python spec where a package such as pyparsing would create
a tree of files under:
site-packages/pyparsing-3.0.9-dist-info/ including RECORD which is
essentially a sha256-based manifest of files and some others.

On Sun, 12 Feb 2023 at 08:12, Ian Norton wrote:
>
> You've made me wonder if it would be feasible to have a debian-centric
> tool that populates .dist-info from debs?
>
> On Sun, 12 Feb 2023 at 08:05, Ian Norton wrote:
> >
> > I requested this kind of thing from the pip folks as
> > https://github.com/pypa/pip/issues/11644 and others have requested
> > similar, such as https://github.com/pypa/pip/issues/11607
> >
> > On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon wrote:
> > >
> > > Hey.
> > >
> > > I hope this is not too off topic.
> > >
> > > As far as I understand, dh-python, when building packages somehow
> > > automatically uses the Debian package names and even prevents e.g.
> > > setuptools from downloading any dependencies by setting a (hopefully
> > > not running) proxy.
> > >
> > >
> > > I wondered whether it's possible to make tools like pip and setuptools
> > > directly use the Debian python packages when resolving dependencies.
> > >
> > > The main motivation are security constraints, so I had to configure
> > > pip so that it cannot just download packages from PyPI (which is
> > > rather easy, simply setting no-index in pip.conf).
> > >
> > > But then of course it also fails to e.g. do an editable install of a
> > > locally developed package, when it tries to resolve the dependencies.
> > >
> > > So I wondered whether it's possible to prevent pip from downloading
> > > any remote stuff, while still resolving dependencies (respectively
> > > consider them as being resolved) *if* the package is locally installed
> > > from the Debian archive?
> > > (If a dependency isn't installed from a package it may of course fail.)
> > >
> > >
> > > Thanks,
> > > Philippe.
> > >
> > > PS: Please keep me CCed.
> > >


[88f0dcef-4969-48af-98f1-0c69549b4875]

Re: python devs complaining about debian packaging

[Donald Stufft]

I happen to be subscribed here, so figured I'd comment :)

FWIW I think the way the discussions are going... really in both locations.. is 
needlessly taking shots at each other.

I've commented on discuss.python.org, but figured I'd repeat myself here.

I think the way these discussions devolve into each "side" taking shots at each 
other and arguing over nonsense that doesn't matter does everyone a disservice. 

The facts of the matter, as far as I can tell:

- Distributors ship Python and a number of people find great value in that and 
it works great or at least fine for them.
- A number of users have needs or wants that are not well served by the Python 
that distributors ship, for one reason or another.

People in either of those groups sniping at each other for wanting the "wrong" 
things is completely unproductive. 

It's probably "fine" if Python.org wants to ship a linux binary, I suspect 
it'll have very little impact on distributors of Python (and might even make 
things better-- since some of the problems flow from an impedance mismatch, and 
it provides a different escape hatch to point people towards if the distro 
Python doesn't suit their needs).

On 5/26/2024 9:39:25 AM, Stefano Rivera  wrote:
Hi Ian (2024.05.26_01:33:09_+)
> I am puzzled about some of the responses there, how can anyone expect to
> randomly update packages on the system using pip and not have it go wrong
> on any distribution? This is why things like pipenv exist.

People don't understand that stuff until they dig to the details. And
even then, sometimes they forget and/or assume we have the resources to
massively revamp our stack.

There are long-standing grievances here (see tiran's gist for example).
I have very little experience with Fedora/RH, but from their grumbling,
I assume they solve some problems for developers that we don't:

1. It appears that multiple versions of cPython are trivially available.
Each version has its own site-packages.
2. Their python packaging is more developer-centric than user-centric.
More modules included in the install.

We're pretty constrained on 1 by the debian security team policy. But
maybe there is discussion room there for non-supported Pythons?

We have taken steps to improve 2 by adding the python3-full package.

I could see a long term strategy for having a system-python package that
provides a python3 binary used by Debian packages that need Python. And
a separate python package for developers that installs all the bells and
whistles immediately.

Achieving this would require reorganizing the way we package almost all
Python, and I don't think we have the interest and resources to do that.

If pyenv was packaged in Debian (ITP #978149) it would probably be great
for new Python developers on Debian.

Not sure what other small tactical steps we could take?

> > Perhaps someone else wants to comment on that conversation

Replied. I've engaged with these guys on this stuff before. Let's see
where it goes...

Stefano

--
Stefano Rivera
http://tumbleweed.org.za/
+1 415 683 3272


[163d8144-a584-4d1d-9d71-f86da864fde2]

Re: python devs complaining about debian packaging

[Donald Stufft]

It's not an accurate characterization that distutils was removed simply because 
it wasn't maintained.

It was as fragile library, and it was difficult to make any changes to it 
because a number of things had implemented themselves by reaching into 
distutils and randomly monkeypatching various aspects. This made it hard to 
maintain it _at all_, because when people had tried to improve it, it caused 
ecosystem wide breakages until things like setuptools, numpy.distutils, etc got 
patched again.

So the unofficial policy became "do not touch distutils", for fear of causing 
these breakages.

In the interim the packaging toolchain evolved to the point that having 
distutils in the stdlib was no longer of general benefit, and in fact made 
things worse because people had grown accustomed to things like `from distutils 
import setup` being transparently monkeypatched to be setuptools under the 
covers.

Ultimately, distutils had diverged so far from modern packaging tooling (due to 
the unofficial policy of not touching it), that it's existence was more or less 
a footgun, and it's only real purpose anymore was to be an implementation 
detail of other libraries, which you had to install from PyPI, so it was 
decided it was better to remove it rather than leave it around.
On 6/3/2024 1:37:32 AM, Salvo Tomaselli  wrote:
Consider that they are the same people that recently removed
"distutils" from the standard library, because it was not maintained.
When they have well enough funding to assign someone to maintain it,
instead of relying on external projects to install packages.

I think they are in the bubble of "people who are here on discuss" and
forgot the 99.9% who is not.

Il giorno lun 3 giu 2024 alle ore 00:08 Paul Boddie
ha scritto:
>
> On Monday, 27 May 2024 04:07:34 CEST Scott Kitterman wrote:
> >
> > While there are technical concerns on both sides, socially I think the
> > Python community isn't that interested in outside perspectives.
>
> I managed to dig up these notes from the packaging summit at PyCon:
>
> https://hackmd.io/@pradyunsg/pycon2024-pack-summit
>
> Here's the summit page itself:
>
> https://us.pycon.org/2024/events/packaging-summit/
>
> There is some fixation on the "system Python" in distributions, and the
> following remarks:
>
> "At least one distro team is working on moving their own Python out of the
> way, so users can install their own Python packages [...] Fedora tried
> platform-python and it broke everything, so it didn't really work"
>
> Given the proliferation of "virtual environments" around Python, where you
> just pick your own Python version and accompanying packages, I find it odd
> that the Python packaging community gets so hung up on the system Python. Do
> they want it to just go away or not be on the path or something? Wouldn't
> having a singular upstream Python just cause the same problems when someone
> finds that it isn't the version they need?
>
> For my own amusement and to confirm my own memories, I went back in time to
> check the Python Web site in the 1990s, and back then there was no problem in
> providing a binary for Linux and the Unix products of choice from that era.
> AIX, FreeBSD, HP-UX, Irix, OSF/1, Solaris, and SunOS 4, plus a Linux binary
> supplied either as an RPM or in a plain archive:
>
> https://web.archive.org/web/20021030010019/http://www.python.org/ftp/python/
> binaries-1.4/
>
> What is stopping anyone from doing that all over again? The user downloads the
> binary, puts it in their current directory, and runs their software. Could it
> be that the burden of support is perhaps a little greater than one might
> expect?
>
> Because from that starting point, you have to build multiple versions, and
> then you have to build accompanying libraries, and then you have to support
> third-party packages which need third-party libraries. It wasn't a surprise
> that things like Sun Freeware (http://www.sunfreeware.com) emerged to cater to
> the proprietary platforms, whereas distributions emerged around Linux to
> manage this complexity and provide all this software.
>
> It is easy for the various language communities to focus only on their own
> ecosystems, but everybody's software has to work together. And then there are
> companies targeting various markets that demand software built on a selection
> of different technologies, so you get perspectives like these:
>
> "Why did the PyPI and Conda ecosystem get created? It was originally created
> as an educational teaching language. If all of your tools are in Python, all
> of the things in your ecosystem are supposed to work well together. However,
> the tools that scientists and data scientists use are very commonly written in
> C, C++, etc. and so there’s something called a “native binary problem”. Making
> this stuff compatible across the board is an incredibly challenging issue!
> Conda was created to resolve those binary compatibility issues."
>
> I honestly don't know what these people th