Re: Solving the multiarch triplet once and for all

[Julien Cristau]

On Thu, Dec 13, 2012 at 11:34:12 +0100, Jakub Wilk wrote:

> How about undoing the Python multi-arch mess until someone provides
> evidence that the changes solve more problems than they create?
> 
Having some explanation of what's going on would definitely help.  Is
there a wiki or mailing list url that I missed and that describes what
this is about?

Cheers,
Julien
-- 
Julien Cristau  
Logilab http://www.logilab.fr/
Informatique scientifique & gestion de connaissances


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20121218132228.ga22...@crater2.logilab.fr

Re: Solving the multiarch triplet once and for all

[Barry Warsaw]

On Dec 18, 2012, at 02:22 PM, Julien Cristau wrote:

>On Thu, Dec 13, 2012 at 11:34:12 +0100, Jakub Wilk wrote:
>
>> How about undoing the Python multi-arch mess until someone provides
>> evidence that the changes solve more problems than they create?
>> 
>Having some explanation of what's going on would definitely help.  Is
>there a wiki or mailing list url that I missed and that describes what
>this is about?

I don't think you missed anything specific to Python.  The Debian multiarch
effort is described in this wiki page (with lots of sub-links):

http://wiki.debian.org/Multiarch

My take on it is this: in order to support installation of packages in
multiple architectures, you need a multiarch triplet-specific directory into
which you can install architecture-dependent code.  For typical (e.g. C)
libraries and headers, you have paths like /usr/lib/ and
/usr/include/.

For Python, we need a multiarch-aware location to install extension module .so
files.  As you know, Python uses /usr/lib/pythonX.Y and
/usr/include/pythonX.Y, which aren't multiarch-aware.  We could extend
Python's ABI flag directories (PEP 3149), e.g. /usr/lib/python3.3m but I don't
think that's a good idea.  Python already defines a platform-specific
location, e.g. /usr/lib/pythonX.Y/plat-linux2, but that isn't specific enough,
so Python was changed in Debian to search for and install to
`plat-`.

So far so good, except that there are lots of other tools that just assume
`plat-linux2` (or really, `plat-${sys.platform}`) because that's what they've
always done.  No big surprise really, since *lots* of tools aren't/weren't
prepared for multiarch.  Welcome to the brave new world.

At the command line, there are two official ways to discover the multiarch
triplet:

$ dpkg-architecture -qDEB_HOST_MULTIARCH
$ gcc --print-multiarch

I don't know which one is preferred, but I'm guessing it has more to do with
whatever packages you want to depend on.

It seems silly to me to require Python programs to shell out to either of
these every time they need to know the triplet.  It's not *entirely* trivial
to do.  A good example is virtualenv's site.py, which for bootstrapping
reasons can't use subprocess.  Why force Python programs to re-invent 1000
different ways to get this information, when we can (and have to) calculate it
when Python is built.

Besides, Python programs don't necessarily want the triplet for the
architecture they're running on, but for the architecture that the Python
executable was built for.  And that's not information that shelling out will
help you with.

So my proposed solution is to calculate the triplet for the architecture that
Python is being built on at configure time, and stash that into the sys module
so that Python code has a consistent, easy, and documented way to get at it.
sys is always going to be available, so that solves the bootstrap problem that
virtualenv has.

In Python 3.3, PEP 421 describes exactly where this information should go.
sys.implementation is the place to put it, and the name must start with an
underscore, so I chose `sys.implementation._architecture`.

In Python 2.7, there is no sys.implementation, so I chose `sys._architecture`
as the place to stash this.

If your code is bilingual, then you'd use:

>>> getattr(sys, 'implementation', sys)._architecture

which doesn't seem too horrible while we still have to support Python 2
.  Once everyone agrees this is the right approach, I'll document this
in the Debian wiki, and it should be used consistently wherever it's needed.
I have opened bugs and uploaded patches to fix Python 2.7 and 3.3, and
python-virtualenv.

Cheers,
-Barry


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20121218102420.06347...@limelight.wooz.org

Re: Solving the multiarch triplet once and for all

[Andrew Starr-Bochicchio]

On Tue, Dec 18, 2012 at 10:24 AM, Barry Warsaw  wrote:
> On Dec 18, 2012, at 02:22 PM, Julien Cristau wrote:
>>Having some explanation of what's going on would definitely help.  Is
>>there a wiki or mailing list url that I missed and that describes what
>>this is about?
>
> I don't think you missed anything specific to Python.  The Debian multiarch
> effort is described in this wiki page (with lots of sub-links):

This is the only thing I've come across:

http://wiki.debian.org/Python/MultiArch

-- Andrew Starr-Bochicchio

   Ubuntu Developer 
   Debian Developer 
   PGP/GPG Key ID: D53FDCB1


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/cal6k_azayckgr8xdkwgp8-wshrcwve_wfgbywzuenkinvot...@mail.gmail.com

Re: Solving the multiarch triplet once and for all

[Jakub Wilk]

* Barry Warsaw , 2012-12-18, 10:24:
For typical (e.g. C) libraries and headers, you have paths like 
/usr/lib/ and /usr/include/.


Quoting :
"If your -dev package contains headers which vary across architectures 
then it cannot be marked as Multi-Arch: same until a policy decision is 
made about architecture-dependant headers and the toolchain is updated."


--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20121218154729.ga3...@jwilk.net

Re: Solving the multiarch triplet once and for all

[Paul Wise]

Seems like a lot of effort when there is no /usr/bin//python
so there can be only one arch of python installed anyway. What are the
use-cases for python multiarch?

-- 
bye,
pabs

http://wiki.debian.org/PaulWise


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/caktje6e4zf_3fu4ph5mqjf3yhcthtc9rwf+ozvbjdpwdhhw...@mail.gmail.com

Re: Solving the multiarch triplet once and for all

[Barry Warsaw]

On Dec 18, 2012, at 10:40 AM, Andrew Starr-Bochicchio wrote:

>This is the only thing I've come across:
>
>http://wiki.debian.org/Python/MultiArch

Ah, of course.  I've added some cross-links.

-Barry


signature.asc
Description: PGP signature

Re: Solving the multiarch triplet once and for all

[Dmitrijs Ledkovs]

On 18 December 2012 15:48, Paul Wise  wrote:
> Seems like a lot of effort when there is no /usr/bin//python
> so there can be only one arch of python installed anyway. What are the
> use-cases for python multiarch?
>

cross-compiling the archive / cross-bootstrapping the archive for a
new architecture.

Regards,

Dmitrijs.


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/canbhlujab32bcb+xptzm0z-oo-iie39jkqv2jbl17o+mnp4...@mail.gmail.com

Re: Solving the multiarch triplet once and for all

[Paul Wise]

On Wed, Dec 19, 2012 at 12:29 AM, Dmitrijs Ledkovs wrote:

> cross-compiling the archive / cross-bootstrapping the archive for a
> new architecture.

I suppose cross-compiling will be useful but I didn't think python was
part of the build-essential set that must be cross-compilable, is that
actually the case?

PS: I'm subscribed.

-- 
bye,
pabs

http://wiki.debian.org/PaulWise


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/caktje6hkanjztu+a7ugzr-ztampx5r50aqxjtd0q5d6zbw9...@mail.gmail.com

Re: Solving the multiarch triplet once and for all

[Steve Langasek]

On Tue, Dec 18, 2012 at 11:48:56PM +0800, Paul Wise wrote:
> Seems like a lot of effort when there is no /usr/bin//python
> so there can be only one arch of python installed anyway. What are the
> use-cases for python multiarch?

Being able to cross-install packages that use libpython; and as Dmitrijs
mentions, being able to cross-build packages that use python too.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: Digital signature

Re: Solving the multiarch triplet once and for all

[Wookey]

+++ Paul Wise [2012-12-19 01:24 +0800]:
> On Wed, Dec 19, 2012 at 12:29 AM, Dmitrijs Ledkovs wrote:
> 
> > cross-compiling the archive / cross-bootstrapping the archive for a
> > new architecture.
> 
> I suppose cross-compiling will be useful but I didn't think python was
> part of the build-essential set that must be cross-compilable, is that
> actually the case?

It's not just build-essential that has to be cross-compilable. It's
everything you need to build the packages in a debootstrappable (inc
build-essential) image.

That's actually at least 254 packages. And a quite a lot of those need
perl or python to be multiarch aware in order to build.

The set could probably be trimmed down further by adding more
DEB_BUILD_PROFILE info to make reduced builds, but there is approx no
chance of getting rid of python IMHO.

The (current best guess) at the list is on
http://wiki.debian.org/Arm64Port (that's actually derived from Quantal
- unstable does not really have enough multiarch info in packages to
get any useful answers currently)

Wookey
-- 
Principal hats:  Linaro, Emdebian, Wookware, Balloonboard, ARM
http://wookware.org/


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20121218174553.gk9...@stoneboat.aleph1.co.uk

Re: Solving the multiarch triplet once and for all

[Jakub Wilk]

* Wookey , 2012-12-18, 17:45:
It's not just build-essential that has to be cross-compilable. It's 
everything you need to build the packages in a debootstrappable (inc 
build-essential) image.


That's actually at least 254 packages. And a quite a lot of those need 
perl or python to be multiarch aware in order to build.


Give us one example. (Of course we're interested in python, not perl.)

--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20121218191102.ga4...@jwilk.net