handling /usr/local/lib/python2.x/site-packages in sys.path
Currently Debian's python has /usr/local/lib/python2.x/site-packages in sys.path allowing for installation of local modules. Barry did point out that this conflicts with a python installation, where /usr/local is the default prefix, and the system python gets modules from the local installation. Some suggestions were made to fix this: - add an env var to not include /usr/local/lib/python2.x/site-packages when the env var is set. Keeps standard behaviour without modifications and allows people to remove it from sys.path. But requires the user to know about that option. - add another path (e.g. /usr/local/python/lib2.x/site-packages), and remove the /usr/local/lib/python2.x/site-packages path after the next release. Does provide an upgrade path, but doesn't solve the probem immediately. Matthias -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
Matthias Klose wrote: Currently Debian's python has /usr/local/lib/python2.x/site-packages in sys.path allowing for installation of local modules. Barry did point out that this conflicts with a python installation, where /usr/local is the default prefix, and the system python gets modules from the local installation. In other words you are trying to make things a bit easier for people who are a) expert enough to build and use their own python version and b) notice things breaking because they use a site-packages path which is also used by the system python. I can sympathise with building your own python (Debian's python uses twice the amount of memory needed for most unicode-using applications due to the use of UCS4 for example) but I think people who know enough to do that also know enough to deal with the resulting possible conflict over /usr/local/lib/python2.x use. Personally I have a number of local python's but they are all isolated in zc.buildout instances. Some suggestions were made to fix this: - add an env var to not include /usr/local/lib/python2.x/site-packages when the env var is set. Keeps standard behaviour without modifications and allows people to remove it from sys.path. But requires the user to know about that option. That makes things unpredicatable: use su or something else that can affect the environment and suddenly python behaves quite differently. That does not feel like a good idea. - add another path (e.g. /usr/local/python/lib2.x/site-packages), and remove the /usr/local/lib/python2.x/site-packages path after the next release. Does provide an upgrade path, but doesn't solve the probem immediately. That just means you will break people's python at a later point again. That certainly can not be the right thing to do. Iff we really must come up with a way to make it a bit easier for people to install their own python version make it a system-wide switch in /etc/default/python2.x and keep the default behaviour as it is now so you will not break existing installations. Wichert. -- Wichert Akkerman <[EMAIL PROTECTED]>It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 11, 2008, at 3:43 AM, Matthias Klose wrote: Currently Debian's python has /usr/local/lib/python2.x/site-packages in sys.path allowing for installation of local modules. Barry did point out that this conflicts with a python installation, where /usr/local is the default prefix, and the system python gets modules from the local installation. Some suggestions were made to fix this: - add an env var to not include /usr/local/lib/python2.x/site-packages when the env var is set. Keeps standard behaviour without modifications and allows people to remove it from sys.path. But requires the user to know about that option. - add another path (e.g. /usr/local/python/lib2.x/site-packages), and remove the /usr/local/lib/python2.x/site-packages path after the next release. Does provide an upgrade path, but doesn't solve the probem immediately. Thanks for raising this issue Matthias. There are a couple of other points I'd like to make. This isn't /just/ a problem for Python experts or developers. Sometimes we recommend that Ordinary Users install Python from source, say to use an application that isn't supported by a distro's packaging system. There can be lots of reasons for this: maybe the app is in beta testing, maybe they want to try things out without affecting the system version, etc. Python developers may not be Debian experts and may not know about this system peculiarity. So when we tell people "just do configure; make; make install" we can actually do inadvertent damage to their Debian system. It also makes Debian the odd man out. Instructions we publish for every other *nix have to have a caveat or FAQ for Debian's (and derivatives) difference. These can go unnoticed until things break, and then we can get difficult to debug problem reports. An experienced helper will be conditioned to first ask "Are you on Debian or Ubuntu? Well, you have to do things differently there." This feature is difficult to discover, and highly surprising to any Python person, especially more so if they are not Debian experts. When I first hit this problem (and complained to Matthias ;), it took me several hours to figure out what was going on, where the hack was being inserted, and that it was actually intentional. A more fundamental question though is, why do this in the first place? What's the use case? What convenience are you providing users, and if that's overwhelmingly beneficial, can it be done in a way that doesn't go counter to the conventions of the upstream project? What confuses me is that this doesn't appear to be a convenience based on permissions, because on my Ubuntu box it takes root to install things in /usr/local just as it would to install things in /usr. Meaning, if I'm going to install an egg in either location, either manually or via easy_install, I still need to be root. Although this doesn't affect things /now/ I think Python 2.6 will allow for user- specific sys.paths so that you could for example install an egg in ~/.local/lib/python2.6 and have it just show up. I'm also uncomfortable with the way that Debian accomplishes this: it hacks Python's standard site.py. There really should be a better way to do this (I have to think about this a bit though). I would opt for Matthias's second option, and would be happy if there was a roadmap for fixing this problem. - -Barry P.S. System applications written in Python should /never/ use "#! /usr/ bin/env python". They should always hardcode /usr/bin/pythonX.Y. Otherwise you can break your system just by tweaking your $PATH. You should have to work harder to break your system . This is why setuptools rewrites the #! line when it installs scripts in packages. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBR9aM8XEjvBPtnXfVAQKpBwP+OBvovf+MZsHTsU5N42TKqqHXi+IQwHKp rnCtqkBju+jYAWmJpE5BWHmYkBIyc9Me4N7tPMM04YYKwipub5tLEa1CUaHmtyWp 5sEHK2rlOJoluBvJy/S2NBgkSExw2RHmgkn5ZUb8IlLHO9HFh67Egc6vWG6Bg6xq GScfgB1bEfQ= =yk2u -END PGP SIGNATURE- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 11, 2008, at 3:58 AM, Wichert Akkerman wrote: - add another path (e.g. /usr/local/python/lib2.x/site-packages), and remove the /usr/local/lib/python2.x/site-packages path after the next release. Does provide an upgrade path, but doesn't solve the probem immediately. That just means you will break people's python at a later point again. That certainly can not be the right thing to do. Depends on how you look at it I guess. I see it as eventually fixing everyone's Python ;). Iff we really must come up with a way to make it a bit easier for people to install their own python version make it a system-wide switch in /etc/default/python2.x and keep the default behaviour as it is now so you will not break existing installations. I'm not sure what the "it" is you want to make a system-wide switch. Would that be which Python to use, or whether to add the sys.path hack? I don't know who's going to be at Pycon later this week, but if enough distro people are around, I'd be very happy to have a discussion about how to make this more consistent across distros, more amenable to users and developers, and how to make things less mysterious, confusing, or fragile. Given how many system tools are being built in Python now (a good thing!), there should be some agreement on the right way to do it. I'll be at Pycon and would be willing to chat about this at a BOF, over lunch, or over beers. :) - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBR9aPNHEjvBPtnXfVAQJCzgP/QaadzyGxnvA6WqT8B3QsouSVvZvsmg9n LAOnoLYzZOhcLzSUlYiukPcU6DMX8HAA8AaAvwpiyCLmk6IZDGMJxDZH+R8u4+w7 9E4KRdt3yxa5nbcJAMa/fWzjpDTlXSecZBAHw7IAYngy55+fv7cJ7UvWgNqsPdIB h4mUoEU32CU= =I8kv -END PGP SIGNATURE- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
On Tue, Mar 11, 2008 at 08:43:06AM +0100, Matthias Klose wrote: > Currently Debian's python has /usr/local/lib/python2.x/site-packages > in sys.path allowing for installation of local modules. Barry did > point out that this conflicts with a python installation, where > /usr/local is the default prefix, and the system python gets modules > from the local installation. Some suggestions were made to fix this: Well, I would expect that package I install in /usr/local are found automatically by the standard Python install. I think locally installing packages is a far more common use case than installing a hand-compiled python. It would be very unusual and inconvenient if installing modules with "python setup.py --prefix=/usr/local" wouldn't work with the standard Python installation. - Sebastian -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
[Matthias Klose, 2008-03-11] > - add another path (e.g. /usr/local/python/lib2.x/site-packages), >and remove the /usr/local/lib/python2.x/site-packages path after >the next release. Does provide an upgrade path, but doesn't solve >the probem immediately. I would choose this option or even remove /usr/local/* from sys.path completely. If one wants to play with unsupported Python modules or Eggs, he should sys.path.append(his_path) by hand, IMHO. If we additionally force ez_* mess to install to /usr/local/ by default, we will get rid of one of the most common problems I receive from users of my packages. -- -=[ Piotr Ozarowski ]=- -=[ http://www.ozarowski.pl ]=- pgp5fH41nar8Z.pgp Description: PGP signature
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
On Tue, Mar 11, 2008 at 06:58:56PM +0100, Piotr Ożarowski wrote: > I would choose this option or even remove /usr/local/* from sys.path > completely. If one wants to play with unsupported Python modules or > Eggs, he should sys.path.append(his_path) by hand, IMHO. I would not consider installing packages that are not packaged in Debian "playing". Installing such packages is a common use case for system administrators and advanced users. With the same reasoning we could remove /usr/local/lib from /etc/ld.so.conf. - Sebastian -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
Matthias Klose wrote: > - add an env var to not include /usr/local/lib/python2.x/site-packages >when the env var is set. Keeps standard behaviour without >modifications and allows people to remove it from sys.path. But >requires the user to know about that option. I would much prefer this. Debian users with local (=unpackaged) packages are probably much more common than Debian users with (=unpackaged) versions of all python, and I would rather reasonably support those than leaving them in the cold. Quite frankly, installing stuff that is also present in the system under user local is a recipe for disaster (also happens with libraries) and rather hard to cater for. > - add another path (e.g. /usr/local/python/lib2.x/site-packages), >and remove the /usr/local/lib/python2.x/site-packages path after >the next release. Does provide an upgrade path, but doesn't solve >the probem immediately. No. Let's not break stuff for people that use the software in Debian that they got from Debian. Quite frankly, the use case seems a bit of bad practice. Installing to /usr/local software to compete with that in /usr leads to all sorts of breakage where one doesn't expect it and if python.org needs people to install experimental versions they should either recommend chroots / test machines or provide packages. Anyone capable of installing a local version of python is also able to run debootstrap to create a chroot. Kind regards T. (who used to maintain a set of libraries where local installations caused no end of trouble for users, maintainers, and upstreams) -- Thomas Viehmann, http://thomas.viehmann.net/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
[Sebastian Rittau, 2008-03-11] > On Tue, Mar 11, 2008 at 06:58:56PM +0100, Piotr Ożarowski wrote: > > I would choose this option or even remove /usr/local/* from sys.path > > completely. If one wants to play with unsupported Python modules or > > Eggs, he should sys.path.append(his_path) by hand, IMHO. > > I would not consider installing packages that are not packaged in Debian > "playing". Installing such packages is a common use case for system > administrators and advanced users. With the same reasoning we could > remove /usr/local/lib from /etc/ld.so.conf. true. /me still doesn't know how to (at the same time) eat a cookie (get rid of Egg problems) and still be able to keep it (give advanced users/administrators possibility to add local modifications). -- -=[ Piotr Ozarowski ]=- -=[ http://www.ozarowski.pl ]=- pgptrogCSkHrZ.pgp Description: PGP signature
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
On Tue, Mar 11, 2008 at 09:45:21AM -0400, Barry Warsaw wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On Mar 11, 2008, at 3:43 AM, Matthias Klose wrote: > >> Currently Debian's python has /usr/local/lib/python2.x/site-packages >> in sys.path allowing for installation of local modules. Barry did >> point out that this conflicts with a python installation, where >> /usr/local is the default prefix, and the system python gets modules >> from the local installation. Some suggestions were made to fix this: >> >> - add an env var to not include /usr/local/lib/python2.x/site-packages >> when the env var is set. Keeps standard behaviour without >> modifications and allows people to remove it from sys.path. But >> requires the user to know about that option. >> >> - add another path (e.g. /usr/local/python/lib2.x/site-packages), >> and remove the /usr/local/lib/python2.x/site-packages path after >> the next release. Does provide an upgrade path, but doesn't solve >> the probem immediately. > > Thanks for raising this issue Matthias. There are a couple of other > points I'd like to make. > > This isn't /just/ a problem for Python experts or developers. Sometimes > we recommend that Ordinary Users install Python from source, say to use > an application that isn't supported by a distro's packaging system. > There can be lots of reasons for this: maybe the app is in beta testing, > maybe they want to try things out without affecting the system version, > etc. > > Python developers may not be Debian experts and may not know about this > system peculiarity. So when we tell people "just do configure; make; > make install" we can actually do inadvertent damage to their Debian > system. Problem is that both are very natural, having local packages being picked up by the system python in /usr/local/lib/python2.X/site-packages as well as being able to install a new python with a prefix of /usr/local. Given this choice I'd prefer the second option as well, it seems more natural then the first and an upgrade path is important (Wichert's argument about the first having problems with su is also important). It would be good if the distutils could be changed to install local packages there by default then, so as to provide the least surprises: just "python setup.py install" as root would do the correct thing. When looking at the grand scale of things the change for the system Python to look for local packages in /usr/local/python/lib2.X/site-packages (or /usr/local/python/site-packages2.X?[1]) is probably best done upstream so it is on all systems the same. The build system could do this when a prefix of /usr is passed to configure. This can not be done by using a system wide configuration file IIRC, since the Python make system will pick up those configuration files hence breaking the /usr/local install again as well as generating FTBFS on systems that have python already installed during the build[0]. This is possibly a bug in the Python make system though, what is the rationale for picking up the distutils configuration files for building python itself? Surely all this info is passed to configure? > What confuses me is that this doesn't appear to be a convenience based > on permissions, It has nothing to do with permissions. The sysadmin should never install things into /usr/ directly, /usr/local/ is their playground. > I'm also uncomfortable with the way that Debian accomplishes this: it > hacks Python's standard site.py. There really should be a better way to > do this (I have to think about this a bit though). In the python we ship to our customers (living in /opt/vendor/python2X as per FHS) we place a .pth file in the site-packages directory to accomplish extra locations on sys.path (pointing somewhere else in our /opt/vendor hierarchy). Maybe /usr/lib/python2.X/site-packages/debian.pth would be a better way to handle this on Debian? Or are there reasons not to do this? Although the suggestion I make above, changing it upstream, would make this unnecessary. [As an aside, I find what happens with python-support much worse. I regularly waste time re-discovering that a whole load of system installed packages can't be found in /usr/lib/python2.X/site-packages/ but live somewhere in /var/lib/python-support/. It's a violation of the FHS as well, /var/lib/ is for program state information, .pyc's are not state).] Regards Floris [0] This has bitten me in the past with FTBFS when having a ~/.pydistutils.cfg defining a `home=~' and trying to rebuild the Debian Python package. I think it also happens when trying to build a new upstream python with an existing distutils configuration somewhere. [1] Both violate FHS though. The directories allowed in /usr/local after a fresh install are limited (http://www.pathname.com/fhs/pub/fhs-2.3.html#USRLOCALLOCALHIERARCHY). But /usr/local/lib/python/site-packages2.X does seem a little cumberso
