[Python-Dev] Proposed: add an environment variable, PYTHONPREFIXES
I've submitted a patch to implement a new environment variable,
PYTHONPREFIXES. The patch is here:
http://bugs.python.org/issue5819
PYTHONPREFIXES is similar to PYTHONUSERBASE: it lets you add "prefix
directories" to be culled for site packages. It differs from
PYTHONUSERBASE in three ways:
* PYTHONPREFIXES has an empty default value. PYTHONUSERBASE has a
default, e.g. ~/.local on UNIX-like systems.
* PYTHONPREFIXES supports multiple directories, separated by the
site-specific directory separator character (os.pathsep). Earlier
directories take precedence. PYTHONUSERBASE supports specifying
at most one directory.
* PYTHONPREFIXES adds its directories to site.PREFIXES, so it reuses
the existing mechanisms for site package directories, exactly
simulating a real prefix directory. PYTHONUSERBASE only adds a
single directory, using its own custom code path.
This last point bears further discussion. PYTHONUSERBASE's custom code
to inspect only a single directory has resulted in at least one bug, if
not more, as follows:
* The bona-fide known bug: the Debian package mantainer for Python
decided to change "site-packages" to "dist-packages" in 2.6,
for reasons I still don't quite understand. He made this change in
site.addsitepackages and distutils.sysconfig.get_python_lib, and
similarly in setuptools, but he missed changing it in
site.addusersitepackages. This meant that if you used setup.py to
install a package to a private prefix directory, PYTHONUSERBASE had
no hope of ever finding the package. (Happily this bug is fixed.)
* I suspect there's a similar bug with PYTHONUSERBASE on the "os2emx"
and "riscos" platforms. site.addsitepackages on those platforms
looks in "{prefix}/Lib/site-packages", but
site.addusersitepackages looks in
"{prefix}/lib/python{version}/site-packages" as it does
on any non-Windows platform. Presumably setup.py on those two
platforms installs site packages to the directory site.addsitepackages
inspects, which means that PYTHONUSERBASE doesn't work on those
two platforms.
PYTHONUSERBASE's custom code path to add site package directories seems
unnecessary to me. I cannot fathom why its implementors chose this
approach; in any case I think reusing site.addsitepackages is a clear
win. I fear it's too late to change PYTHONUSERBASE so it simply called
site.addsitepackages, as that would change its established semantics.
Though if that idea found support I'd be happy to contribute a patch.
A few more notes on PYTHONPREFIXES:
* PYTHONPREFIXES is gated by the exact same mechanisms that shut off
PYTHONUSERBASE.
* Specifying "-s" on the Python command line shuts it off.
* Setting the environment variable PYTHONNOUSERSITE to a non-empty
string shuts it off.
* If the effective uid / gid doesn't match the actual uid / gid it
automatically shuts off.
* I'm not enormously happy with the name. Until about an hour or two
ago I was calling it "PYTHONUSERBASES". I'm open to other
suggestions.
* I'm not sure that PYTHONPREFIX should literally modify site.PREFIXES.
If that's a bad idea I'd be happy to amend the patch so it didn't
touch site.PREFIXES.
* Reaction in python-ideas has been reasonably positive, though I gather
Nick Coughlan and Scott David Daniels think it's unnecessary. (To
read the discussion, search for the old name: "PYTHONUSERBASES".)
* Ben Finney prefers a counter-proposal he made in the python-ideas
discussion: change the existing PYTHONUSERBASE to support multiple
directories. I don't like this approach, because:
a) it means you have to explicitly add the local default if you
want to use it, and
b) PYTHONUSERBASE only inspects one directory, whereas PYTHONPREFIX
inspects all the directories Python might use for site packages.
I do admit this approach would be preferable to no change at all.
The patch is thrillingly simple and works fine. However it's not ready
to be merged because I haven't touched the documentation. I figured I'd
hold off until I see which way the wind blows.
I'd also be happy to convert this into a PEP if that's what's called for.
/larry/
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Reminder: Python Bug Day on Saturday
Hi, I'd like to remind everyoone that there will be a Python Bug Day on April 25. As always, this is a perfect opportunity to get involved in Python development, or bring your own issues to attention, discuss them and (hopefully) resolve them together with the core developers. We will coordinate over IRC, in #python-dev on irc.freenode.net, and the Wiki page http://wiki.python.org/moin/PythonBugDay has all important information and a short list of steps how to get set up. Hope to see you there! Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Location of OS-installed versus Python-installed libraries (was: Proposed: add an environment variable, PYTHONPREFIXES)
Larry Hastings writes: > the Debian package mantainer for Python decided to change > "site-packages" to "dist-packages" in 2.6, for reasons I still don't > quite understand. For reference, Larry is referring to changes announced by Matthias Klose on 2009-02-16 in Message-ID: <[email protected]> http://lists.debian.org/debian-devel/2009/02/msg00431.html>: > Local installation path > --- > > When installing Python modules using distutils, the resulting files > end up in the same location wether they are installed by a Debian > package, or by a local user or administrator, unless the installation > path is overwritten on the command line. Compare this with most > software based on autoconf, where an explicit prefix has to be > provided for the packaging, while the default install installs into > /usr/local. For new Python versions packaged in Debian this will > change so that an installation into /usr (not /usr/local) requires an > extra option to distutils install command (--install-layout=deb). To > avoid breaking the packaging of existing code the distutils install > command for 2.4 and 2.5 will just accept this option and ignore it. > For the majority of packages we won't see changes in the packaging, > provided that the python packaging helpers can find the files in the > right location and move it to the expected target path. > > A second issue raised by developers was the clash of modules and > extensions installed by a local python installation (with default > prefix /usr/local) with the modules provided by Debian packages > (/usr/local/lib/pythonX.Y/site-packages shared by the patched "system" > python and the locally installed python. To avoid this clash the > directory `site-packages' should be renamed to `dist-packages' in > both locations: > > - /usr/lib/pythonX.Y/dist-packages (installation location for code >packaged for Debian) > - /usr/local/lib/pythonX.Y/dist-packages (installation location >for locally installed code using distutils install without >options). > > The path /usr/lib/pythonX.Y/site-packages is not found on sys.path > anymore. > > About the name: Discussed this with Barry Warsaw and Martin v. Loewis, > and we came to the conclusion that using the same directory name for > both locations would be the most consistent way. -- \ “In any great organization it is far, far safer to be wrong | `\ with the majority than to be right alone.” —John Kenneth | _o__)Galbraith, 1989-07-28 | Ben Finney ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Suggested doc patch for tarfile
Hi, The current documentation for tarfile.TarFile.extractfile() does not mention that the returned 'file-like object' supports close() and also iteration. The attached patch (against svn trunk) fixes this. (Background: I was wondering whether I could write def process_and_close_file(f_in): with closing(f_in) as f: # Do stuff with f. and have it work whether f_in was a true file or the return value of extractfile(), and thought from the documentation that I couldn't. Of course, I could have just tried it, but I think fixing the documentation wouldn't hurt.) Alternative: enhance the tarfile.ExFileObject class to support use as a context manager? Thanks, Ben. tarfile.rst.patch Description: Binary data ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Suggested doc patch for tarfile
On Thu, Apr 23, 2009, Ben North wrote: > > The current documentation for tarfile.TarFile.extractfile() does not > mention that the returned 'file-like object' supports close() and also > iteration. The attached patch (against svn trunk) fixes this. Please post the patch to bugs.python.org -- Aahz ([email protected]) <*> http://www.pythoncraft.com/ "If you think it's expensive to hire a professional to do the job, wait until you hire an amateur." --Red Adair ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Suggested doc patch for tarfile
>> The current documentation for tarfile.TarFile.extractfile() does not >> mention that the returned 'file-like object' supports close() and also >> iteration. The attached patch (against svn trunk) fixes this. > > Please post the patch to bugs.python.org Done: http://bugs.python.org/issue5821 Thanks, Ben. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python3 and arm-linux
Somebody knowns, is python3 works on arm-linux. Is it possible to build it? Where to find related discussions? Maybe some special patches already available? Should i try to get sources from svn or get known version snapshot? - ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python3 and arm-linux
On Thu, Apr 23, 2009 at 11:21 AM, cyberGn0m wrote: > Somebody knowns, is python3 works on arm-linux. Is it possible to build it? > Where to find related discussions? Maybe some special patches already > available? Should i try to get sources from svn or get known version > snapshot? > I haven't tried, but there's an interesting distro at http://www.vanille-media.de/site/index.php/projects/python-for-arm-linux/ -- I don't know if other such distros have better-updated Python versions (eg. current 2.6.* vs that one's 2.4.*) but that one includes a lot of very useful add-ons. Alex ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python3 and arm-linux
Yes, i visited it.. but.. looks that it was not updated for a few years, maybe it was integrated and python-for-arm-linux is a part of openembedded project? 2009/4/23 Alex Martelli > On Thu, Apr 23, 2009 at 11:21 AM, cyberGn0m wrote: > >> Somebody knowns, is python3 works on arm-linux. Is it possible to build >> it? Where to find related discussions? Maybe some special patches already >> available? Should i try to get sources from svn or get known version >> snapshot? >> > > I haven't tried, but there's an interesting distro at > http://www.vanille-media.de/site/index.php/projects/python-for-arm-linux/-- I > don't know if other such distros have better-updated Python versions > (eg. current 2.6.* vs that one's 2.4.*) but that one includes a lot of very > useful add-ons. > > > Alex > > -- - Всего наилучшего ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 383: Non-decodable Bytes in System Character Interfaces
On 22Apr2009 08:50, Martin v. L�wis wrote: | File names, environment variables, and command line arguments are | defined as being character data in POSIX; Specific citation please? I'd like to check the specifics of this. | the C APIs however allow | passing arbitrary bytes - whether these conform to a certain encoding | or not. Indeed. | This PEP proposes a means of dealing with such irregularities | by embedding the bytes in character strings in such a way that allows | recreation of the original byte string. [...] So you're proposing that all POSIX OS interfaces (which use byte strings) interpret those byte strings into Python3 str objects, with a codec that will accept arbitrary byte sequences losslessly and is totally reversible, yes? And, I hope, that the os.* interfaces silently use it by default. | For most applications, we assume that they eventually pass data | received from a system interface back into the same system | interfaces. For example, and application invoking os.listdir() will | likely pass the result strings back into APIs like os.stat() or | open(), which then encodes them back into their original byte | representation. Applications that need to process the original byte | strings can obtain them by encoding the character strings with the | file system encoding, passing "python-escape" as the error handler | name. -1 This last sentence kills the idea for me, unless I'm missing something. Which I may be, of course. POSIX filesystems _do_not_ have a file system encoding. The user's environment suggests a preferred encoding via the locale stuff, and apps honouring that will make nice looking byte strings as filenames for that user. (Some platforms, like MacOSX' HFS filesystems, _do_ enforce an encoding, and a quite specific variety of UTF-8 it is; I would say they're not a full UNIX filesystem _precisely_ because they reject certain byte strings that are valid on other UNIX filesystems. What will your proposal do here? I can imagine it might cope with existing names, but what happens when the user creates a new name?) Further, different users can use different locales and encodings. If they do it in different work areas they'll be perfectly happy; if they do it in a shared area doubtless confusion will reign, but only in the users' minds, not in the filesystem. If I'm writing a general purpose UNIX tool like chmod or find, I expect it to work reliably on _any_ UNIX pathname. It must be totally encoding blind. If I speak to the os.* interface to open a file, I expect to hand it bytes and have it behave. As an explicit example, I would be just fine with python's open(filename, "w") to take a string and encode it for use, but _not_ ok for os.open() to require me to supply a string and cross my fingers and hope something sane happens when it is turned into bytes for the UNIX system call. I'm very much in favour of being able to work in strings for most purposes, but if I use the os.* interfaces on a UNIX system it is necessary to be _able_ to work in bytes, because UNIX file pathnames are bytes. If there isn't a byte-safe os.* facility in Python3, it will simply be unsuitable for writing low level UNIX tools. And I very much like using Python2 for that. Finally, I have a small python program whose whole purpose in life is to transcode UNIX filenames before transfer to a MacOSX HFS directory, because of HFS's enforced particular encoding. What approach should a Python app take to transcode UNIX pathnames under your scheme? Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ The nice thing about standards is that you have so many to choose from; furthermore, if you do not like any of them, you can just wait for next year's model. - Andrew S. Tanenbaum ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 383: Non-decodable Bytes in System Character Interfaces
On 24Apr2009 09:27, I wrote: | If I'm writing a general purpose UNIX tool like chmod or find, I expect | it to work reliably on _any_ UNIX pathname. It must be totally encoding | blind. If I speak to the os.* interface to open a file, I expect to hand | it bytes and have it behave. As an explicit example, I would be just fine | with python's open(filename, "w") to take a string and encode it for use, | but _not_ ok for os.open() to require me to supply a string and cross | my fingers and hope something sane happens when it is turned into bytes | for the UNIX system call. | | I'm very much in favour of being able to work in strings for most | purposes, but if I use the os.* interfaces on a UNIX system it is | necessary to be _able_ to work in bytes, because UNIX file pathnames | are bytes. Just to follow up to my own words here, I would be ok for all the pure-byte stuff to be off in the "posix" module if os.* goes pure character instead of bytes or bytes+strings. -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ ... that, in a few years, all great physical constants will have been approximately estimated, and that the only occupation which will be left to men of science will be to carry these measurements to another place of decimals. - James Clerk Maxwell (1813-1879) Scientific Papers 2, 244, October 1871 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 383: Non-decodable Bytes in System Character Interfaces
On 22Apr2009 21:17, Martin v. L�wis wrote: | > -1. On UNIX, character data is not sufficient to represent paths. We | > must, must, must continue to have a simple bytes interface to these | > APIs. | | I'd like to respond to this concern in three ways: | | 1. The PEP doesn't remove any of the existing interfaces. So if the |interfaces for byte-oriented file names in 3.0 work fine for you, |feel free to continue to use them. Ok. I think I had read things as supplanting byte-oriented interfaces with this exciting new strings-can-do-it-all approach. | 2. Even if they were taken away (which the PEP does not propose to do), |it would be easy to emulate them for applications that want them. |For example, listdir could be wrapped as | |def listdir_b(bytestring): |fse = sys.getfilesystemencoding() Alas, no, because there is no sys.getfilesystemencoding() at the POSIX level. It's only the user's current locale stuff on a UNIX system, and has _nothing_ to do with the filesystem because UNIX filesystems don't have encodings. In particular, because the "best" (or to my mind "misleading") you can do for this is report what the current user thinks: http://docs.python.org/library/sys.html#sys.getfilesystemencoding then there's no guarrentee that what is chosen has any releationship to what was in use when the files being consulted were made. Now, if I were writing listdir_b() I'd want to be able to do something along these lines: - set LC_ALL=C (or some equivalent mechanism) - have os.listdir() read bytes as numeric values and transcode their values _directly_ into the corresponding Unicode code points. - yield bytes( ord(c) for c in os_listdir_string ) - have os.open() et al transcode unicode code points back into bytes. i.e. a straight one-to-one mapping, using only codepoints in the range 1..255. Then I'd have some confidence that I had got hold of the bytes as they had come from the underlying UNIX system call, and a way to get those bytes _back_ to a UNIX system call intact. |string = bytestring.decode(fse, "python-escape") |for fn in os.listdir(string): |yield fn.encoded(fse, "python-escape") | | 3. I still disagree that we must, must, must continue to provide these |interfaces. I don't understand from the rest of your message what |would *actually* break if people would use the proposed interfaces. My other longer message describes what would break, if I understand your proposal. -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 383: Non-decodable Bytes in System Character Interfaces
On Apr 22, 2009, at 2:50 AM, Martin v. Löwis wrote: I'm proposing the following PEP for inclusion into Python 3.1. Please comment. +1. Even if some people still want a low-level bytes API, it's important that the easy case be easy. That is: the majority of Python applications should *just work, damnit* even with not-properly-encoded- in-current-LC_CTYPE filenames. It looks like this proposal accomplishes that, and does so in a relatively nice fashion. James ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 383: Non-decodable Bytes in System Character Interfaces
Martin v. Löwis wrote: MRAB wrote: Martin v. Löwis wrote: [snip] To convert non-decodable bytes, a new error handler "python-escape" is introduced, which decodes non-decodable bytes using into a private-use character U+F01xx, which is believed to not conflict with private-use characters that currently exist in Python codecs. The error handler interface is extended to allow the encode error handler to return byte strings immediately, in addition to returning Unicode strings which then get encoded again. If the locale's encoding is UTF-8, the file system encoding is set to a new encoding "utf-8b". The UTF-8b codec decodes non-decodable bytes (which must be >= 0x80) into half surrogate codes U+DC80..U+DCFF. If the byte stream happens to include a sequence which decodes to U+F01xx, shouldn't that raise an exception? I apparently have not expressed it clearly, so please help me improve the text. What I mean is this: - if the environment encoding (for lack of better name) is UTF-8, Python stops using the utf-8 codec under this PEP, and switches to the utf-8b codec. - otherwise (env encoding is not utf-8), undecodable bytes get decoded with the error handler. In this case, U+F01xx will not occur in the byte stream, since no other codec ever produces this PUA character (this is not fully true - UTF-16 may also produce PUA characters, but they can't appear as env encodings). So the case you are referring to should not happen. I think what's confusing me is that you talk about mapping non-decodable bytes to U+F01xx, but you also talk about decoding to half surrogate codes U+DC80..U+DCFF. If the bytes are mapped to single half surrogate codes instead of the normal pairs (low+high), then I can see that decoding could never be ambiguous and encoding could produce the original bytes. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python3 and arm-linux
> > Somebody knowns, is python3 works on arm-linux. Is it possible to build it? > > Where to find related discussions? Maybe some special patches already > > available? Should i try to get sources from svn or get known version > > snapshot? > > > > I haven't tried, but there's an interesting distro at http://www.vanille- > media.de/site/index.php/projects/python-for-arm-linux/ -- I don't know if > other such distros have better-updated Python versions (eg. current 2.6.* > vs that one's 2.4.*) but that one includes a lot of very useful add-ons. You may want to look at the Ångström distro. http://www.angstrom-distribution.org/ http://www.angstrom-distribution.org/repo/?pkgname=libpython2.6-1.0 That's where I'll be heading in a couple of weeks. (I have a new BeagleBoard with an ARM Cortex-A8.) Larry ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
