Re: [Python-Dev] PEP 382: Namespace Packages
At 10:32 AM 4/2/2009 -0500, Martin v. Löwis wrote: I propose the following PEP for inclusion to Python 3.1. Please comment. An excellent idea. One thing I am not 100% clear on, is how to get additions to sys.path to work correctly with this. Currently, when pkg_resources adds a new egg to sys.path, it uses its existing registry of namespace packages in order to locate which packages need __path__ fixups. It seems under this proposal that it would have to scan sys.modules for objects with __path__ attributes that are lists that begin with a '*', instead... which is a bit troubling because sys.modules doesn't always only contain module objects. Many major frameworks place lazy module objects, and module proxies or wrappers of various sorts in there, so scanning through it arbitrarily is not really a good idea. Perhaps we could add something like a sys.namespace_packages that would be updated by this mechanism? Then, pkg_resources could check both that and its internal registry to be both backward and forward compatible. Apart from that, this mechanism sounds great! I only wish there was a way to backport it all the way to 2.3 so I could drop the messy bits from setuptools. -- http://mail.python.org/mailman/listinfo/python-list
Re: [Python-Dev] PEP 382: Namespace Packages
At 10:33 PM 4/2/2009 +0200, M.-A. Lemburg wrote: That's going to slow down Python package detection a lot - you'd replace an O(1) test with an O(n) scan. I thought about this too, but it's pretty trivial considering that the only time it takes effect is when you have a directory name that matches the name you're importing, and that it will only happen once for that directory, unless there is no package on sys.path with that name, and the program tries to import the package multiple times. In other words, the overhead isn't likely to be much, compared to the time needed to say, open and marshal even a trivial __init__.py file. Alternative Approach: - Wouldn't it be better to stick with a simpler approach and look for "__pkg__.py" files to detect namespace packages using that O(1) check ? I thought the same thing (or more precisely, a single .pkg file), but when I got lower in the PEP I saw the reason was to support system packages not having overlapping filenames. The PEP could probably be a little clearer about the connection between needing *.pkg and the system-package use case. One of the namespace packages, the defining namespace package, will have to include a __init__.py file. Note that there is no such thing as a "defining namespace package" -- namespace package contents are symmetrical peers. The above mechanism allows the same kind of flexibility we already have with the existing normal __init__.py mechanism. * It doesn't add yet another .pth-style sys.path extension (which are difficult to manage in installations). * It always uses the same naive sys.path search strategy. The strategy is not determined by some file contents. The above are also true for using only a '*' in .pkg files -- in that event there are no sys.path changes. (Frankly, I'm doubtful that anybody is using extend_path and .pkg files to begin with, so I'd be fine with a proposal that instead used something like '.nsp' files that didn't even need to be opened and read -- which would let the directory scan stop at the first .nsp file found. * The search is only done once - on the first import of the package. I believe the PEP does this as well, IIUC. * It's possible to have a defining package dir and add-one package dirs. Also possible in the PEP, although the __init__.py must be in the first such directory on sys.path. (However, such "defining" packages are not that common now, due to tool limitations.) -- http://mail.python.org/mailman/listinfo/python-list
Re: [Python-Dev] PEP 382: Namespace Packages
At 03:21 AM 4/3/2009 +0200, Matthias Klose wrote: +1 speaking as a downstream packaging python for Debian/Ubuntu I welcome this approach. The current practice of shipping the very same file (__init__.py) in different packages leads to conflicts for the installation of these packages (this is not specific to dpkg, but is true for rpm packaging as well). Current practice of packaging (for downstreams) so called "name space packages" is: - either to split out the namespace __init__.py into a separate(linux distribution) package (needing manual packaging effort for eachname space package) - using downstream specific packaging techniques to handle conflicting files(diversions) - replicating the current behaviour of setuptools simply overwriting thefile conflicts. Following this proposal (downstream) packaging of namespace packages is made possible independent of any manual downstream packaging decisions or any downstream specific packaging decisions A clarification: setuptools does not currently install the __init__.py file when installing in --single-version-externally-managed or --root mode. Instead, it uses a project-version-nspkg.pth file that essentially simulates a variation of Martin's .pkg proposal, by abusing .pth file support. If this PEP is adopted, setuptools would replace its nspkg.pth file with a .pkg file on Python versions that provide native support for .pkg imports, keeping the .pth file only for older Pythons. (.egg files and directories will not be affected by the change, unless the zipimport module will also supports .pkg files... and again, only for Python versions that support the new approach.) -- http://mail.python.org/mailman/listinfo/python-list
Re: [Python-Dev] PEP 382: Namespace Packages
At 10:15 PM 4/3/2009 +0200, Martin v. Löwis wrote:
I should make it clear that this is not the case. I envision it to work
this way: import zope
- searches sys.path, until finding either a directory zope, or a file
zope.{py,pyc,pyd,...}
- if it is a directory, it checks for .pkg files. If it finds any,
it processes them, extending __path__.
- it *then* checks for __init__.py, taking the first hit anywhere
on __path__ (just like any module import would)
- if no .pkg was found, nor an __init__.py, it proceeds with the next
sys.path item (skipping the directory entirely)
Ah, I missed that. Maybe the above should be added to the PEP to clarify.
--
http://mail.python.org/mailman/listinfo/python-list
Re: [Python-Dev] PEP 382: Namespace Packages
At 02:00 PM 4/6/2009 +0100, Chris Withers wrote: Martin v. Löwis wrote: Chris Withers wrote: Would this support the following case: I have a package called mortar, which defines useful stuff: from mortar import content, ... I now want to distribute large optional chunks separately, but ideally so that the following will will work: from mortar.rbd import ... from mortar.zodb import ... from mortar.wsgi import ... Does the PEP support this? That's the primary purpose of the PEP. Are you sure? Does the pep really allow for: from mortar import content from mortar.rdb import something ...where 'content' is a function defined in mortar/__init__.py and 'something' is a function defined in mortar/rdb/__init__.py *and* the following are separate distributions on PyPI: - mortar - mortar.rdb ...where 'mortar' does not contain 'mortar.rdb'. See the third paragraph of http://www.python.org/dev/peps/pep-0382/#discussion -- http://mail.python.org/mailman/listinfo/python-list
Re: [Python-Dev] PEP 382: Namespace Packages
At 02:30 PM 4/7/2009 +0200, M.-A. Lemburg wrote: >> Wouldn't it be better to stick with a simpler approach and look for >> "__pkg__.py" files to detect namespace packages using that O(1) check ? > > Again - this wouldn't be O(1). More importantly, it breaks system > packages, which now again have to deal with the conflicting file names > if they want to install all portions into a single location. True, but since that means changing the package infrastructure, I think it's fair to ask distributors who want to use that approach to also take care of looking into the __pkg__.py files and merging them if necessary. Most of the time the __pkg__.py files will be empty, so that's not really much to ask for. This means your proposal actually doesn't add any benefit over the status quo, where you can have an __init__.py that does nothing but declare the package a namespace. We already have that now, and it doesn't need a new filename. Why would we expect OS vendors to start supporting it, just because we name it __pkg__.py instead of __init__.py? -- http://mail.python.org/mailman/listinfo/python-list
Re: [Python-Dev] PEP 382: Namespace Packages
At 04:58 PM 4/7/2009 +0200, M.-A. Lemburg wrote: On 2009-04-07 16:05, P.J. Eby wrote: > At 02:30 PM 4/7/2009 +0200, M.-A. Lemburg wrote: >> >> Wouldn't it be better to stick with a simpler approach and look for >> >> "__pkg__.py" files to detect namespace packages using that O(1) >> check ? >> > >> > Again - this wouldn't be O(1). More importantly, it breaks system >> > packages, which now again have to deal with the conflicting file names >> > if they want to install all portions into a single location. >> >> True, but since that means changing the package infrastructure, I think >> it's fair to ask distributors who want to use that approach to also take >> care of looking into the __pkg__.py files and merging them if >> necessary. >> >> Most of the time the __pkg__.py files will be empty, so that's not >> really much to ask for. > > This means your proposal actually doesn't add any benefit over the > status quo, where you can have an __init__.py that does nothing but > declare the package a namespace. We already have that now, and it > doesn't need a new filename. Why would we expect OS vendors to start > supporting it, just because we name it __pkg__.py instead of __init__.py? I lost you there. Since when do we support namespace packages in core Python without the need to add some form of magic support code to __init__.py ? My suggestion basically builds on the same idea as Martin's PEP, but uses a single __pkg__.py file as opposed to some non-Python file yaddayadda.pkg. Right... which completely obliterates the primary benefit of the original proposal compared to the status quo. That is, that the PEP 382 way is more compatible with system packaging tools. Without that benefit, there's zero gain in your proposal over having __init__.py files just call pkgutil.extend_path() (in the stdlib since 2.3, btw) or pkg_resources.declare_namespace() (similar functionality, but with zipfile support and some other niceties). IOW, your proposal doesn't actually improve the status quo in any way that I am able to determine, except that it calls for loading all the __pkg__.py modules, rather than just the first one. (And the setuptools implementation of namespace packages actually *does* load multiple __init__.py's, so that's still no change over the status quo for setuptools-using packages.) -- http://mail.python.org/mailman/listinfo/python-list
Re: [Python-Dev] PEP 382: Namespace Packages
At 05:02 PM 4/14/2009 +0200, M.-A. Lemburg wrote: I don't see the emphasis in the PEP on Linux distribution support and the remote possibility of them wanting to combine separate packages back into one package as good argument for adding yet another separate hierarchy of special files which Python scans during imports. That said, note that most distributions actually take the other route: they try to split up larger packages into smaller ones, so the argument becomes even weaker. I think you've misunderstood something about the use case. System packaging tools don't like separate packages to contain the *same file*. That means that they *can't* split a larger package up with your proposal, because every one of those packages would have to contain a __pkg__.py -- and thus be in conflict with each other. Either that, or they would have to make a separate system package containing *only* the __pkg__.py, and then make all packages using the namespace depend on it -- which is more work and requires greater co-ordination among packagers. Allowing each system package to contain its own .pkg or .nsp or whatever files, on the other hand, allows each system package to be built independently, without conflict between contents (i.e., having the same file), and without requiring a special pseudo-package to contain the additional file. Also, executing multiple __pkg__.py files means that when multiple system packages are installed to site-packages, only one of them could possibly be executed. (Note that, even though the system packages themselves are not "combined", in practice they will all be installed to the same directory, i.e., site-packages or the platform equivalent thereof.) -- http://mail.python.org/mailman/listinfo/python-list
Re: [Python-Dev] PEP 382: Namespace Packages
At 10:59 PM 4/14/2009 +0200, M.-A. Lemburg wrote: You are missing the point: When breaking up a large package that lives in site-packages into smaller distribution bundles, you don't need namespace packages at all, so the PEP doesn't apply. The way this works is by having a base distribution bundle that includes the needed __init__.py file and a set of extension bundles the add other files to the same directory (without including another copy of __init__.py). The extension bundles include a dependency on the base package to make sure that it always gets installed first. If we're going to keep that practice, there's no point to having the PEP: all three methods (base+extensions, pkgutil, setuptools) all work just fine as they are, with no changes to importing or the stdlib. In particular, without the feature of being able to drop that practice, there would be no reason for setuptools to adopt the PEP. That's why I'm -1 on your proposal: it's actually inferior to the methods we already have today. -- http://mail.python.org/mailman/listinfo/python-list
Re: [Python-Dev] PEP 382: Namespace Packages
At 09:51 AM 4/15/2009 +0200, M.-A. Lemburg wrote: On 2009-04-15 02:32, P.J. Eby wrote: > At 10:59 PM 4/14/2009 +0200, M.-A. Lemburg wrote: >> You are missing the point: When breaking up a large package that lives in >> site-packages into smaller distribution bundles, you don't need namespace >> packages at all, so the PEP doesn't apply. >> >> The way this works is by having a base distribution bundle that includes >> the needed __init__.py file and a set of extension bundles the add >> other files to the same directory (without including another copy of >> __init__.py). The extension bundles include a dependency on the base >> package to make sure that it always gets installed first. > > If we're going to keep that practice, there's no point to having the > PEP: all three methods (base+extensions, pkgutil, setuptools) all work > just fine as they are, with no changes to importing or the stdlib. Again: the PEP is about creating a standard for namespace packages. It's not about making namespace packages easy to use for Linux distribution maintainers. Instead, it's targeting *developers* that want to enable shipping a single package in multiple, separate pieces, giving the user the freedom to the select the ones she needs. Of course, this is possible today using various other techniques. The point is that there is no standard for namespace packages and that's what the PEP is trying to solve. > In particular, without the feature of being able to drop that practice, > there would be no reason for setuptools to adopt the PEP. That's why > I'm -1 on your proposal: it's actually inferior to the methods we > already have today. It's simpler and more in line with the Python Zen, not inferior. You are free not to support it in setuptools - the methods implemented in setuptools will continue to work as they are, but continue to require support code and, over time, no longer be compatible with other tools building upon the standard defined in the PEP. In the end, it's the user that decides: whether to go with a standard or not. Up until this point, I've been trying to help you understand the use cases, but it's clear now that you already understand them, you just don't care. That wouldn't be a problem if you just stayed on the sidelines, instead of actively working to make those use cases more difficult for everyone else than they already are. Anyway, since you clearly understand precisely what you're doing, I'm now going to stop trying to explain things, as my responses are apparently just encouraging you, and possibly convincing bystanders that there's some genuine controversy here as well. -- http://mail.python.org/mailman/listinfo/python-list
Re: [Python-Dev] PEP 382: Namespace Packages
At 09:10 AM 4/15/2009 -0700, Aahz wrote: For the benefit of us bystanders, could you summarize your vote at this point? Given the PEP's intended goals, if you do not oppose the PEP, are there any changes you think should be made? I'm +1 on Martin's original version of the PEP, subject to the point brought up by someone that .pkg should be changed to a different extension. I'm -1 on all of MAL's proposed revisions, as IMO they are a step backwards: they "standardize" an approach that will create problems that don't need to exist, and don't exist now. Martin's proposal is an improvement on the status quo, Marc's proposal is a dis-improvement. -- http://mail.python.org/mailman/listinfo/python-list
Re: [Python-Dev] PEP 382: Namespace Packages
At 06:15 PM 4/15/2009 +0200, M.-A. Lemburg wrote: The much more common use case is that of wanting to have a base package installation which optional add-ons that live in the same logical package namespace. Please see the large number of Zope and PEAK distributions on PyPI as minimal examples that disprove this being the common use case. I expect you will find a fair number of others, as well. In these cases, there is NO "base package"... the entire point of using namespace packages for these distributions is that a "base package" is neither necessary nor desirable. In other words, the "base package" scenario is the exception these days, not the rule. I actually know specifically of only one other such package besides your mx.* case, the logilab ll.* package. -- http://mail.python.org/mailman/listinfo/python-list
Re: [Python-Dev] .pth files are evil
At 04:18 PM 5/9/2009 +0200, Martin v. Löwis wrote: Zooko O'Whielacronx wrote: > .pth files are why I can't easily use GNU stow with easy_install. > If installing a Python package involved writing new files into the > filesystem, but did not require reading, updating, and re-writing any > extant files such as .pth files, then GNU stow would Just Work with > easy_install the way it Just Works with most things. Please understand that this is the fault of easy_install, not of .pth files. There is no technical need for easy_install to rewrite .pth files on installation. It could just as well have created new .pth files, rather than modifying existing ones. If you always use --single-version-externally-managed with easy_install, it will stop editing .pth files on installation. It's --multi-version (-m) that does that. --single-version-externally-managed is a "setup.py install" option. Both have the effect of not editing .pth files, but they do so in different ways. The "setup.py install" option causes it to install in a distutils-compatible layout, whereas --multi-version simply drops .egg files or directories in the target location and leaves it to the user (or the generated script wrappers) to add them to sys.path. -- http://mail.python.org/mailman/listinfo/python-list
Re: [Python-Dev] .pth files are evil
At 04:42 PM 5/9/2009 +0200, Martin v. Löwis wrote: >> If you always use --single-version-externally-managed with easy_install, >> it will stop editing .pth files on installation. > > It's --multi-version (-m) that does that. > --single-version-externally-managed is a "setup.py install" option. > > Both have the effect of not editing .pth files, but they do so in > different ways. The "setup.py install" option causes it to install in a > distutils-compatible layout, whereas --multi-version simply drops .egg > files or directories in the target location and leaves it to the user > (or the generated script wrappers) to add them to sys.path. Ah, ok. Is there also an easy_install invocation that unpacks the zip file into some location of sys.path (which then wouldn't require editing sys.path)? Not as yet. I'm sort of waiting to see what comes out of PEP 376 discussions re: an installation manifest... but then, if I actually had time to work on it right now, I'd probably just implement something. Currently, you can use pip to do that, though, as long as the packages you want are in source form. pip doesn't unzip eggs as yet. It would be really straightforward, though, for someone to implement an easy_install variant that does this. Just invoke "easy_install -Zmaxd /some/tmpdir packagelist" to get a full set of unpacked .egg directories in /some/tmpdir, and then move the contents of the resulting .egg subdirs to the target location, renaming EGG-INFO subdirs to projectname-version.egg-info subdirs. (Of course, this ignores the issue of uninstalling previous versions, or overwriting of conflicting files in the target -- does pip handle these?) -- http://mail.python.org/mailman/listinfo/python-list
Re: how GNU stow is complementary rather than alternative to distutils
At 12:04 PM 5/10/2009 -0600, Zooko Wilcox-O'Hearn wrote: The thing that prevents this from working with setuptools is that setuptools creates a file named easy_install.pth during the "python ./ setup.py install --prefix=foo" if you build two different Python packages this way, they will each create an easy_install.pth file, and then when you ask GNU stow to link the two resulting packages into your system, it will say "You are asking me to install two different packages which both claim that they need to write a file named '/usr/local/lib/python2.5/site-packages/easy_install.pth'. Adding --record and --single-version-externally-managed to that command line will prevent the .pth file from being used or needed, although I believe you already know this. (What that mode won't do is install dependencies automatically.) -- http://mail.python.org/mailman/listinfo/python-list
Re: [Python-Dev] .pth files are evil
At 04:42 PM 5/9/2009 +0200, Martin v. Löwis wrote:
>> If you always use --single-version-externally-managed with easy_install,
>> it will stop editing .pth files on installation.
>
> It's --multi-version (-m) that does that.
> --single-version-externally-managed is a "setup.py install" option.
>
> Both have the effect of not editing .pth files, but they do so in
> different ways. The "setup.py install" option causes it to install in a
> distutils-compatible layout, whereas --multi-version simply drops .egg
> files or directories in the target location and leaves it to the user
> (or the generated script wrappers) to add them to sys.path.
Ah, ok. Is there also an easy_install invocation that unpacks the zip
file into some location of sys.path (which then wouldn't require
editing sys.path)?
No; you'd have to use the -e option to easy_install to download and
extract a source version of the package; then run that package's
setup.py, e.g.:
easy_install -eb /some/tmpdir SomeProject
cd /some/tmpdir/someproject # subdir is always lowercased/normalized
setup.py install --single-version-externally-managed --record=...
I suspect that this is basically what pip is doing under the hood, as
that would explain why it doesn't support .egg files.
I previously posted code to the distutils-sig that was an .egg
unpacker with appropriate renaming, though. It was untested, and
assumes you already checked for collisions in the target directory,
and that you're handling any uninstall manifest yourself. It could
probably be modified to take a filter function, though, something like:
def flatten_egg(egg_filename, extract_dir, filter=lambda s,d: d):
eggbase = os.path.filename(egg_filename)+'-info'
def file_filter(src, dst):
if src.startswith('EGG-INFO/'):
src = eggbase+s[8:]
dst = os.path.join(extract_dir, *src.split('/'))
return filter(src, dst)
return unpack_archive(egg_filename, extract_dir, file_filter)
Then you could pass in a None-returning filter function to check and
accumulate collisions and generate a manifest. A second run with the
default filter would do the unpacking.
(This function should work with either .egg files or .egg directories
as input, btw, since unpack_archive treats a directory input as if it
were an archive.)
Anyway, if you used "easy_install -mxd /some/tmpdir [specs]" to get
your target eggs found/built, you could then run this flattening
function (with appropriate filter functions) over the *.egg contents
of /some/tmpdir to do the actual installation.
(The reason for using -mxd instead of -Zmaxd or -zmaxd is that we
don't care whether the eggs are zipped or not, and we leave out the
-a so that dependencies already present on sys.path aren't copied or
re-downloaded to the target; only dependencies we don't already have
will get dropped in /some/tmpdir.)
Of course, the devil of this is in the details; to handle conflicts
and uninstalls properly you would need to know what namespace
packages were in the eggs you are installing. But if you don't care
about blindly overwriting things (as the distutils does not), then
it's actually pretty easy to make such an unpacker.
I mainly haven't made one myself because I *do* care about things
being blindly overwritten.
--
http://mail.python.org/mailman/listinfo/python-list
Re: [Distutils] uses for setup.cfg and extracting data from it
At 11:25 PM 9/9/2009 +1000, Ben Finney wrote: That's one of the pain points of the current distutils capability: there's no standard-library way to extract that information. If you're talking about setup.cfg (and all the other distutils .cfg files), all you need to do is create a Distribution object and call parse_config_files() on it, then access the appropriate attributes. Take you maybe 3 or 4 lines of code. If you're talking about setup.py, all you need to do is use the distutils functions that allow you to run a setup.py without executing any of its commands. (Of course, you then need to be able to deal with badly-behaved setup scripts, and it's true there's no stdlib support for that.) -- http://mail.python.org/mailman/listinfo/python-list
Re: [Distutils] uses for setup.cfg and extracting data from it
At 08:14 AM 9/12/2009 +1000, Ben Finney wrote:
Specifically, I want to programmatically access the metadata that is
held in the arguments to the âdistutils.setup()â call. Without,
as you say, executing any Distutils command. I am not aware of any
âdistutilsâ public functions that can do that, and reading
http://docs.python.org/distutils/> again doesn't enlighten me.
Would you be more specific about what these functions are?
http://docs.python.org/distutils/apiref.html#module-distutils.core -
specifically the run_setup() function. (It appears the docs do not
have link anchors for individual functions, alas.)
distutils.core.run_setup("setup.py", [], "init") will return you an
initialized Distribution object from running the setup script,
without parsing any configuration files or executing any
commands. You could use "config" or "commandline" instead of "init"
if you wanted those pieces of processing to be done as well - i.e. if
you wanted to be sure any setup.cfg options were processed.
This will work with a sufficiently well-behaved setup.py; setup
scripts that try to do build or install steps outside of any
distutils command may produce side-effects when run. (Which is why
setuptools always runs setup scripts in a loose sandbox that detects
when a script tries to modify the filesystem outside the setup directory.)
--
http://mail.python.org/mailman/listinfo/python-list
