Re: [Python-Dev] Mercurial migration: progress report (PEP 385)
On Fri, Jul 3, 2009 at 20:04, Brett Cannon wrote: > Fine by me as long as people realize that if anything is questionable then > the switch will not happen. Getting this right takes precedence over any > deadline. And obviously we will need to do at least one live conversion on > python.org hardware to make sure everything will work smoothly. I'm not sure I see the need to do a (live? what does that mean in this context) on python.org hardware. Why exactly is that better than me doing it on one of my boxes, as long as all the necessary tools and an idea of how to do it are publically available (from the pymigr repo, for example)? > And will make the idea of splitting out the standard library and tests a > reasonable thing to do. In due time, yes. > While I really like the idea of using named branches for each release so > that there is a single py3k branch that contains all relevant history for > every release, I think we should start simple and go with cloned branches. > That way the workflow does not radically shift from what we do now for svn > to start. Once the conversion is done and people are comfortable with hg we > can then discuss moving towards a named branch approach. I don't think the cloned branches is much simpler than the named branches approach, in several ways. For example, populating the branch part of a sys.whatever value is significantly harder. Also, if you follow a useful tagging approach, doing cloned branches means that release tags aren't available on trunk/main/default. That seems like a step backwards. > Sounds reasonable to me. We can just make a list and send it to > python-committers to make final decisions of what should stick around. This list exists and has been referenced in my email a few times. > I don't use tags so I don't really care, but in the name of easy transition > I say we don't change the naming scheme (although I have no issue dropping > obscure tags). The current proposal is to clean up old tags to agree with the current naming scheme (and dropping obscure and partial tags). > Something else that can go out to python-committers before the switch. This should just be done ASAP, it helps with a smooth conversion process. > I don't think there is a single project we host -- all two of them -- that > have not said they want to convert. So I say convert everything and let's > turn off the svn server by the end of the year. I say we tackle each one as we go. I say doing a good conversion job is valuable, and we should take as much time as we need (though not more). You advocate something similar below for the Python conversion. > Can we check these scripts into the repository itself? That way there is a > chance of reuse as hg commands, e.g. ``hg pydev-ci`` as a replacement for > ``make patchcheck``. I'm not sure there's an easy way to make them into commands (although I guess you could make an extension to that effect), but hooks would be very easy. > How about hg.python.org for the official branches and we keep > code.python.org for personal branches of the developers like we have done > with the bzr experiments? I think that's just confusing. Most people seem to like hg.python.org, and it's obvious that hg goes to hg.p.o. Dividing up the namespace only makes it harder to find things. > As I have said, we should change our workflow habits after the switch and > people are comfortable with hg. Well, not everyone agrees, and although I think it doesn't matter much for the conversion itself, it may affect the branching strategy discussion. (sys.revision discussion elided because some others have already been bikeshedding on it.) Cheers, Dirkjan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: progress report (PEP 385)
On Sat, Jul 4, 2009 at 07:13, Nick Coghlan wrote: > I'd guess that the only way to keep those functional is to keep > svn.python.org around in read-only mode. No, actually: the idea (I think I floated it in the PEP, as well), is that I can write a simple extension for hgweb that knows the mapping of SVN rev to hg rev and so can make hg.python.org/python/2.x/rev/r32542 come out to the changeset that resulted from converting that revision. Cheers, Dirkjan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: progress report (PEP 385)
On Fri, Jul 3, 2009 at 17:17, Georg Brandl wrote: > Do you have a key to the second column in that file? E.g. the difference > between "strip" and "discard" is not clear to me. "strip partial"? strip == discard. strip = remove, merged should be obvious, keep-clone means we'll keep the branch, in a separate clone, and keep-named means we'll keep the branch as a named branch in one of the other clones. > Why are there branch names starting with "../"? Because some people in the history put their branch at /python/branchname instead of /python/branches/branchname. That part will get cleaned up. [Regarding clone#branch syntax:] > * Does it work with "hg pull" etc. too, afterwards? Yes. > * Does it support more than one branch? You can only reference one branch at a time using the fragment specifier. > I would just kill all feature branches unless someone indicates it is > still used. There are very few active feature branches. > > (I guess in the case a branch gets killed erroneously it could still be > re-created after the conversion?) Re-creating it after the conversion could be done, I guess, but I'd prefer to get it right, the first time. As the branch map says, I'm mostly removing all the feature branches anyway. > As long as it can send diffs... It can. > That gets very ugly as soon as you start pulling from repos that just > fix a small typo or so. Okay, so we do no automatic checking. We can always write a script that walks the repository and finds all significant contributors. Cheers, Dirkjan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: progress report (PEP 385)
On Sat, Jul 4, 2009 at 00:09, "Martin v. Löwis" wrote: > I would drop the roundup integration from the things that need to > be done pre-migration - there currently is no svn integration, so > not having it for hg is not a step backwards. Yeah, I mean just the linking here. > In the first sentence, you say that it can't actually work - so I > think we should drop the test. Okay. > I would like to see this well before the switch also. It could be > a patch (unified diff) stored in the tracker, or it could be an actual > branch that then needs to get merged with all active branches (IIUC). Yeah. Some help here would be welcome, too, as C is not a language I use often. Cheers, Dirkjan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: progress report (PEP 385)
On Sat, Jul 4, 2009 at 00:37, Barry Warsaw wrote: > Doesn't Mercurial support an Subversion bridge? Would it be possible to > provide a /read-only/ copy of the hg branches for 2.4 (maybe), 2.5, 2.6, and > 3.1? If so, then the release managers would simply have to cut their > releases from the svn copy instead of the hg master. /All/ other work would > be done from the hg master. This shouldn't be too much of a burden since > it's done so rarely and would end with the EOL of each of those branches. There is some push support in hgsubversion, but it doesn't do tags at this time, for example. I think this setup would be needlessly complicated (and tools will need to learn the new revision specifier anyway, so why not have it learn them sooner rather than later)? Cheers, Dirkjan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: progress report (PEP 385)
Brett Cannon wrote: I would very much like the 'k' dropped from the py3 name. It was a funny joke when py3 was vaporware, now it is excess baggage which only puzzles non-insiders and newcomers. Is it really that confusing? I have never heard of anyone asking "what is py3k?" Do you read python-list? It has been asked. Also, some people seem to think that py3k is different from python 3. tjr ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: progress report (PEP 385)
> I see that George Brandl and Martin van Loewis seem to be accomodating > your viewpoint, but I don't get the impression that either you (as the > hg migration proponent) nor they (as core committers) realize how far > apart your assumptions are. Actually, I (probably) don't agree to a merge flow different from the one that we currently have. All I asked that *if* Dirkjan is in favor of a change, he should propose it explicitly, and fleshed-out, in the PEP, so that people can tear it down. I know that several committers are unhappy with the current merge flow (Georg in particular), so this is important to discuss. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: progress report (PEP 385)
Dirkjan Ochtman wrote: > On Sat, Jul 4, 2009 at 07:13, Nick Coghlan wrote: >> I'd guess that the only way to keep those functional is to keep >> svn.python.org around in read-only mode. > > No, actually: the idea (I think I floated it in the PEP, as well), is > that I can write a simple extension for hgweb that knows the mapping > of SVN rev to hg rev and so can make > hg.python.org/python/2.x/rev/r32542 come out to the changeset that > resulted from converting that revision. Ah, that's an excellent solution - good to hear it :) Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
2009/7/3 Tarek Ziadé : > You can give me a bitbucket account so I can give you write access to the > repo, > There are tests as long as you install Nose. How do I get the tests to work? Just running nosetests gives an error (probably because pkgutil is being imported from the stdlib, rather than from this directory). If I set PYTHONPATH=. then I get errors. I suspect path normalisation (for backslashes) in the zipfile handling. Yes, it appears to be caused by the fact that Distribution uses os.path.join to construct self.pkg_info_path (and self.record_path) and yet zipfiles expect '/' separated pathnames. Paul. Error log: >\Apps\Python26\Scripts\nosetests-2.6.exe -v test_pkgutil.test_distributions ... ok test_pkgutil.test_distribution ... FAIL test_pkgutil.test_directory ... ok test_pkgutil.test_get_file_users ... ok test_pkgutil.test_egginfo_dirname ... ok test_pkgutil.test_zipped_directory ... ERROR test_pkgutil.test_zipped_distribution ... ERROR == ERROR: test_pkgutil.test_zipped_directory -- Traceback (most recent call last): File "C:\Apps\Python26\Lib\site-packages\nose\case.py", line 183, in runTest self.test(*self.arg) File "C:\Users\Gustav\Data\pep376\test_pkgutil.py", line 80, in test_zipped_directory dir = ZippedDistributionDir(SITE_PKG+'.zip') File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 324, in __init__ self.add(ZippedDistribution(self._zip_file, paths[0])) File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 259, in __init__ super(ZippedDistribution, self).__init__(path) File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 154, in __init__ pkginfo = self._open_pkginfo() File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 264, in _open_pkginfo return self.zipfile.open(self.pkg_info_path) File "C:\Apps\Python26\lib\zipfile.py", line 859, in open zinfo = self.getinfo(name) File "C:\Apps\Python26\lib\zipfile.py", line 826, in getinfo 'There is no item named %r in the archive' % name) KeyError: "There is no item named 'mercurial-1.0.1.egg-infoPKG-INFO' in the archive" == ERROR: test_pkgutil.test_zipped_distribution -- Traceback (most recent call last): File "C:\Apps\Python26\Lib\site-packages\nose\case.py", line 253, in setUp try_run(self.test, names) File "C:\Apps\Python26\Lib\site-packages\nose\util.py", line 487, in try_run return func() File "C:\Users\Gustav\Data\pep376\test_pkgutil.py", line 90, in setup_zip pkgutil._dist_dirs.load(SITE_PKG+'.zip') File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 380, in load dist_dir = ZippedDistributionDir(path) File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 324, in __init__ self.add(ZippedDistribution(self._zip_file, paths[0])) File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 259, in __init__ super(ZippedDistribution, self).__init__(path) File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 154, in __init__ pkginfo = self._open_pkginfo() File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 264, in _open_pkginfo return self.zipfile.open(self.pkg_info_path) File "C:\Apps\Python26\lib\zipfile.py", line 859, in open zinfo = self.getinfo(name) File "C:\Apps\Python26\lib\zipfile.py", line 826, in getinfo 'There is no item named %r in the archive' % name) KeyError: "There is no item named 'mercurial-1.0.1.egg-infoPKG-INFO' in the archive" == FAIL: test_pkgutil.test_distribution -- Traceback (most recent call last): File "C:\Apps\Python26\Lib\site-packages\nose\case.py", line 183, in runTest self.test(*self.arg) File "C:\Users\Gustav\Data\pep376\test_pkgutil.py", line 60, in test_distribution os.path.join(SITE_PKG, 'mercurial-1.0.1.egg-info/RECORD')]) AssertionError: ['C:\\Users\\Gustav\\Data\\pep376\\site-packages\\mercurial-1.0.1.egg-info\\PKG_INFO', 'C:\\Users\\ Gustav\\Data\\pep376\\site-packages\\mercurial-1.0.1.egg-info\\RECORD'] != ['C:\\Users\\Gustav\\Data\\pep376\\site- packages\\mercurial-1.0.1.egg-info/PKG_INFO', 'C:\\Users\\Gustav\\Data\\pep376\\site-packages\\mercurial-1.0.1.egg- info/RECORD'] -- Ran 7 tests in 0.019s FAILED (errors=2, failures=1) 13:40 C:\Users\Gustav\Data\pep376 >unzip -l site-packages.zip Archive: site-packages.zip Length EAs ACLsDate TimeName --- 2531 0 0 19/06/09 17:14 mercurial/filelog.py 3449 0 0 19/06/09 17:14 mercurial/filelog.pyc 234 0 0 19/06/09 17:14 mercurial-1.0.1.egg-info/PKG-INFO
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
2009/7/3 Tarek Ziadé : > 2009/7/3 Paul Moore : >> Does this sound sensible? Tarek, would you be OK with me attempting to >> modify your prototype to support this protocol? Are there any tests >> for PEP 376, so that I can confirm I haven't completely broken >> something? If I can, I'll knock up some simple prototype importers for >> non-standard examples, and see how they work with all this. > > Yes that's exactly what I was thinking about after the discussion we > had in the other thread. This change would allow much more flexibility. One important note - I plan on using the fact that DistributionDirMap is not public, and hacking it about drastically, or possibly even removing it. (After all, the likes of the load method don't make sense when you've got sys.meta_path, sys.path_importer_cache and the like to consider). If the loss of any of the "internal" classes is an issue, say so now! Paul. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
2009/7/4 Paul Moore :
> 2009/7/3 Tarek Ziadé :
>> You can give me a bitbucket account so I can give you write access to the
>> repo,
>> There are tests as long as you install Nose.
>
> How do I get the tests to work? Just running nosetests gives an error
> (probably because pkgutil is being imported from the stdlib, rather
> than from this directory).
>
> If I set PYTHONPATH=. then I get errors. I suspect path normalisation
> (for backslashes) in the zipfile handling.
Actually, the test
assert_equals(list(dist.get_egginfo_files(local=True)),
[os.path.join(SITE_PKG, 'mercurial-1.0.1.egg-info/PKG_INFO'),
os.path.join(SITE_PKG, 'mercurial-1.0.1.egg-info/RECORD')])
is broken, because the expected value uses slashes, which are *not*
the local separator on win32.
I've attached a patch.
But there's 2 comments I'd make (one minor, one major)
Minor one: The tests often seem to be exercising the internal classes,
not so much the public API, so many of them will probably not be of
much use to me :-(
Major one: I'm not entirely sure what the purpose of the "local" flag
is. With local=True, you are supposed to get a filesystem path in the
local form. But that makes no sense for non-filesystem based loaders
(e.g. zipfiles). So how is a user expected to use the value? With
local=false, you get slash-separated files (which is good, because
it's consistent) which are either absolute (see my earlier comments
about how broken this is in practice) or relative to the "directory"
containing the egg-info file (where this is of course not necessarily
a real filesystem location). But these aren't useful as filesystem
names, because they aren't local format.
So what is the point of the get_installed_files, uses, and
get_egginfo_files functions? Who will use the returned values, and
what for? Where will a user get a value to pass to uses()? Given that
get_egginfo_file takes names relative to the egginfo dir, shouldn't
get_egginfo_files *return* values relative to the same location, so
that they can be used there?
I guess the same type of argument applies with get_inetalled_files()
and uses() - the values returned by get_installed_files can be seen as
a set of opaque tokens (OK, realistically they will be "filenames" in
some sense) which can be passed to uses() - but if they aren't
absolute filesystem paths, you can't compare them. Example:
distribution a contains file p.py
distribution b contains file p.py
pkgutil.get_distribution(a).get_installed_files contains 'p.py'
pkgutil.get_distribution(b).uses('p.py') is True
What does this mean? If the 2 distributions are in the same sys.path
element, there's a clash. If not, is there a clash? What if they are
in 2 different directories? What if they are in the same directory,
but it's repeated under different names (e.g. symlinks) on sys.path?
What if one is in a directory and the other in a zip file?
This isn't even related to non-filesystem loaders. It's a fundamental
issue with the whole API. What you seem to want is actually a
canonical "name" for a file object - for filesystem files,
os.path.normcase(os.path.normpath(filename)) is probably good enough,
although it doesn't deal with symlinks. For other types of loader, you
have to rely on the loader itself - loader.get_fullname() is probably
OK. But even then, it's not clear how user code would actually *use*
these APIs.
I think you need some real-world use cases, with actual sample
(pseudo-)code, to validate the design here. As things stand, it's both
confusing and (I suspect) unusable in practice. Sorry, I know that
sounds negative, but if this isn't to be a source of subtle bugs for
years to come, it needs to be clarified now. PEP 302 is still hitting
this type of issue - runpy and importlib have brought out errors and
holes in the protocol quite recently - even though Just and I went to
great lengths to try to tease out hidden assumptions up front.
Concrete proposal:
get_metadata_files() - returns slash-separated names, relative to the
egginfo dir
get_metadata_file(path) - path must be slash-separated, relative to
the egginfo dir
get_installed_files - returns the contents of RECORD unaltered
uses(path) - checks if path is in RECORD
The latter 2 are not very useful in practice - you can't say anything
about entries in different RECORD files, which is likely the real use
case you want. Maybe RECORD could have an extra "Location" entry,
which determines where it exists globally (this would be the directory
to which the filenames were relative, in the case of filesystem-based
distributions) and RECORD entries are comparable if the Location
values in the 2 RECORD files match. That's a lot more complex - but
depending on what use people expect to make of these 2 APIs, it may be
justified.
Paul.
Patch tests to work with Win32
diff -r 31b39c766ebe test_pkgutil.py
--- a/test_pkgutil.py Thu Jul 02 12:17:42 2009 +0200
+++ b/test_pkgutil.py Sat Jul 04 14:51:54 2009 +0100
@@
Re: [Python-Dev] Mercurial migration: progress report (PEP 385)
On Sat, Jul 4, 2009 at 01:03, Dirkjan Ochtman wrote: > On Fri, Jul 3, 2009 at 20:04, Brett Cannon wrote: > > Fine by me as long as people realize that if anything is questionable > then > > the switch will not happen. Getting this right takes precedence over any > > deadline. And obviously we will need to do at least one live conversion > on > > python.org hardware to make sure everything will work smoothly. > > I'm not sure I see the need to do a (live? what does that mean in this > context) on python.org hardware. "Live" as in run through all the steps on python.org hardware w/o hitting any final switch that turns off svn. > Why exactly is that better than me > doing it on one of my boxes, as long as all the necessary tools and an > idea of how to do it are publically available (from the pymigr repo, > for example)? Because there are different OSs, installed software, etc. Basically because you just never know. =) Plus it will make Martin sleep better. > > > > And will make the idea of splitting out the standard library and tests a > > reasonable thing to do. > > In due time, yes. > > > While I really like the idea of using named branches for each release so > > that there is a single py3k branch that contains all relevant history for > > every release, I think we should start simple and go with cloned > branches. > > That way the workflow does not radically shift from what we do now for > svn > > to start. Once the conversion is done and people are comfortable with hg > we > > can then discuss moving towards a named branch approach. > > I don't think the cloned branches is much simpler than the named > branches approach, in several ways. For example, populating the branch > part of a sys.whatever value is significantly harder. Also, if you > follow a useful tagging approach, doing cloned branches means that > release tags aren't available on trunk/main/default. That seems like a > step backwards. > I personally prefer named branches, but that's just me and I am not about to force my preferences on everyone. Guess we just have to see if others have opinions against named branches. > > > Sounds reasonable to me. We can just make a list and send it to > > python-committers to make final decisions of what should stick around. > > This list exists and has been referenced in my email a few times. > Sure, but as inlining the PEP in this email thread has shown, not making people click a link helps. =) Plus a separate email makes it very obvious that people need to check their email instead of making it a bullet point in a much larger email. > > > I don't use tags so I don't really care, but in the name of easy > transition > > I say we don't change the naming scheme (although I have no issue > dropping > > obscure tags). > > The current proposal is to clean up old tags to agree with the current > naming scheme (and dropping obscure and partial tags). > Fine by me. > > > Something else that can go out to python-committers before the switch. > > This should just be done ASAP, it helps with a smooth conversion process. > > > I don't think there is a single project we host -- all two of them -- > that > > have not said they want to convert. So I say convert everything and let's > > turn off the svn server by the end of the year. > > I say we tackle each one as we go. I say doing a good conversion job > is valuable, and we should take as much time as we need (though not > more). You advocate something similar below for the Python conversion. > I am not suggesting we do all conversions on the same day, just that everything should eventually be converted. > > > Can we check these scripts into the repository itself? That way there is > a > > chance of reuse as hg commands, e.g. ``hg pydev-ci`` as a replacement for > > ``make patchcheck``. > > I'm not sure there's an easy way to make them into commands (although > I guess you could make an extension to that effect), but hooks would > be very easy. > OK. I was just hoping we could factor the code in such a way as to share the basic steps the hooks were checking so as to reuse them in a command. > > > How about hg.python.org for the official branches and we keep > > code.python.org for personal branches of the developers like we have > done > > with the bzr experiments? > > I think that's just confusing. Most people seem to like hg.python.org, > and it's obvious that hg goes to hg.p.o. Dividing up the namespace > only makes it harder to find things. > Yeah, I realize I have lost this battle. > > > As I have said, we should change our workflow habits after the switch and > > people are comfortable with hg. > > Well, not everyone agrees, and although I think it doesn't matter much > for the conversion itself, it may affect the branching strategy > discussion. > Sure, to an extent. > > (sys.revision discussion elided because some others have already been > bikeshedding on it.) I think it has been answered; sys.subversion goes away and sys.mer
Re: [Python-Dev] Mercurial migration: progress report (PEP 385)
On Sat, Jul 4, 2009 at 01:55, Terry Reedy wrote: > Brett Cannon wrote: > > I would very much like the 'k' dropped from the py3 name. It was a >>funny joke when py3 was vaporware, now it is excess baggage which >>only puzzles non-insiders and newcomers. >> >> >> Is it really that confusing? I have never heard of anyone asking "what is >> py3k?" >> > > Do you read python-list? No as it would take up so much of my Python time I wouldn't be able to code anymore. > It has been asked. Also, some people seem to think that py3k is different > from python 3. > Well, I still would not like to lose the py3k label. But if we do I still say 2.x/3.x instead of py2/py3. -Brett > > tjr > > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
On Fri, Jul 3, 2009 at 17:47, Nick Coghlan wrote: > Tarek Ziadé wrote: > > 2009/7/3 Paul Moore : > >> Does this sound sensible? Tarek, would you be OK with me attempting to > >> modify your prototype to support this protocol? Are there any tests > >> for PEP 376, so that I can confirm I haven't completely broken > >> something? If I can, I'll knock up some simple prototype importers for > >> non-standard examples, and see how they work with all this. > > > > Yes that's exactly what I was thinking about after the discussion we > > had in the other > > thread. This change would allow much more flexibility. > > It's also very similar to the ideas that started to tick over in the > back of my head after the previous discussion (only far more fleshed out > that anything I got to - fortunately I read Paul's message before > spending too much time on it!) > > Specifically regarding this comment from Paul: > > > Finder.uninstall(distribution_name, filter=callable, installer=name) > > Uninstall the given distribution. It's likely that many finders will > > be read-only. In that case, this function should return None. > > Otherwise, return a list of the "files" removed. (This may need some > > clarification, as many finders won't have a concept of a "file name"). > > PEP 302 finders and loaders actually *are* expected to have a concept of > a "file name". For Python modules they they are expected to have it so > they can populate __file__ correctly Not only that, lots of code out there expects values from __path__ to contain os.path.sep on top of __file__, so the idea of file paths is already hard-coded into how import works, whether we like it or not. > (runpy is also a lot happier with > them when they expose get_filename(), a relatively recent addition to > the PEP 302 API). An addition I was not even aware of. Looks like importlib needs a little updating. > For other resources they're expected to have it so the > get_data() API can work correctly. > > One other thing to consider: it may be desirable to have hooks for > meta_path and path_hooks in distutils that are checked *before* the full > import hooks in the sys module. The reason I say this is that it would > allow someone to override just the metadata retrieval (e.g. to pick up > the extra dependency information saved by setuptools and friends) > without having to worry about breaking the actual imports from those > locations. So as the guy who plans to try to clean up import, I just want to say that implicit finders will eventually go away. Do not start off making them implicit, but you can have them installed on sys.meta_path or sys.path_hooks by default. > > > Cheers, > Nick. > > P.S. +lots on using 'metadata' in the PEP 376 method names rather than > the jargon 'egginfo'. Jargon isn't always bad, but using it seems fairly > gratuitous in this case. Ditto from here. Plus I have an aversion to terminology that goes down the reptile route instead of the Monty Python route. -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial migration: progress report (PEP 385)
On Sat, 4 Jul 2009 at 12:28, Brett Cannon wrote: On Sat, Jul 4, 2009 at 01:03, Dirkjan Ochtman wrote: On Fri, Jul 3, 2009 at 20:04, Brett Cannon wrote: Fine by me as long as people realize that if anything is questionable then the switch will not happen. Getting this right takes precedence over any deadline. And obviously we will need to do at least one live conversion on python.org hardware to make sure everything will work smoothly. I'm not sure I see the need to do a (live? what does that mean in this context) on python.org hardware. "Live" as in run through all the steps on python.org hardware w/o hitting any final switch that turns off svn. "Live" should also (and I think this is critical) include a period of at least a week where we tell python-dev/python-committers "hg.python.org looks like it is going to look at cutover, please try out the workflow". The idea would be to have committers actually exercise the workflow on their platform for at least one patch, including whatever replaces the svnmerge step. I can almost guarantee we will find some issues that need to be resolved, and that people won't try it out thoroughly until it is "live". --David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
2009/7/4 Brett Cannon : >> > Finder.uninstall(distribution_name, filter=callable, installer=name) >> > Uninstall the given distribution. It's likely that many finders will >> > be read-only. In that case, this function should return None. >> > Otherwise, return a list of the "files" removed. (This may need some >> > clarification, as many finders won't have a concept of a "file name"). >> >> PEP 302 finders and loaders actually *are* expected to have a concept of >> a "file name". For Python modules they they are expected to have it so >> they can populate __file__ correctly > > Not only that, lots of code out there expects values from __path__ to > contain os.path.sep on top of __file__, so the idea of file paths is already > hard-coded into how import works, whether we like it or not. Yes, consider me duly reprimanded :-) They aren't *file* names, but they still have to look similar enough to satisfy code that uses things like filename manipulation functions on them. >> (runpy is also a lot happier with >> them when they expose get_filename(), a relatively recent addition to >> the PEP 302 API). > > An addition I was not even aware of. Looks like importlib needs a little > updating. Yes, I didn't notice that sneak into PEP 302, either :-) >> For other resources they're expected to have it so the >> get_data() API can work correctly. >> >> One other thing to consider: it may be desirable to have hooks for >> meta_path and path_hooks in distutils that are checked *before* the full >> import hooks in the sys module. The reason I say this is that it would >> allow someone to override just the metadata retrieval (e.g. to pick up >> the extra dependency information saved by setuptools and friends) >> without having to worry about breaking the actual imports from those >> locations. > > So as the guy who plans to try to clean up import, I just want to say that > implicit finders will eventually go away. Do not start off making them > implicit, but you can have them installed on sys.meta_path or sys.path_hooks > by default. I'm against having distutils-specific hooks. I don't see a genuine need, and it adds complexity so let's wait for a proper use case before adding that. With some suitably ugly hacking, you could probably add a finder to sys.meta_path which never actually found a module, but which reported one or more distributions for the distutils metadata APIs. Playing around with that idea could likely give you any flexibility you needed (at the hopefully minor cost of an extra dummy meta_path call on each import). As far as the currently implicit filesystem path handling is concerned, I'm aware that the intention is for importlib to move these into an explicit importer at some point. For my prototype changes to Tarek's prototype, I'm creating a dummy filesystem finder with just the new methods needed for the distribution metadata. It will be easy to move these into a real filesystem finder when one emerges. Paul. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
Paul Moore wrote: > 2009/7/4 Brett Cannon : >>> (runpy is also a lot happier with >>> them when they expose get_filename(), a relatively recent addition to >>> the PEP 302 API). >> An addition I was not even aware of. Looks like importlib needs a little >> updating. > > Yes, I didn't notice that sneak into PEP 302, either :-) I probably should have been noisier about that when I added it. I'm pretty sure it did come up on this list, but it would have been somewhere in the middle of a runpy discussion: runpy was stuck because it only uses PEP 302 to *find* the modules it needs, but not to actually load them. In the original version of PEP 302 the only way to get a loader to tell you the filename was to load the module and see what it set __file__ to, which wasn't useful in the runpy case. If I recall correctly, at the time when PJE was rationalising the code duplication between runpy and pkglib the comment was also made that runpy's get_filename() optional loader extension should be mentioned in PEP 302. runpy still works for loaders that don't provide it, it just can't set __file__ or sys.argv[0] correctly in those cases. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
I'm not complaining, just saying I didn't notice. I essentially did the same with get_source/get_bytecode so as to be able to set __file__. When I get around to it I will create a RunpyLoader in importlib.abc and add the method in the proper machinery loaders. On Jul 4, 2009 3:43 PM, "Nick Coghlan" wrote: Paul Moore wrote: > 2009/7/4 Brett Cannon : >>> (runpy is also a lot happier with >>> them when they expose get_filename(), a relatively recent ... I probably should have been noisier about that when I added it. I'm pretty sure it did come up on this list, but it would have been somewhere in the middle of a runpy discussion: runpy was stuck because it only uses PEP 302 to *find* the modules it needs, but not to actually load them. In the original version of PEP 302 the only way to get a loader to tell you the filename was to load the module and see what it set __file__ to, which wasn't useful in the runpy case. If I recall correctly, at the time when PJE was rationalising the code duplication between runpy and pkglib the comment was also made that runpy's get_filename() optional loader extension should be mentioned in PEP 302. runpy still works for loaders that don't provide it, it just can't set __file__ or sys.argv[0] correctly in those cases. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
