Re: [Python-Dev] [Fwd: Re: PEP 384: Defining a Stable ABI]
My perspective is as follows: 1) If PEP-384 had always been in place, my life would now be a lot easier. 2) Since it hasn't always been in place, its introduction won't help me in the short term: there are an awful lot of extension modules that use excluded functions (for example, all(?) PyCxx modules use PyCode_New and PyFrame_New to get nicer tracebacks), and I'll still have to handle all these cases until everyone is up-to-date with whatever version of Python this gets accepted into. 3) Regardless, this PEP makes me very happy, because I can now look forward to the glorious day when all extension modules are 384-compatible (and even *some* modules becoming compatible will make me pretty happy). However, I'm not sure exactly how we can get there from here; I suspect that certain features of certain extensions already depend critically upon implementation details which will become hidden. The most extreme illustrative example I know is from NumPy (in scalarmathmodule.c), and looks like this: PyInt_Type.tp_as_number = PyLongArrType_Type.tp_as_number; PyInt_Type.tp_compare = PyLongArrType_Type.tp_compare; PyInt_Type.tp_richcompare = PyLongArrType_Type.tp_richcompare; ...and I fear that many many similar (if perhaps less frightening) dependencies exist elsewhere. Regardless, in answer to the two specific questions you ask: a) We don't really have that option. However, I would have a much higher degree of confidence in running PEP-384-compatible modules under Ironclad than I do with current modules, simply because I would no longer need to worry about (say) edge cases in which extension writers suddenly try to directly access op->ob_type->tp_as_number->nb_power. b) I can't think of any more useful restrictions. The PEP would solve my biggest current worry, which is that my current implementation allows managed/unmanaged lists to fall out of sync in certain circumstances (but if every list mutation happened via an API call, it wouldn't be an issue). Best Regards William Michael Foord wrote: The questions from Martin v. Lowis are in the email below. The PEP under discussion is: http://www.python.org/dev/peps/pep-0384/ I can proxy any replies you want to send, or you can join Python-dev. All the best, Michael Original Message Subject:Re: [Python-Dev] PEP 384: Defining a Stable ABI Date: Mon, 18 May 2009 08:00:57 +0200 From: "Martin v. Löwis" To: Michael Foord CC: Dino Viehland , Python-Dev , Unladen Swallow , Python List References: <[email protected]> <[email protected]> <1a472770e042064698cb5adc83a12acd016e8...@tk5ex14mbxc116.redmond.corp.microsoft.com> <[email protected]> <[email protected]> >>> It also might make it easier for alternate implementations to support >>> the same API so some modules could work cross implementation - but I >>> suspect that's a non-goal of this PEP :). >>> >> >> Indeed :-) I'm also skeptical that this would actually allow >> cross-implementation modules to happen. The list of functions that >> an alternate implementation would have to provide is fairly long. >> >> > > Just in case you're unaware of it; the company I work for has an open > source project called Ironclad. I was unaware indeed; thanks for pointing this out. IIUC, it's not just an API emulation, but also an ABI emulation. > In particular we have had to address the issue of the GIL and extensions > (IronPython has no GIL) and reference counting (which IronPython also > doesn't) use. I think this somewhat strengthens the point I was trying to make: An alternate implementation that tries to be API compatible has to consider so many things that it is questionable whether making Py_INCREF/DECREF functions would be any simplification. So I just ask: a) Would it help IronClad if it could restrict itself to PEP 384 compatible modules? b) Would further restrictions in the PEP help that cause? Regards, Martin -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 : Changing the .egg-info structure
On 17 May, 2009, at 15:04, MRAB wrote: Alexander Shigin wrote: В Сбт, 16/05/2009 в 23:15 +0100, MRAB пишет: FYI, on RISC OS '/' is a valid filename character and '.' is used as the directory separator. I'd probably say that TAB is s reasonable character to use, even though it's OK in POSIX; after all, should anyone really be using a control character in a filename? The '\0' char is invalid in both windows and posix. I don't know if one valid on RISC OS. '\0' isn't a valid filename character on RISC OS. Wouldn't it be possible to use a CSV file for this? That way we wouldn't have to invent yet another escaping mechanism and there's already good suppport for reading and writing CSV files in the standard library. Ronald smime.p7s Description: S/MIME cryptographic signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 : Changing the .egg-info structure
Ronald Oussoren mac.com> writes: > > Wouldn't it be possible to use a CSV file for this? That way we > wouldn't have to invent yet another escaping mechanism and there's > already good suppport for reading and writing CSV files in the > standard library. +1 We can even customize the delimiter if you want to make it more readable (or if there's a shortage of bikeshed material ;-)). cheers Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 : Changing the .egg-info structure
On Sat, May 16, 2009 at 6:55 PM, P.J. Eby wrote: > > 1. Why ';' separation, instead of tabs as in PEP 262? Aren't semicolons a > valid character in filenames? I am changing this into a . for now. What about Antoine's idea about doing a quote() on the names ? >From my point of view seems more simple to deal with, if 3rd-party tools want to work on these files without using pkgutil or Python. > > 4. There should probably be a way to iterate over the projects in a > directory, since it's otherwise impossible for an installation tool to find > out what project(s) "own" a file that conflicts with something being > installed. Alternatively, reshaping the file API to allow querying by path > as well as by project might work. I am adding a "get_projects" api: get_projects() -> iterator Provides an iterator that will return (name, path) tuples, where `name` is the name of a registered project and `path` the path to its `egg-info` directory. But for the use case you are mentioning, what about an explicit API: get_owners(paths) -> sequence of project names returns a sequence of tuple. For each path in the "paths" list, a tuple of project names is returned > > 5. If any cache mechanisms are to be used by the API, the API *must* make it > possible to bypass or explicitly manage that cache, as otherwise > installation tools and tools that manipulate sys.path at runtime may end up > using incorrect data. work in progress - (I am afraid I have to write an advanced prototype to be able to know exaclty how the cache might work, and so, what API we should have) > > 6. get_files() doesn't document whether the yielded paths are absolute or > relative, local or cross-platform, etc. I am fixing this as well >> I need to find back your comments for this part, I must have missed >> them. That's >> the last part I didn't work out yet on the current PEP revision. > > Well, if you can't find them, the EggFormats doc explains how these file/dir > structures are currently laid out by setuptools, easy_install, pip, etc., > and the PEP should probably reference that. work in progress Tarek -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 : Changing the .egg-info structure
On Tue, May 19, 2009 at 4:03 PM, Antoine Pitrou wrote: > Ronald Oussoren mac.com> writes: >> >> Wouldn't it be possible to use a CSV file for this? That way we >> wouldn't have to invent yet another escaping mechanism and there's >> already good suppport for reading and writing CSV files in the >> standard library. > > +1 > > We can even customize the delimiter if you want to make it more readable (or > if > there's a shortage of bikeshed material ;-)). +1 and the default csv delimiter "," makes it perfectly readable ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 : Changing the .egg-info structure
Tarek Ziadé wrote: On Sat, May 16, 2009 at 6:55 PM, P.J. Eby wrote: 1. Why ';' separation, instead of tabs as in PEP 262? Aren't semicolons a valid character in filenames? I am changing this into a . for now. What about Antoine's idea about doing a quote() on the names ? From my point of view seems more simple to deal with, if 3rd-party tools want to work on these files without using pkgutil or Python. 4. There should probably be a way to iterate over the projects in a directory, since it's otherwise impossible for an installation tool to find out what project(s) "own" a file that conflicts with something being installed. Alternatively, reshaping the file API to allow querying by path as well as by project might work. I am adding a "get_projects" api: get_projects() -> iterator Provides an iterator that will return (name, path) tuples, where `name` is the name of a registered project and `path` the path to its `egg-info` directory. But for the use case you are mentioning, what about an explicit API: get_owners(paths) -> sequence of project names returns a sequence of tuple. For each path in the "paths" list, a tuple of project names is returned 5. If any cache mechanisms are to be used by the API, the API *must* make it possible to bypass or explicitly manage that cache, as otherwise installation tools and tools that manipulate sys.path at runtime may end up using incorrect data. work in progress - (I am afraid I have to write an advanced prototype to be able to know exaclty how the cache might work, and so, what API we should have) 6. get_files() doesn't document whether the yielded paths are absolute or relative, local or cross-platform, etc. I am fixing this as well I need to find back your comments for this part, I must have missed them. That's the last part I didn't work out yet on the current PEP revision. Well, if you can't find them, the EggFormats doc explains how these file/dir structures are currently laid out by setuptools, easy_install, pip, etc., and the PEP should probably reference that. work in progress Is it Pythonic for the methods to starts with "get_", or should they be projects(), owners(), etc? ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 : Changing the .egg-info structure
2009/5/19 Tarek Ziadé : > On Sat, May 16, 2009 at 6:55 PM, P.J. Eby wrote: >> >> 1. Why ';' separation, instead of tabs as in PEP 262? Aren't semicolons a >> valid character in filenames? > > I am changing this into a . for now. I'm not following this thread at all, but can I put a strong vote *against* tabs in, please. You're just asking for bug reports from people who edit the file and expand tabs to spaces (either deliberately, or via an automatic editor setting they forgot about) and then can't see why a file that looks the same works differently. OK, so it's not meant to be a human editable file, but that won't stop some people :-) Paul ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 : Changing the .egg-info structure
At 04:04 PM 5/19/2009 +0200, Tarek Ziadé wrote: On Sat, May 16, 2009 at 6:55 PM, P.J. Eby wrote: > > 1. Why ';' separation, instead of tabs as in PEP 262? Aren't semicolons a > valid character in filenames? I am changing this into a . for now. What about Antoine's idea about doing a quote() on the names ? I like the CSV idea better, since the csv module is available in 2.3 and up. We should just pick a dialect with unambiguous quoting rules. From my point of view seems more simple to deal with, if 3rd-party tools want to work on these files without using pkgutil or Python. True, but then CSV files are still pretty common. One other possibility that might work is using a vertical bar as a separator. My preference rank at the moment is probably tabs, CSV, or vertical bar. But I don't really care all that much, so let the people who care decide. Personally, though, I don't see much point to cross-language manipulation of the file. System packaging tools have their own way of keeping track of this stuff. So unless somebody's using it to *build* system packages (e.g. making an RPM builder), they don't need this. Now, about the APIs... > 4. There should probably be a way to iterate over the projects in a > directory, since it's otherwise impossible for an installation tool to find > out what project(s) "own" a file that conflicts with something being > installed. Alternatively, reshaping the file API to allow querying by path > as well as by project might work. I am adding a "get_projects" api: get_projects() -> iterator Provides an iterator that will return (name, path) tuples, where `name` is the name of a registered project and `path` the path to its `egg-info` directory. But for the use case you are mentioning, what about an explicit API: get_owners(paths) -> sequence of project names returns a sequence of tuple. For each path in the "paths" list, a tuple of project names is returned > > 5. If any cache mechanisms are to be used by the API, the API *must* make it > possible to bypass or explicitly manage that cache, as otherwise > installation tools and tools that manipulate sys.path at runtime may end up > using incorrect data. work in progress - (I am afraid I have to write an advanced prototype to be able to know exaclty how the cache might work, and so, what API we should have) I think it would be simpler to have explicit object types representing things like a directory, a collection of directories, and individual projects, and these object types should be part of the API. Any function-oriented API should just be exposed as the methods of a default singleton. Other Python modules follow this pattern -- and it's what I copied for the pkg_resources design. It gives a nice tradeoff between keeping the simple things simple, and complex things possible, as well as keeping mechanism and policy separate. Right now, the API design you're trying to do is being burdened by using strings and tuples to represent things that could just as easily be objects with their own methods, instead of things you have to pass back into other APIs. This also makes caching more complex, because you can't just have one main object with stuff hanging off; you've got to have a bunch of dictionaries, tuples, lists, sets, etc. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] IronPython specific code in inspect module
Hello all, The inspect module (inspect.get_argspec etc) work fine for Python functions and classes in IronPython, but they don't work on .NET types which don't have the Python function attributes like im_func etc. I have IronPython specific versions of several of these functions which use .NET reflection and inspect could fallback to if sys.platform == 'cli'. Would it be ok for me to add these to the inspect module? Obviously the tests would only run on IronPython... The behaviour for CPython would be unaffected. All the best, Michael Foord -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IronPython specific code in inspect module
2009/5/19 Michael Foord : > I have IronPython specific versions of several of these functions which use > .NET reflection and inspect could fallback to if sys.platform == 'cli'. > Would it be ok for me to add these to the inspect module? Obviously the > tests would only run on IronPython... The behaviour for CPython would be > unaffected. I wish we had more of a policy about this. There seems to be a long tradition of special casing other implementations in the stdlib. For example, see types.py and tests/test_support.py for remnants of Jython compatibility. However, I suspect this code has languished with out core-developers using the trunk stdlib with Jython. I suppose this is a good reason why we are going to split the stdlib out of the main repo. However that still leaves the question of how to handle putting code like this in. Should we ask that all code be implementation-independent as much as possible from the original authors? Do all all changes against the stdlib have to be run against several implementations? Should we sprinkle if switches all over the codebase for different implementations, or should new support files be added? -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IronPython specific code in inspect module
On Tue, May 19, 2009 at 7:26 PM, Benjamin Peterson wrote: > 2009/5/19 Michael Foord : >> I have IronPython specific versions of several of these functions which use >> .NET reflection and inspect could fallback to if sys.platform == 'cli'. >> Would it be ok for me to add these to the inspect module? Obviously the >> tests would only run on IronPython... The behaviour for CPython would be >> unaffected. > > I wish we had more of a policy about this. There seems to be a long > tradition of special casing other implementations in the stdlib. For > example, see types.py and tests/test_support.py for remnants of Jython > compatibility. However, I suspect this code has languished with out > core-developers using the trunk stdlib with Jython. I suppose this is > a good reason why we are going to split the stdlib out of the main > repo. > > However that still leaves the question of how to handle putting code > like this in. Should we ask that all code be > implementation-independent as much as possible from the original > authors? Do all all changes against the stdlib have to be run against > several implementations? Should we sprinkle if switches all over the > codebase for different implementations, or should new support files be > added? > >From my observation (mostly according to jython), such changes easily get out >of sync. The net result is that you have one, outdated, version in stdlib and other implementation, like IronPython is maintaining it's own anyway. IMO it's easy enough to maintain clearly implementation-specific parts out of cpython's stdlib. What I would rather like to see is that stdlib does not contain impl specific parts, even for cpython and cpython maintains it's own things outside of stdlib. This would be in line with what we discussed at pycon I think, please correct me if I'm wrong. Cheers, fijal ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IronPython specific code in inspect module
On Tue, May 19, 2009 at 9:26 PM, Benjamin Peterson wrote: > 2009/5/19 Michael Foord : >> I have IronPython specific versions of several of these functions which use >> .NET reflection and inspect could fallback to if sys.platform == 'cli'. >> Would it be ok for me to add these to the inspect module? Obviously the >> tests would only run on IronPython... The behaviour for CPython would be >> unaffected. > > I wish we had more of a policy about this. There seems to be a long > tradition of special casing other implementations in the stdlib. For > example, see types.py and tests/test_support.py for remnants of Jython > compatibility. However, I suspect this code has languished with out > core-developers using the trunk stdlib with Jython. I suppose this is > a good reason why we are going to split the stdlib out of the main > repo. > > However that still leaves the question of how to handle putting code > like this in. Should we ask that all code be > implementation-independent as much as possible from the original > authors? Do all all changes against the stdlib have to be run against > several implementations? Should we sprinkle if switches all over the > codebase for different implementations, or should new support files be > added? > It seems that using a technique similar to dependency injection could provide some value. DI allows implementations conforming to some interface to be injected into a running application without the messy construction logic. The simple construction-by-hand pattern is to create the dependencies and pass them into the dependent objects. Frameworks build on top of this to allow the dependencies to be wired together without having any construction logic in code, like switch statements, to do the wiring. I think a similar pattern could be used in the standard library. When the interpreter goes through its normal bootstrapping process in can just execute a module provided by the vendor that specifies the platform specific implementations. Some defaults can be provided since Python already has a bunch of platform specific implementations. An over simplified design to make this happen may look like: 1. Create a simple configuration that allows a mapping of interfaces to implementations. This is where the vendor would say when using inspect you really should be using cli.inspect. 2. Add executing this new configuration to the bootstrapping process. 3. Add generic hooks into the library where needed to load the dependency instead of platform specific if statements. 4. Rip out the platform specific code that is hidden in the if statements and use that as the basis for the sane injected defaults. 5. Document the interfaces for each component that can be changed by the vendor. -- David blog: http://www.traceback.org twitter: http://twitter.com/dstanek ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IronPython specific code in inspect module
2009/5/19 Maciej Fijalkowski : > From my observation (mostly according to jython), such changes easily get out > of > sync. The net result is that you have one, outdated, version in stdlib > and other implementation, like IronPython is maintaining it's own > anyway. IMO it's easy enough > to maintain clearly implementation-specific parts out of cpython's stdlib. Hopefully, it will be easier to visualize how this might work once the plan for hg migration is finalized. > > What I would rather like to see is that stdlib does not contain impl > specific parts, > even for cpython and cpython maintains it's own things outside of stdlib. This > would be in line with what we discussed at pycon I think, please correct me if > I'm wrong. I was not present, but that's my impression, too. -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IronPython specific code in inspect module
Michael Foord wrote: > I have IronPython specific versions of several of these functions which > use .NET reflection and inspect could fallback to if sys.platform == > 'cli'. Would it be ok for me to add these to the inspect module? > Obviously the tests would only run on IronPython... The behaviour for > CPython would be unaffected. What about instead defining __argspec__ for built-in functions/method objects and allowing all the implementations to implement it? We could all agree to return: [ (return_type, (arg_types,...)), (return_type, (arg_types,...)), ] Then inspect can check for that attribute and support introspection on built-ins. This would be an easy feature for us to implement and it may also be for Jython as well given that we both get the power of our platforms reflection capabilities. Any platform that implements it lights up w/o new platform specific code. And maybe this needs to go to python-ideas now :) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IronPython specific code in inspect module
Dino Viehland wrote: > What about instead defining __argspec__ for built-in functions/method > objects and allowing all the implementations to implement it? We could > all agree to return: > > [ > (return_type, (arg_types,...)), > (return_type, (arg_types,...)), > ] > > Then inspect can check for that attribute and support introspection on > built-ins. This would be an easy feature for us to implement and it > may also be for Jython as well given that we both get the power of our > platforms reflection capabilities. Any platform that implements it > lights up w/o new platform specific code. And maybe this needs to go > to python-ideas now :) Curiously, inspect limitations on CPython (can't inspect functools.partial, has issues with some descriptors and decorators) got us chatting about PEP 362: Function Signature Object[0] on #python-dev today. PEP 362 was also brought up in a recent thread where the executive summary was 'it just needs someone to guide it through the last steps'[1], and it would make this kind of introspection nice and clean[2]. It makes even more sense now we have PEP 3107: Function Annotations[3] in place. Cheers, Daniel [0] http://www.python.org/dev/peps/pep-0362/ [1] http://mail.python.org/pipermail/python-dev/2009-April/088517.html [2] http://mail.python.org/pipermail/python-dev/2009-April/088597.html [3] http://www.python.org/dev/peps/pep-3107/#parameters ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
