Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
I am in full agreement with Tarek here. At ActiveState, we maintain our own index that differs from PyPI in two ways (among others): I think you are saying something very different from what Tarek says. IIUC, you are saying that egg-info is ill-defined and may cause subtle problems. So you are saying there is a problem for the users of the data. I could live with that - it's the user's choice to use these data, after all. Tarek is saying that it will be evil and bad for the community to unpack some zip files. I find that statement itself counter-productive. Ideally, in future - I should be able to query static metadata (with environment markers[2] and such) for *any* package from PyPI. And this static metadata is simply a DIST-INFO file (instead of being a directory with a bunch of files in it). I don't really see a point in providing access to, say, the list of entry points of each package. Again, that is completely different from what Tarek is saying. You said (just now, and literally): "I don't really see a point". Converting this to the -1/0/+1 system, it sounds like +0 to me: you are unlikely to ever use this data. This is perfectly fine with me: I won't use the data myself, either. However, I fail to see why this should stop me from providing the data, when there are people requesting this feature over and over again. I'd like to see some of Python's "consenting adults" attitude here. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Some news from my sandbox project
Victor Stinner wrote: I'm still developing irregulary my sandbox project since last june. Today, the biggest problem is the creation of a read only view of the __builtins__ dictionary. Why do you think you need to do this? What form of attack would a writable __builtins__ expose you to that would be prevented by making it read-only? Seems to me that the only way you could exploit a writable __builtins__ would be to put a function in there that does something bad. But if you can create or obtain such a function in the first place, you can just call it directly. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Some news from my sandbox project
On Sat, Sep 18, 2010 at 8:24 PM, Greg Ewing wrote: > Victor Stinner wrote: > >> I'm still developing irregulary my sandbox project since last june. > >> Today, the biggest problem is the creation of a read only view of the >> __builtins__ dictionary. > > Why do you think you need to do this? What form of attack > would a writable __builtins__ expose you to that would be > prevented by making it read-only? > > Seems to me that the only way you could exploit a writable > __builtins__ would be to put a function in there that does > something bad. But if you can create or obtain such a > function in the first place, you can just call it directly. __builtins__ is in everyone's global namespace, so if it can be mutated, different python programs running in the same sandbox can affect each other. Ditto sys.modules and os environ, but I guess that those are already addressed. -Rob ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] egg_info in PyPI
Hi there, I'm going to add my own 2 cents to the discussion as I'm involved in the matter here at the DZUG conference. Tarek Ziadé wrote: > Now you want to publish another metadata format at PyPI ? If PyPI takes > that direction and adopts, promotes and publishes a standard that is not > the one we worked on in the past year, this will increase our difficulty > to push the new format so its adopted by the tools then the community. Actually, I see it being the other way around: People use what used to work for them and most people won't switch to using the new standard just for the fun of it. In order for the new standard to acquire some momentum, people need to be interested in the subject the standard is about and see some concrete advantage in switching. Exposing existing metadata will make it possible to build new applications that use metadata information; it's those applications' responsibility then to promote the new standard. Let me take the tl.eggdeps package as a concrete example. The package currently analyses dependencies between installed packages. I'd like to expand it to analyse dependencies between any packages on PyPI but I can't as long as dependency information is not available without actually installing things. Exposing PEP 345 metadata on PyPI won't be of any practical use until a critical number of packages actually specify that information. I simply won't bother implementing the feature either until some point far in the future or unless I can use legacy metadata as a fall-back. Having the legacy metadata available enables me to build the feature now; this could raise interest in metadata as such, and I might even make visually apparent which dependencies have been specified by PEP 345 means and which I could only guess from legacy information. > People will just get confused because they will find two competing > metadata formats That's exactly the situation where we were at, and > that's exactly where I don't want to go back. That's simply a matter of documentation. It has to be clearly communicated that the legacy information is provided as a transitional service and that users of that metadata use it only until the newly specified information is available for a critical number of packages (the exact value of which may of course depend on the application in question). > I am not even understanding what's the benefit of doing this since an > egg_info directory is obtained at *build* time and can differ from a > machine to another, so it seems pretty useless for me to publish this. We do understand that legacy metadata has its issues, but taken with the appropriate amount of salt, it's better than nothing in most cases, enough so at least to solve some chicken-and-egg problems. > We worked hard to build some standards, but if PyPI doesn't help us here, > everything we did and are doing is useless. Nothing that we're talking about in this thread will contradict what you want to achieve and what work you've done. By the reasoning I tried to explain above, what PyPI is going to do will actually help you. OTOH, I personally don't think it would be a constructive way of helping you if PyPI tried to enforce the new standard by keeping people from using information they may at the moment only get through legacy means. -- Thomas ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
On 18/09/2010 08:52, "Martin v. Löwis" wrote: I am in full agreement with Tarek here. At ActiveState, we maintain our own index that differs from PyPI in two ways (among others): I think you are saying something very different from what Tarek says. IIUC, you are saying that egg-info is ill-defined and may cause subtle problems. So you are saying there is a problem for the users of the data. I could live with that - it's the user's choice to use these data, after all. Tarek is saying that it will be evil and bad for the community to unpack some zip files. I find that statement itself counter-productive. With the distutils2 work very close to landing in the standard library, providing infrastructure that will cause tools to *depend* on the old formats is a very bad idea. If tool use this metadata then it could well prevent packages that want to be compatible with these tools from using distutils2. What PyPI does effectively becomes "the standard" for a large chunk of the Python world (which is why changing the format PyPI provides data in can be so hard). Now seems a really dumb time to bless the setuptools metadata format as the defacto standard after so much work has gone into replacing it and that effort is so close to completion. So - I agree with Tarek. Exposing this information on PyPI would be problematic for the Python community. Not only does the data have the problems that Tarek and Sridhar point out, but it would actively hinder adoption of the better replacement. All the best, Michael Foord Ideally, in future - I should be able to query static metadata (with environment markers[2] and such) for *any* package from PyPI. And this static metadata is simply a DIST-INFO file (instead of being a directory with a bunch of files in it). I don't really see a point in providing access to, say, the list of entry points of each package. Again, that is completely different from what Tarek is saying. You said (just now, and literally): "I don't really see a point". Converting this to the -1/0/+1 system, it sounds like +0 to me: you are unlikely to ever use this data. This is perfectly fine with me: I won't use the data myself, either. However, I fail to see why this should stop me from providing the data, when there are people requesting this feature over and over again. I'd like to see some of Python's "consenting adults" attitude here. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Some news from my sandbox project
Le samedi 18 septembre 2010 10:24:49, Greg Ewing a écrit :
> Victor Stinner wrote:
> > I'm still developing irregulary my sandbox project since last june.
> >
> > Today, the biggest problem is the creation of a read only view of the
> > __builtins__ dictionary.
>
> Why do you think you need to do this? What form of attack
> would a writable __builtins__ expose you to that would be
> prevented by making it read-only?
Replace builtin functions make it possible to modify (indirectly) functions
outside the sandbox. Eg. one of the most important function of pysandbox is
proxy() (a function to create a read only view of a object outside the
sandbox, especially on an import), if you replace isinstance() by a function
which always return True: you can get unmodified objects (read+write access).
So it's possible to get the real file type from sys.stdout and so open a new
file.
The example on a modified version of pysandbox (to get the vulnerability):
sandbox>>> __builtins__['isinstance']=lambda obj, cls: True
sandbox>>> import sys
sandbox>>> type(sys.stdout)('/etc/issue').read()
'Debian GNU/Linux squeeze/sid \\n \\l\n\n'
---
It would be possible to create a local copy of each builtin in all pysandbox
functions, but it doesn't protect functions outside pysandbox.
--
Victor Stinner
http://www.haypocalc.com/
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
With the distutils2 work very close to landing in the standard library, providing infrastructure that will cause tools to *depend* on the old formats is a very bad idea. I think you are misunderstanding. The infrastructure will *not* depend on the old formats. Instead, packaging that have that information will provide it, packages that don't will not. The infrastructure is entirely agnostic on whether the data is available or not. In particular, it will not try to interpret the data in any way. If tool use this metadata then it could well prevent packages that want to be compatible with these tools from using distutils2. I don't think this can well happen. In most known use cases, the tools could support both forms of metadata. It may be that tools want to use information that is not provided by PEP 345. However, the tool will then likely fall back to just not having the information, as even existing packages only provide that information occasionally. What PyPI does effectively becomes "the standard" for a large chunk of the Python world (which is why changing the format PyPI provides data in can be so hard). Now seems a really dumb time to bless the setuptools metadata format as the defacto standard after so much work has gone into replacing it and that effort is so close to completion. Please understand that the information is not being "blessed" (in any sense of the word that comes out of the dictionary). It's just being made available slightly more conveniently - please understand that it was available all the years already. So - I agree with Tarek. Exposing this information on PyPI would be problematic for the Python community. Not only does the data have the problems that Tarek and Sridhar point out, but it would actively hinder adoption of the better replacement. That's really sad. So people will have to wait a few years to efficiently implement tools that they could implement today. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Some news from my sandbox project
Le samedi 18 septembre 2010 10:39:58, Robert Collins a écrit : > __builtins__ is in everyone's global namespace, so if it can be > mutated, different python programs running in the same sandbox can > affect each other. > > Ditto sys.modules and os environ, but I guess that those are already > addressed. sys.modules and os.environ are not accessible in pysandbox :-) If you create a rule in the security policy to allow them, you will get read only views. Example with sys.modules: sandbox>>> from sys import modules sandbox>>> modules['sys'] TypeError: Unable to proxy a value of type sandbox>>> modules['sys']=1 SandboxError: Read only object Builtin module type is blocked because it is unsafe. -- Victor Stinner http://www.haypocalc.com/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r84877 - python/branches/release27-maint/Lib/posixpath.py
Le samedi 18 septembre 2010 07:40:44, hirokazu.yamamoto a écrit : > Author: hirokazu.yamamoto > Date: Sat Sep 18 07:40:44 2010 > New Revision: 84877 > > Log: > Added missing import. > > Modified: python/branches/release27-maint/Lib/posixpath.py > === > === --- python/branches/release27-maint/Lib/posixpath.py (original) > +++ python/branches/release27-maint/Lib/posixpath.py Sat Sep 18 07:40:44 > 2010 @@ -11,6 +11,7 @@ > """ > > import os > +import sys Woops, thanks :-) -- Victor Stinner http://www.haypocalc.com/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
On 18/09/2010 11:03, "Martin v. Löwis" wrote: With the distutils2 work very close to landing in the standard library, providing infrastructure that will cause tools to *depend* on the old formats is a very bad idea. I think you are misunderstanding. The infrastructure will *not* depend on the old formats. Instead, packaging that have that information will provide it, packages that don't will not. The infrastructure is entirely agnostic on whether the data is available or not. In particular, it will not try to interpret the data in any way. No, not at all. If tools *use* the information (and if they don't then what is the point?) then packages that want to be compatible with those tools will have to provide this information. That will require those packages to use setuptools and prevent them using distutils2. By providing information in this format PyPI will (like it or not) be blessing this format as the 'standard' way of providing this information. That I think is a very bad idea at this point in the evolution of Python packaging tools. If tool use this metadata then it could well prevent packages that want to be compatible with these tools from using distutils2. I don't think this can well happen. In most known use cases, the tools could support both forms of metadata. Well, a) I would like to see that demonstrated and b) having one standard is *far* preferable and having the distutils2 format be that standard is also far preferable. Please wait a bit (or start on supporting the distutils2 metadata format now). It may be that tools want to use information that is not provided by PEP 345. However, the tool will then likely fall back to just not having the information, as even existing packages only provide that information occasionally. What PyPI does effectively becomes "the standard" for a large chunk of the Python world (which is why changing the format PyPI provides data in can be so hard). Now seems a really dumb time to bless the setuptools metadata format as the defacto standard after so much work has gone into replacing it and that effort is so close to completion. Please understand that the information is not being "blessed" (in any sense of the word that comes out of the dictionary). It's just being made available slightly more conveniently - please understand that it was available all the years already. I'm sorry but this is wrong. By providing this information about packages in the *standard* Python package repository it is very much (whether intentionally or not) blessing it as a standard. So - I agree with Tarek. Exposing this information on PyPI would be problematic for the Python community. Not only does the data have the problems that Tarek and Sridhar point out, but it would actively hinder adoption of the better replacement. That's really sad. So people will have to wait a few years to efficiently implement tools that they could implement today. Why a few years? All the best, Michael Regards, Martin -- http://www.ironpythoninaction.com/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
On Sat, Sep 18, 2010 at 7:39 PM, Michael Foord wrote: > On 18/09/2010 11:03, "Martin v. Löwis" wrote: >> That's really sad. So people will have to wait a few years to efficiently >> implement tools that they could implement today. > > Why a few years? That's the time it will take for all packages to support distutils2 ? cheers, David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
On 18/09/2010 11:48, David Cournapeau wrote: On Sat, Sep 18, 2010 at 7:39 PM, Michael Foord wrote: On 18/09/2010 11:03, "Martin v. Löwis" wrote: That's really sad. So people will have to wait a few years to efficiently implement tools that they could implement today. Why a few years? That's the time it will take for all packages to support distutils2 ? Not "all packages" support setuptools. Michael cheers, David -- http://www.ironpythoninaction.com/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
On Sat, Sep 18, 2010 at 7:50 PM, Michael Foord wrote: > On 18/09/2010 11:48, David Cournapeau wrote: >> >> On Sat, Sep 18, 2010 at 7:39 PM, Michael Foord >> wrote: >>> >>> On 18/09/2010 11:03, "Martin v. Löwis" wrote: That's really sad. So people will have to wait a few years to efficiently implement tools that they could implement today. >>> >>> Why a few years? >> >> That's the time it will take for all packages to support distutils2 ? > > Not "all packages" support setuptools. Sure, but supporting setuptools was kind of possible for packages relying heavily on distutils, even if it was not simple and fragile. Distutils2 being incompatible API-wise with distutils, I am not sure it will be as "easy" as with setuptools. It may be, but the only way to know is to do it, and the incentive rather unclear. It means that anyway, a lot of infrastructure will have to support both "standards" for the time being. cheers, David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
I think you are misunderstanding. The infrastructure will *not* depend on the old formats. Instead, packaging that have that information will provide it, packages that don't will not. The infrastructure is entirely agnostic on whether the data is available or not. In particular, it will not try to interpret the data in any way. No, not at all. If tools *use* the information (and if they don't then what is the point?) then packages that want to be compatible with those tools will have to provide this information. Not true. Tools could/should also support PEP 345 data, and then they can support either kind of package. By providing information in this format PyPI will (like it or not) be blessing this format as the 'standard' way of providing this information. By that definition, *both* formats are "blessed". The PEP 345 data is already blessed. Depending on the definition of "providing", the egg-info data are also already "provided", ever since PyPI started accepting egg files. I don't think this can well happen. In most known use cases, the tools could support both forms of metadata. Well, a) I would like to see that demonstrated The tool in question is tl.eggdepend. It can easily support both kinds of metadata. and b) having one standard is *far* preferable and having the distutils2 format be that standard is also far preferable. Please wait a bit (or start on supporting the distutils2 metadata format now). The latter is already the case: the distutils2 metadata *is* supported *now*. It's just that no package is using it (except for pep345demo). As for a bit: how long exactly? That's really sad. So people will have to wait a few years to efficiently implement tools that they could implement today. Why a few years? Because it will take that long until a significant number of packages will use distutils 2. People still use very old versions of packaging tools (e.g. the ones that come with Debian); it will just take time to get the new tools and API adopted. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
On 18/09/2010 12:25, "Martin v. Löwis" wrote: I think you are misunderstanding. The infrastructure will *not* depend on the old formats. Instead, packaging that have that information will provide it, packages that don't will not. The infrastructure is entirely agnostic on whether the data is available or not. In particular, it will not try to interpret the data in any way. No, not at all. If tools *use* the information (and if they don't then what is the point?) then packages that want to be compatible with those tools will have to provide this information. Not true. Tools could/should also support PEP 345 data, and then they can support either kind of package. But you are proposing that PyPI will provide an interface / API for exposing the setuptools information. So tools that get metadata from PyPI in this way (and depend on it - as they will if you expose it) will have functionality that only works for packages that provide this information in this form. Saying tools "should" work with other formats too is empty words. By providing information in this format PyPI will (like it or not) be blessing this format as the 'standard' way of providing this information. By that definition, *both* formats are "blessed". The PEP 345 data is already blessed. Depending on the definition of "providing", the egg-info data are also already "provided", ever since PyPI started accepting egg files. No. See above comment. If exposing this information has no value then don't do it. If it does have value, then we are blessing it - and therefore blessing it *over* other formats. I accept this is not your intention. It *will* be the effect. I don't think this can well happen. In most known use cases, the tools could support both forms of metadata. Well, a) I would like to see that demonstrated The tool in question is tl.eggdepend. It can easily support both kinds of metadata. I couldn't find any references "tl.eggdepend" on the web. It is a minor point though. and b) having one standard is *far* preferable and having the distutils2 format be that standard is also far preferable. Please wait a bit (or start on supporting the distutils2 metadata format now). The latter is already the case: the distutils2 metadata *is* supported *now*. As in exported by PyPI though an API / interface? If not then again, irrelevant. Tools that use the new interface you are proposing *won't* use that information. Saying that they *could* is more empty words if our *actions* promote the use of another format. It's just that no package is using it (except for pep345demo). As for a bit: how long exactly? Distutils2 1a2 will be released in the next few days. That's really sad. So people will have to wait a few years to efficiently implement tools that they could implement today. Why a few years? Because it will take that long until a significant number of packages will use distutils 2. People still use very old versions of packaging tools (e.g. the ones that come with Debian); it will just take time to get the new tools and API adopted. Promoting another format in preference to distutils2 will very much prolong that. Michael Regards, Martin -- http://www.ironpythoninaction.com/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
No. See above comment. If exposing this information has no value then don't do it. If it does have value, then we are blessing it - and therefore blessing it *over* other formats. No: not *over*. Only over formats that don't get exposed. However, the PEP 345 data are *already* exposed, via HTML, JSON, XML-RPC. So they are much more prominently presented than what I'm proposing to do. I fail to see why just extracting the egg-info would be exposing it *over* the PEP 345 data. The tool in question is tl.eggdepend. It can easily support both kinds of metadata. I couldn't find any references "tl.eggdepend" on the web. It is a minor point though. Oops, see http://pypi.python.org/pypi/tl.eggdeps As in exported by PyPI though an API / interface? Sure. See, for example, http://pypi.python.org/pypi/pep345demo/json It's also available through the XML-RPC release_data. Saying that they *could* is more empty words if our *actions* promote the use of another format. But they do. Please stop saying that they might not, when they actually do (and have been for a while). It's just that no package is using it (except for pep345demo). As for a bit: how long exactly? Distutils2 1a2 will be released in the next few days. Sure. But when can tools computing dependencies for widely used packages actually expect that these metadata will be available? Promoting another format in preference to distutils2 will very much prolong that. IT WILL BE NOT IN PREFERENCE TO DISTUTILS2. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] egg_info in PyPI
On 9/18/2010 9:21 AM, "Martin v. Löwis" wrote: > IT WILL BE NOT IN PREFERENCE TO DISTUTILS2. No need to shout. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
Ok, I'm sorry - PEP 345 information is available via the PyPI API. (So exposing egg_info would not be promoting it *over* distutils2 but it would still be promoting and blessing it). Tarek's main point still stands though. The dependency information in the egg_info is tied to the platform and Python version that the package was *built* on. Neither pip nor easy_install use the egg_info to determine this; instead they re-generate it to get the correct information for the target platform. So if the use case is to provide dependency information exposing egg_info is not the right way to do it - and tools that use it will be using potentially (and frequently) inaccurate information. I stand by the point that once we start providing this information tools will start using it, and they shouldn't be. (So your question "when can tools computing dependencies for widely used packages actually expect that these metadata will be available?" is not answered by providing access to egg_info.) This problem (static availability of dependency information) is *exactly* the problem PEP 345 solves, so we should be pushing for progress on this front (distutils2a1 is already out and distutils2a2 will be out soon). All the best, Michael Foord On 18/09/2010 14:21, "Martin v. Löwis" wrote: No. See above comment. If exposing this information has no value then don't do it. If it does have value, then we are blessing it - and therefore blessing it *over* other formats. No: not *over*. Only over formats that don't get exposed. However, the PEP 345 data are *already* exposed, via HTML, JSON, XML-RPC. So they are much more prominently presented than what I'm proposing to do. I fail to see why just extracting the egg-info would be exposing it *over* the PEP 345 data. The tool in question is tl.eggdepend. It can easily support both kinds of metadata. I couldn't find any references "tl.eggdepend" on the web. It is a minor point though. Oops, see http://pypi.python.org/pypi/tl.eggdeps As in exported by PyPI though an API / interface? Sure. See, for example, http://pypi.python.org/pypi/pep345demo/json It's also available through the XML-RPC release_data. Saying that they *could* is more empty words if our *actions* promote the use of another format. But they do. Please stop saying that they might not, when they actually do (and have been for a while). It's just that no package is using it (except for pep345demo). As for a bit: how long exactly? Distutils2 1a2 will be released in the next few days. Sure. But when can tools computing dependencies for widely used packages actually expect that these metadata will be available? Promoting another format in preference to distutils2 will very much prolong that. IT WILL BE NOT IN PREFERENCE TO DISTUTILS2. Regards, Martin -- http://www.ironpythoninaction.com/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
On Sat, Sep 18, 2010 at 4:01 PM, Michael Foord wrote: > Ok, I'm sorry - PEP 345 information is available via the PyPI API. (So > exposing egg_info would not be promoting it *over* distutils2 but it would > still be promoting and blessing it). > > Tarek's main point still stands though. The dependency information in the > egg_info is tied to the platform and Python version that the package was > *built* on. Neither pip nor easy_install use the egg_info to determine this; > instead they re-generate it to get the correct information for the target > platform. > > So if the use case is to provide dependency information exposing egg_info is > not the right way to do it - and tools that use it will be using potentially > (and frequently) inaccurate information. I stand by the point that once we > start providing this information tools will start using it, and they > shouldn't be. (So your question "when can tools computing dependencies for > widely used packages actually expect that these metadata will be available?" > is not answered by providing access to egg_info.) > > This problem (static availability of dependency information) is *exactly* > the problem PEP 345 solves, so we should be pushing for progress on this > front (distutils2a1 is already out and distutils2a2 will be out soon). Sorry to take back the discussion late I was away for a while. My main concern, like Michael says, is to see PyPI publish some information that are known to be accurate only on one platform, the one that is used to create the egg_info. Granted, most of the time the information is the same no matter what the platform is, but still: you don't know if this is the case when you read those metadata . The only way to do it properly is to re-run egg_info. So, right now Pip and easy_install are running egg_info to re-build those data before they start the install process. They re-generate the egg_info every time, and then browse the list of dependencies to get them. If you upload the egg_info data at PyPI so people can browse the dependencies, you basically make the assumption that the egg_info produced will work no matter what the platform is, unless you associate with the egg_info the platform that was used. So, I don't understand what is the benefit here, since a serious installer will re-run egg_info every time. Coming from the Plone/buildout community, I would be concerned if buildout would rely on this. json, ldap, mysql you name it --I have tons of other examples-- all those dependencies will not be accurate unless you re-run setup.py egg_info. "Good enough metadata" sounds completely wrong to me. Cheers Tarek -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
So if the use case is to provide dependency information exposing egg_info is not the right way to do it - and tools that use it will be using potentially (and frequently) inaccurate information. I stand by the point that once we start providing this information tools will start using it, and they shouldn't be. I think this is precisely the flaw in this line of reasoning: that we refuse a service because we think the data is useless. Why can't the users decide for themselves whether the data is useless? As Thomas points out, people running into this will actually have a reason to migrate to PEP 345. So exposing this presumed-incorrect information will actually help promoting PEP 345. This problem (static availability of dependency information) is *exactly* the problem PEP 345 solves, so we should be pushing for progress on this front (distutils2a1 is already out and distutils2a2 will be out soon). And I don't mind pushing it, unless that means to deny users service now based on the promise of offering better service tomorrow. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
So, I don't understand what is the benefit here, since a serious installer will re-run egg_info every time. I think the main applications that people are after are not builds. They want the dependency information without downloading the packages, and dependency information for packages they have no plans to install. In the specific case of tl.eggdeps, the dependency information is only used to create printable graphs. If this turns out to be slightly incorrect, people would notice if they try to use the packages in question. Coming from the Plone/buildout community, I would be concerned if buildout would rely on this. json, ldap, mysql you name it --I have tons of other examples-- all those dependencies will not be accurate unless you re-run setup.py egg_info. "Good enough metadata" sounds completely wrong to me. Still, people ask for it. I'm fine with telling them that the data is flawed for various reasons. I object to denying them the data, and I really dislike having to discard the patch that I already wrote to implement it. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
2010/9/18 "Martin v. Löwis" : >> So, I don't understand what is the benefit here, since a serious >> installer will re-run egg_info every time. > > I think the main applications that people are after are not builds. > They want the dependency information without downloading the packages, > and dependency information for packages they have no plans to install. Yes they want to build the graph of dependencies, which will be potentially false, as I explained. > > In the specific case of tl.eggdeps, the dependency information is only > used to create printable graphs. If this turns out to be slightly incorrect, > people would notice if they try to use the packages in > question. So you are fine with publishing "slightly incorrect" metadata at PyPI ? I am not. > >> Coming from the Plone/buildout community, I would be concerned if >> buildout would rely on this. json, ldap, mysql you name it --I have >> tons of other examples-- all those dependencies will not be accurate >> unless you re-run setup.py egg_info. "Good enough metadata" sounds >> completely wrong to me. > > Still, people ask for it. I'm fine with telling them that the data > is flawed for various reasons. I object to denying them the data, > and I really dislike having to discard the patch that I already > wrote to implement it. The can have those data, by downloading the tarball of the project and running the egg_info command. They will in this case have accurate data in fact. So right now you don't deny them the data, it's one command away from them. I don't understand the rational behind providing flawed data at PyPI. I am -1 on that. Regards Tarek > > Regards, > Martin > -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
At 05:19 PM 9/18/2010 +0200, Martin v. Löwis wrote: In the specific case of tl.eggdeps, the dependency information is only used to create printable graphs. If this turns out to be slightly incorrect, people would notice if they try to use the packages in question. By the way, just providing this information for .egg files and *not* for sdists would ensure accuracy of the metadata for that platform/python version. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
Tarek Ziadé writes: > "Good enough metadata" sounds completely wrong to me. I hate to break it to you, but experience shows that the XEmacs package system, whose dependency tracking is in theory a pile of braindamaged rubbish, an abomination in the sight of She Who Created The World With 4-Space Indentation, is definitely good enough to be an big improvement over nothing at all. I don't know why people would choose to implement their tools based on egg_info rather than PEP 345, but if there's some momentum in that direction already, those developers and users are not going to be pleased to hear "we're sorry, but what's good enough for you is not allowed on PyPi." If PEP 345 is as much better than previous frameworks as it sounds to be, you really shouldn't have much trouble building momentum. In the meantime, it's better to let people using a competing "standard" (even if it's neither very good nor a "real" standard) do their thing until they see the light. If you can't pass that test, then the question "what was all our work on PEP 345 for, anyway" becomes quite acute, but not in the way you're using it. :-/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
So you are fine with publishing "slightly incorrect" metadata at PyPI ? I am not. I really have no intuition for in how many cases the data will be incorrect. However, if users find that the data is incorrect for specific package, they ought to complain to the maintainer. I don't understand the rational behind providing flawed data at PyPI. I am -1 on that. -1 sounds much better than vetoing it :-) Taking the votes together, I currently arrive at Tarek -1 Michael -1 (?) Jannis -1 (?) Jim +1 Thomas +1 Any other opinions? Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
Am 18.09.10 17:49, schrieb P.J. Eby: At 05:19 PM 9/18/2010 +0200, Martin v. Löwis wrote: In the specific case of tl.eggdeps, the dependency information is only used to create printable graphs. If this turns out to be slightly incorrect, people would notice if they try to use the packages in question. By the way, just providing this information for .egg files and *not* for sdists would ensure accuracy of the metadata for that platform/python version. True (I presume - unless there are also dependencies on the specific OS version or system installation that may affect the metadata). OTOH, I do think that the users asking for that prefer per-release information, despite the limitations that this may have. OTTH, if the concerns could be relieved if egg-info would provided for all files that have it, I could provide that as well/instead. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
Stephen J. Turnbull wrote: > In the meantime, it's better to let people using a competing "standard" > (even if it's neither very good nor a "real" standard) do their thing > until they see the light. It's not even about the people who consume egg-info data seeing the light, it's about PEP-345 data actually being available. It's package authors who have to see the light, if you want to put it that way. In the case of the dependency graphs, "good enough" is actually good enough as long as "100% correct" is not available. I am actually quite aware of the existence of PEP 345 and its advantages over legacy egg-info data and I wouldn't build an installer based on egg-info data straight from PyPI, but I feel it's good enough for my application that aims to visualise dependencies that people "know anyway". I have the following possibilities: - build graphs from PEP-345 info exclusively, which currently gives me a single graph with a single node (not useful now or for a long time), - find a volunteer who fills in PEP-345 information for all existing packages (not likely to happen), - give up and have a beer instead of building the application (not satisfying in the long run), - use PEP-345 information wherever possible and fall back to egg-info data otherwise, acknowledging and advertising the fact that the fall-back data may not be 100% correct and having the package author specify PEP-345 information would be the way to go (good enough for my use case). -- Thomas ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
At 06:06 PM 9/18/2010 +0200, Martin v. Löwis wrote: Am 18.09.10 17:49, schrieb P.J. Eby: At 05:19 PM 9/18/2010 +0200, Martin v. Löwis wrote: In the specific case of tl.eggdeps, the dependency information is only used to create printable graphs. If this turns out to be slightly incorrect, people would notice if they try to use the packages in question. By the way, just providing this information for .egg files and *not* for sdists would ensure accuracy of the metadata for that platform/python version. True (I presume - unless there are also dependencies on the specific OS version or system installation that may affect the metadata). No, because an egg's egg-info is what it is. easy_install doesn't rebuild that information, so it is correct by definition. ;-) (Certainly, it is what it will be used for dependency information.) OTOH, I do think that the users asking for that prefer per-release information, despite the limitations that this may have. OTTH, if the concerns could be relieved if egg-info would provided for all files that have it, I could provide that as well/instead. I am +0 on the idea myself, as I don't think the plan is quite enough to be able to provide a user-experience upgrade for use cases besides "make me a dependency graph without downloading the distributions themselves". It certainly would be nice to be able to say to the user, "here are the things I will need to download in order to fulfill your request," but if you have to download individual files to get at that information, I'm not sure how much it helps vs. just downloading the files. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
On Sat, Sep 18, 2010 at 10:24 PM, Michael Foord wrote: > On 18/09/2010 12:25, "Martin v. Löwis" wrote: I think you are misunderstanding. The infrastructure will *not* depend on the old formats. Instead, packaging that have that information will provide it, packages that don't will not. The infrastructure is entirely agnostic on whether the data is available or not. In particular, it will not try to interpret the data in any way. >>> No, not at all. If tools *use* the information (and if they don't then >>> what is the point?) then packages that want to be compatible with those >>> tools will have to provide this information. >> >> Not true. Tools could/should also support PEP 345 data, and then they can >> support either kind of package. >> > But you are proposing that PyPI will provide an interface / API for exposing > the setuptools information. So tools that get metadata from PyPI in this way > (and depend on it - as they will if you expose it) will have functionality > that only works for packages that provide this information in this form. > > Saying tools "should" work with other formats too is empty words. If the idea is solely to use legacy setuptools data as a fallback if the actual PEP 345 data is unavailable, it sounds like it would be far more robust to have *PyPI* do the fallback, implementing it correctly *once*, rather than simply exposing the raw setuptools data (which *will* lead to client applications that *only* understand the setuptools data and can't understand packages that only provide PEP 345 compliant metadata). Throwing the raw data at client applications and saying "here, you figure it out" when they can already do that by downloading and interrogating the packages directly seems likely to cause more confusion than anything else. So +1 to a "allow_fallback_metadata" flag in appropriate PyPI APIs, -1 on exposing the legacy data directly. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Polymorphic best practices
On 9/18/2010 2:29 AM, [email protected] wrote: Polymorphic best practices [was: (Not) delaying the 3.2 release] If you're hung up on this, try writing the user-level documentation first. Your target audience is a working-level Web programmer, not someone who knows six programming languages and has a CS degree. If the explanation is too complex, so is the design. Coding in this area is quite hard to do right. There are issues with character set, HTML encoding, URL encoding, and internationalized domain names. It's often done wrong; I recently found a Google service which botched it. Python libraries should strive to deliver textual data to the programmer in clean Unicode. If someone needs the underlying wire representation it should be available, but not the default. John Nagle ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
On 18/09/2010 18:27, Nick Coghlan wrote: On Sat, Sep 18, 2010 at 10:24 PM, Michael Foord wrote: On 18/09/2010 12:25, "Martin v. Löwis" wrote: I think you are misunderstanding. The infrastructure will *not* depend on the old formats. Instead, packaging that have that information will provide it, packages that don't will not. The infrastructure is entirely agnostic on whether the data is available or not. In particular, it will not try to interpret the data in any way. No, not at all. If tools *use* the information (and if they don't then what is the point?) then packages that want to be compatible with those tools will have to provide this information. Not true. Tools could/should also support PEP 345 data, and then they can support either kind of package. But you are proposing that PyPI will provide an interface / API for exposing the setuptools information. So tools that get metadata from PyPI in this way (and depend on it - as they will if you expose it) will have functionality that only works for packages that provide this information in this form. Saying tools "should" work with other formats too is empty words. If the idea is solely to use legacy setuptools data as a fallback if the actual PEP 345 data is unavailable, it sounds like it would be far more robust to have *PyPI* do the fallback, implementing it correctly *once*, rather than simply exposing the raw setuptools data (which *will* lead to client applications that *only* understand the setuptools data and can't understand packages that only provide PEP 345 compliant metadata). Throwing the raw data at client applications and saying "here, you figure it out" when they can already do that by downloading and interrogating the packages directly seems likely to cause more confusion than anything else. So +1 to a "allow_fallback_metadata" flag in appropriate PyPI APIs, -1 on exposing the legacy data directly. If the two different data formats can be exposed in a compatible way then this sounds good to me. Michael Cheers, Nick. -- http://www.ironpythoninaction.com/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
2010/9/18 "Martin v. Löwis" : > Any other opinions? -1 from me as well; I see no reason to encourage use of bad metadata, given mechanisms to get correct metadata exist (running "setup.py egg_info", as others have pointed out). I understand there are perceived uses for such data, but it's just as available for those uses as for tools like pip and buildout. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
"Martin v. Löwis" wrote: >> So you are fine with publishing "slightly incorrect" metadata at PyPI >> ? I am not. > > I really have no intuition for in how many cases the data will be > incorrect. However, if users find that the data is incorrect for > specific package, they ought to complain to the maintainer. > >> I don't understand the rational behind providing flawed data at PyPI. >> I am -1 on that. > > -1 sounds much better than vetoing it :-) Taking the votes together, > I currently arrive at > > Tarek -1 > Michael -1 (?) > Jannis -1 (?) > Jim +1 > Thomas +1 > > Any other opinions? -1 as well. The formats and file-names are just a complete mess and not all packages on PyPI are available as eggs or compatible to setuptools. I think the information from PEP 345 which is already made available via the XML-RPC .release_data() interface is sufficient for most cases. In those cases where it is not, we should extend the meta data. http://www.python.org/dev/peps/pep-0345/ http://wiki.python.org/moin/PyPiXmlRpc It may be worthwhile providing the same sort of information as static PKG-INFO file alongside the /simple index entries as well (e.g. via a link on the package page), to make it possible to extract the meta information without having to use the RPC mechanism and making use of the mirror infrastructure. http://www.python.org/dev/peps/pep-0314/ http://www.python.org/dev/peps/pep-0390/ PS: Could you put your DZUG talk up online somewhere. It includes some very valuable information on the various available APIs that is difficult to find elswhere. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 18 2010) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] egg_info in PyPI
Am 18.09.2010 15:27, schrieb Steve Holden: > On 9/18/2010 9:21 AM, "Martin v. Löwis" wrote: >> IT WILL BE NOT IN PREFERENCE TO DISTUTILS2. > > No need to shout. I really felt that this otherwise wouldn't be heard - I tried to say it a number of times before, and it was ignored. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] os.path.normcase rationale?
Hi All,
I'm curious as to why, with a file called "Foo.txt" on a case
descriminating but case insensitive filesystem,
os.path.normcase('FoO.txt') will return "foo.txt" rather than "Foo.txt"?
Yes, I know the behaviour is documented, but I'm wondering if anyone can
remember the rationale for that behaviour?
cheers,
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] os.path.normcase rationale?
On Sat, Sep 18, 2010 at 2:39 PM, Chris Withers wrote:
> I'm curious as to why, with a file called "Foo.txt" on a case descriminating
> but case insensitive filesystem, os.path.normcase('FoO.txt') will return
> "foo.txt" rather than "Foo.txt"?
>
> Yes, I know the behaviour is documented, but I'm wondering if anyone can
> remember the rationale for that behaviour?
Because normcase() and friends never look at the filesystem. Of
course, exists() and isdir() etc. do, and so does realpath(), but the
pure parsing functions don't. They can be used without a working
filesystem even. (E.g. you can import ntpath on a Unix box and happily
parse Windows paths.)
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Some news from my sandbox project
Robert Collins wrote: __builtins__ is in everyone's global namespace, so if it can be mutated, different python programs running in the same sandbox can affect each other. So give each program its own copy of __builtins__. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Some news from my sandbox project
Victor Stinner wrote: Eg. one of the most important function of pysandbox is proxy() (a function to create a read only view of a object outside the sandbox, especially on an import), if you replace isinstance() by a function which always return True: you can get unmodified objects I don't follow. Trusted functions such as proxy() shouldn't be sharing a __builtins__ dict with sandboxed code. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Catalog-sig] egg_info in PyPI
On 2010-09-18, at 2:29 AM, Thomas Lotze wrote: > I'd like to expand [tl.eggdeps] > to analyse dependencies between any packages on PyPI but I can't > as long as dependency information is not available without actually > installing things. [...] On 2010-09-18, at 2:29 AM, Thomas Lotze wrote: >> I am not even understanding what's the benefit of doing this since an >> egg_info directory is obtained at *build* time and can differ from a >> machine to another, so it seems pretty useless for me to publish this. > > We do understand that legacy metadata has its issues, but taken with the > appropriate amount of salt, it's better than nothing in most cases, enough > so at least to solve some chicken-and-egg problems. Just curious: aside from the static metadata issue which you recognize, how would you make tl.eggdeps analyze dependencies between any packages registered on PyPI when not all of them upload their sdists (hence no egg_info) to PyPI? Also, if I may ask, what is the intended use for tl.eggdeps? For use in package managers (buildout, pip)? If yes, then ... would providing egg_info in PyPI obviate the need for package managers such as pip to download sources before resolving dependencies (keeping in mind that even packages that do upload their sdists to PyPI sometimes end up not doing the same - perhaps out of forgetfulness - for theirs latest releases)? -srid ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices)
On Sun, Sep 19, 2010 at 4:18 AM, John Nagle wrote: > On 9/18/2010 2:29 AM, [email protected] wrote: >> >> Polymorphic best practices [was: (Not) delaying the 3.2 release] > > If you're hung up on this, try writing the user-level documentation > first. Your target audience is a working-level Web programmer, not > someone who knows six programming languages and has a CS degree. > If the explanation is too complex, so is the design. > > Coding in this area is quite hard to do right. There are > issues with character set, HTML encoding, URL encoding, and > internationalized domain names. It's often done wrong; > I recently found a Google service which botched it. > Python libraries should strive to deliver textual data to the programmer > in clean Unicode. If someone needs the underlying wire representation > it should be available, but not the default. Even though URL byte sequences are defined as using only an ASCII subset, I'm currently inclined to add raw bytes supports to urlib.parse by providing parallel APIs (i.e. urlib.parse.urlsplitb, etc) rather than doing it implicitly in the normal functions. My rationale is as follows: - while URLs are *meant* to be encoded correctly as an ASCII subset, the real world isn't always quite so tidy (i.e. applications treat as URLs things that technically are not because the encoding is wrong) - separating the APIs forces the programmer to declare that they know they're working with the raw bytes off the wire to avoid the decode/encode overhead that comes with working in the Unicode domain - easier to change our minds later. Adding implicit bytes support to the normal names can be done any time, but removing it would require an extensive deprecation period Essentially, while I can see strong use cases for wanting to manipulate URLs in wire format, I *don't* see strong use cases for manipulating URLs without *knowing* whether they're in wire format (encoded bytes) or display format (Unicode text). For some APIs that work for arbitrary encodings (e.g. os.listdir) switching based on argument type seems like a reasonable idea. For those that may silently produce incorrect output for ASCII-incompatible encodings, the os.environ/os.environb seems like a better approach. I could probably be persuaded to merge the APIs, but the email6 precedent suggests to me that separating the APIs better reflects the mental model we're trying to encourage in programmers manipulating text (i.e. the difference between the raw octet sequence and the text character sequence/parsed data). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices)
On 9/18/2010 10:03 PM, Nick Coghlan wrote: > On Sun, Sep 19, 2010 at 4:18 AM, John Nagle wrote: >> On 9/18/2010 2:29 AM, [email protected] wrote: >>> >>> Polymorphic best practices [was: (Not) delaying the 3.2 release] >> >> If you're hung up on this, try writing the user-level documentation >> first. Your target audience is a working-level Web programmer, not >> someone who knows six programming languages and has a CS degree. >> If the explanation is too complex, so is the design. >> >> Coding in this area is quite hard to do right. There are >> issues with character set, HTML encoding, URL encoding, and >> internationalized domain names. It's often done wrong; >> I recently found a Google service which botched it. >> Python libraries should strive to deliver textual data to the programmer >> in clean Unicode. If someone needs the underlying wire representation >> it should be available, but not the default. > > Even though URL byte sequences are defined as using only an ASCII > subset, I'm currently inclined to add raw bytes supports to > urlib.parse by providing parallel APIs (i.e. urlib.parse.urlsplitb, > etc) rather than doing it implicitly in the normal functions. > > My rationale is as follows: > - while URLs are *meant* to be encoded correctly as an ASCII subset, > the real world isn't always quite so tidy (i.e. applications treat as > URLs things that technically are not because the encoding is wrong) > - separating the APIs forces the programmer to declare that they know > they're working with the raw bytes off the wire to avoid the > decode/encode overhead that comes with working in the Unicode domain > - easier to change our minds later. Adding implicit bytes support to > the normal names can be done any time, but removing it would require > an extensive deprecation period > > Essentially, while I can see strong use cases for wanting to > manipulate URLs in wire format, I *don't* see strong use cases for > manipulating URLs without *knowing* whether they're in wire format > (encoded bytes) or display format (Unicode text). For some APIs that > work for arbitrary encodings (e.g. os.listdir) switching based on > argument type seems like a reasonable idea. For those that may > silently produce incorrect output for ASCII-incompatible encodings, > the os.environ/os.environb seems like a better approach. > > I could probably be persuaded to merge the APIs, but the email6 > precedent suggests to me that separating the APIs better reflects the > mental model we're trying to encourage in programmers manipulating > text (i.e. the difference between the raw octet sequence and the text > character sequence/parsed data). > That sounds pretty sane and coherent to me. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
