Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 20 February 2013 04:07, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 02/19/2013 09:37 PM, Paul Moore wrote: >> On 20 February 2013 00:54, Fred Drake wrote: >>> I'd posit that anything successful will no longer need to be added >>> to the standard library, to boot. Packaging hasn't done well >>> there. >> >> distlib may be the exception, though. Packaging tools are somewhat >> unique because of the chicken and egg issue involved in having a >> packaging tool with external dependencies - who installs your >> dependencies for you? So enabling technology (library code to perform >> packaging-related tasks, particularly in support of standardised >> formats) could be better available from the stdlib. > > The big blocker there is that users can't rely on having it in the stdlib > until after they drop Python < 3.4 (assuming accelearated absorption) or > even 3.5. Understood - that's why I suggested that distlib reach a point where it's stable as an external package and supported on (some) older versions. I'm hoping for an experience more like unittest2 than packaging/distutils2. But we shouldn't be blocking new stdlib features just because they won't be available in older Python versions... Paul ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
> On Wednesday, February 20, 2013 at 2:48 AM, Chris Jerdonek wrote: > > I meant that bringing distlib into http://hg.python.org/cpython/ would > give it more visibility to core devs and others that already keep an > eye on python-checkins (the mailing list). And I think seeing the > Sphinx-processed docs integrated and cross-referenced with > http://docs.python.org/dev/ will help people understand better what > has been done and how it fits in with the rest of CPython -- which I > think would be useful to the community. It may also encourage > involvement (e.g. by being part of the main tracker). On Tue, Feb 19, 2013 at 11:53 PM, Donald Stufft wrote: > On the other hand it makes contributing to it more annoying since it > does not have pull requests, unless it was just a mirror. Maybe just the finished/production-ready pieces could be added as they are ready, with the main development happening outside. My understanding of distlib is that it's a collection of independent, bite-sized pieces of functionality, which could lend itself well to such a process. --Chris ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Built with VS2012 Express for desktop
> Date: Tue, 19 Feb 2013 12:48:02 -0600 > Subject: Re: [Python-Dev] Built with VS2012 Express for desktop > From: [email protected] > To: [email protected] > CC: [email protected] > > On Tue, Feb 19, 2013 at 12:37 PM, rahul garg wrote: > > Hi. > > > > I downloaded Python 3.3 source, opened up the solution in VS2012 Express for > > Desktop and built the "python" subproject using "Release" and "x64" > > configurations. I now have a "python.exe" in the PCBuild/amd64 subfolder > > that appears to be working as far as i can see. > > > > I have a few questions: > > a) Is there a test suite that I can run to verify that the build is working > > fine? > Last I checked there are some issues, but most of the tests should > pass. You would run "PCBuild\python.exe -m test" from the top level of > your checkout to see this. It's also covered at > http://docs.python.org/devguide/ Thanks! That page is what I should have looked for! > > b) I now intend to build some extensions (such as NumPy). Not sure if this > > is the right list, but would I need to modify something in distutils to get > > it working with VS2012? > > Yes. You'll probably need to point distutils to the correct batch file > that sets up a VS2012 build environment. Thanks again! rahul ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Built with VS2012 Express for desktop
On Wed, Feb 20, 2013 at 08:23:19AM +0100, Antoine Pitrou wrote: > On Tue, 19 Feb 2013 20:37:36 -0800 > Eli Bendersky wrote: > > On Tue, Feb 19, 2013 at 10:50 AM, Oleg Broytman wrote: > > Oleg, lately I have the feeling you're getting too automatic with this > > template response. > > +1. This isn't the python-robots mailing-list. Ok, I'll stop. Oleg. -- Oleg Broytmanhttp://phdru.name/[email protected] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 20.02.2013 03:37, Paul Moore wrote: > On 20 February 2013 00:54, Fred Drake wrote: >> I'd posit that anything successful will no longer need to be added to >> the standard library, to boot. Packaging hasn't done well there. > > distlib may be the exception, though. Packaging tools are somewhat > unique because of the chicken and egg issue involved in having a > packaging tool with external dependencies - who installs your > dependencies for you? So enabling technology (library code to perform > packaging-related tasks, particularly in support of standardised > formats) could be better available from the stdlib. > >> I'd rather see a successful packaging story develop than bundle it into the >> standard library. The later just isn't that interesting any more. > > Bundling too early is a bad idea though. distlib is developing fast > and to do so it needs (1) a development cycle independent of python's > and (2) compatibility and ease of use with earlier versions of Python > (the latter is also critical for adoption in place of custom code in > packaging tools). > > Aiming for an accelerated level of development targeting inclusion in > Python 3.4 is plausible, though. MAL pointed out that agreeing > standards but not offering tools to support them in the stdlib is > risky, as people have no incentive to adopt those standards. We've got > 6 months or more until 3.4 feature freeze, let's not make any decision > too soon, though. I'm fine with not adding distlib to Python 3.4. The point I wanted to make was that there has to be an reference implementation of the PEP that tool makers can use to avoid reinventing the wheel over and over again (each with its own set of problems). If distlib implements the PEP, then it just needs to be mentioned there as a suitable reference implementation. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 20 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Built with VS2012 Express for desktop
Le Wed, 20 Feb 2013 13:09:13 +0400, Oleg Broytman a écrit : > On Wed, Feb 20, 2013 at 08:23:19AM +0100, Antoine Pitrou > wrote: > > On Tue, 19 Feb 2013 20:37:36 -0800 > > Eli Bendersky wrote: > > > On Tue, Feb 19, 2013 at 10:50 AM, Oleg Broytman > > > wrote: Oleg, lately I have the feeling you're getting too > > > automatic with this template response. > > > > +1. This isn't the python-robots mailing-list. > >Ok, I'll stop. Thanks! Perhaps we may have to become more automatic if some day we are flooded with newbie messages on the list, but this isn't the case currently. Moreover, the original message here was a build question that probably wouldn't receive any meaningful answer on python-list, which makes it suitable for python-dev. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 426 is now the draft spec for distribution metadata 2.0
Le Tue, 19 Feb 2013 19:54:21 -0500, Fred Drake a écrit : > On Tue, Feb 19, 2013 at 6:19 PM, Donald Stufft > wrote: > > Let's not add anything to the stdlib till it has real world usage. > > Doing otherwise is putting the cart before the horse. > > I'd posit that anything successful will no longer need to be added to > the standard > library, to boot. Packaging hasn't done well there. If only "successful", agreed. However, packaging is a core part of the Python experience, and as such should be provided in the stdlib. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Built with VS2012 Express for desktop
On Wed, Feb 20, 2013 at 10:54:06AM +0100, Antoine Pitrou wrote: > Le Wed, 20 Feb 2013 13:09:13 +0400, > Oleg Broytman a ??crit : > > On Wed, Feb 20, 2013 at 08:23:19AM +0100, Antoine Pitrou > > wrote: > > > On Tue, 19 Feb 2013 20:37:36 -0800 > > > Eli Bendersky wrote: > > > > On Tue, Feb 19, 2013 at 10:50 AM, Oleg Broytman > > > > wrote: Oleg, lately I have the feeling you're getting too > > > > automatic with this template response. > > > > > > +1. This isn't the python-robots mailing-list. > > > >Ok, I'll stop. > > Thanks! Perhaps we may have to become more automatic if some day we are > flooded with newbie messages on the list, but this isn't the case > currently. That was the case when I started. > Moreover, the original message here was a build question that probably > wouldn't receive any meaningful answer on python-list, which makes it > suitable for python-dev. You are right. I am sorry. Oleg. -- Oleg Broytmanhttp://phdru.name/[email protected] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 20.02.2013 00:16, Daniel Holth wrote:
> On Tue, Feb 19, 2013 at 5:10 PM, M.-A. Lemburg wrote:
>
>> On 19.02.2013 23:01, Daniel Holth wrote:
>>> On Tue, Feb 19, 2013 at 4:34 PM, M.-A. Lemburg wrote:
>>>
On 19.02.2013 14:40, Nick Coghlan wrote:
> On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg
>> wrote:
>> * PEP 426 doesn't include any mention of the egg distribution format,
>> even though it's the most popular distribution format at the moment.
>> It should at least include the location of the metadata file
>> in eggs (EGG-INFO/PKG-INFO) and egg installations
>> (/EGG-INFO/PKG-INFO).
>
> "Other tools involved in Python distribution may also use this format."
>
> The egg format has never been, and never will be, officially endorsed
> by python-dev. The wheel format is the standard format for binary
> distribution, and PEP 376 defines the standard location for metadata
> on installed distributions.
Oh, come on, Nick, that's just silly. setuptools was included in stdlib
for a short while, so the above is simply wrong. Eggs are the most
widely used binary distribution format for Python package on PyPI:
# wc *files.csv
25585 25598 1431013 2013-02-19-egg-files.csv
46194640 236694 2013-02-19-exe-files.csv
254 255 13402 2013-02-19-msi-files.csv
104691 104853 5251962 2013-02-19-tar-gz-files.csv
24 241221 2013-02-19-whl-files.csv
17937 18022 905913 2013-02-19-zip-files.csv
153110 153392 7840205 total
(based on todays PyPI stats)
It doesn't really help ignoring realities... and I'm saying
that as one of the core devs who got setuptools kicked out of
the stdlib again.
--
Marc-Andre Lemburg
eGenix.com
>>>
>>> The wheel philosophy is that it should be supported by both python-dev
>> and
>>> setuptools and that you should feel happy about using setuptools if you
>>> like it whether or not python-dev (currently) endorses that. If you are
>>> using setuptools (distribute's pkg_resources) then you can use both at
>> the
>>> same time.
>>>
>>> Distribute, distutils and setuptools' problems have not been well
>>> understood which I think is why there has been a need to discredit
>>> setuptools by calling it non-standard. It is the defacto standard. If
>> your
>>> packages have dependencies there is no other choice. Wheel tries to solve
>>> the real problem by allowing you to build a package with setuptools while
>>> giving the end-user the choice of installing setuptools or not.
>>>
>>> Of course eggs are the most popular right now. The wheel format is very
>>> egg-like while avoiding some of egg's problems. See the comparison in the
>>> PEP or read the story on wheel's rtfd. The wheel project includes tools
>> to
>>> losslessly convert eggs or bdist_wininst to wheel.
>>
>> That's all fine, but it doesn't explain the refusal to add the
>> documentation of the location of the PKG-INFO file in eggs ?
>
>
> It would just be a sentence, I wouldn't have a problem with it but I also
> don't see why it would be necessary. Even setuptools doesn't touch the file
> usually. Right now distribute's pkg_resources currently only understands
> Requires-Dist if it is inside a .dist-info directory.
Perhaps I'm not clear enough. I'll try again :-)
The wording in the PEP alienates the egg format by defining
an incompatible new standard for the location of the metadata
file:
"""
There are three standard locations for these metadata files:
* the PKG-INFO file included in the base directory of Python source
distribution archives (as
created by the distutils sdist command)
* the {distribution}-{version}.dist-info/METADATA file in a wheel binary
distribution archive (as
described in PEP 425, or a later version of that specification)
* the {distribution}-{version}.dist-info/METADATA files in a local Python
installation database (as
described in PEP 376, or a later version of that specification)
"""
It's easy to upgrade distribute and distutils to write
metadata 1.2 format, simply by changing the version in the
PKG-INFO files.
These addition are necessary to fix the above and also include
the standard location of the metadata for pip and distutils installations:
* the EGG-INFO/PKG-INFO file in an egg binary distribution archive (as created
by the distribute
bdist_egg command)
* the {distribution}-{version}.egg/EGG-INFO/PKG-INFO file in an installed egg
distribution archive
* the {distribution}-{version}.egg-info/PKG-INFO file for packages installed
with "pip install" or
distribute's "python setup.py install"
* the {distribution}-{version}.egg-info file for packages installed with
distutils' "python setup.py
install"
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Feb 20 2013)
>>> Python Projects, Consulting and Support ... http://www.egenix.com/
>>> mx
[Python-Dev] The last Python 2.6 security release
I just updated PEP 361 (Python 2.6 release schedule). Python 2.6 is in security maintenance, source only release mode. Official support for Python 2.6 expires on October 1 2013, and I would like to do one last release[1], i.e. 2.6.9 as close to that date as possible. I know of issue 16248, but if there are any other security issues that must be applied to 2.6, please let me know. Cheers, -Barry [1] Guido's time machine works in both directions and prevents a 2.6.10 from ever happening. signature.asc Description: PGP signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
2013/2/19 Christian Heimes : > Hello, > > in August 2012 I found a DoS vulnerability in expat and XML libraries in > Python's standard library. Since then I have found several more issues. > I have been working on fixes ever since. > > The README of https://pypi.python.org/pypi/defusedxml contains detailed > explanations of my research and all issues > > > Blog post: > http://blog.python.org/2013/02/announcing-defusedxml-fixes-for-xml.html > > Hotfixes: > https://pypi.python.org/pypi/defusedxml > https://pypi.python.org/pypi/defusedexpat Are these going to become patches for Python, too? -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Am 20.02.2013 17:25, schrieb Benjamin Peterson: > Are these going to become patches for Python, too? I'm working on it. The patches need to be discussed as they break backward compatibility and AFAIK XML standards, too. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wed, Feb 20, 2013 at 8:24 PM, Christian Heimes wrote: > Am 20.02.2013 17:25, schrieb Benjamin Peterson: >> Are these going to become patches for Python, too? > > I'm working on it. The patches need to be discussed as they break > backward compatibility and AFAIK XML standards, too. That's not very good. XML parsers are supposed to parse XML according to standards. Is the goal to have them actually do that, or just address DDOS issues? Cheers, fijal ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (3.3): Rebuild importlib.h after the changes introduced in 0f65bf6063ca.
On Wed, Feb 20, 2013 at 1:16 PM, ezio.melotti wrote: > http://hg.python.org/cpython/rev/9d00c79b27e1 > changeset: 82280:9d00c79b27e1 > branch: 3.3 > parent: 82278:96b4acb253f8 > user:Ezio Melotti > date:Wed Feb 20 21:42:46 2013 +0200 > summary: > Rebuild importlib.h after the changes introduced in 0f65bf6063ca. Thanks, Ezio. -eric ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
> > I'm working on it. The patches need to be discussed as they break > > backward compatibility and AFAIK XML standards, too. > > That's not very good. XML parsers are supposed to parse XML according > to standards. Is the goal to have them actually do that, or just > address DDOS issues? Having read through Christian's mail and several of his references, it seems to me that addressing the DDoS issues is preferable to blindly following a standard that predates the Morris worm by a couple years. Everyone played nice before that watershed event. Heck, back then you could telnet to [email protected] without a password! Any incompatibility should have minimal impact. I haven't looked into the defusedxml package to see what limits it introduces to protect against attacks, but it seems that most well-behaved entities will use little, if any, recursion, and result in a size increase of less than a factor of 10 when fully expanded. Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On 02/20/2013 01:53 PM, Skip Montanaro wrote: >> That's not very good. XML parsers are supposed to parse XML according >> to standards. Is the goal to have them actually do that, or just >> address DDOS issues? > > Having read through Christian's mail and several of his references, it > seems to me that addressing the DDoS issues is preferable to blindly > following a standard that predates the Morris worm by a couple years. > Everyone played nice before that watershed event. Heck, back then you > could telnet to [email protected] without a password! Also, despite the title of this thread, the vulnerabilities include fetching of external DTDs and entities (per standard), which opens up attacks that are worse than just denial-of-service. In our initial Django release advisory we carelessly lumped the potential XML vulnerabilities together under the "DoS" label, and were quickly corrected. An XML parser that follows the XML standard is never safe to expose to untrusted input. This means the choice is just whether the stdlib XML parsers should be safe by default, or follow the standard by default. (Given either choice, the other option can still be made available via flags). Carl ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Am 20.02.2013 21:17, schrieb Maciej Fijalkowski: > On Wed, Feb 20, 2013 at 8:24 PM, Christian Heimes > wrote: >> Am 20.02.2013 17:25, schrieb Benjamin Peterson: >>> Are these going to become patches for Python, too? >> >> I'm working on it. The patches need to be discussed as they break >> backward compatibility and AFAIK XML standards, too. > > That's not very good. XML parsers are supposed to parse XML according > to standards. Is the goal to have them actually do that, or just > address DDOS issues? But the standard is flawed. It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A single 1 kB XML document can kill virtually any machine, even servers with more than hundred GB RAM. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Am 20.02.2013 22:02, schrieb Carl Meyer: > Also, despite the title of this thread, the vulnerabilities include > fetching of external DTDs and entities (per standard), which opens up > attacks that are worse than just denial-of-service. In our initial > Django release advisory we carelessly lumped the potential XML > vulnerabilities together under the "DoS" label, and were quickly corrected. Right, I tried to address both kinds of issues in the title: XML DoS vulnerabilities and (other XML) exploits Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Carl Meyer wrote: An XML parser that follows the XML standard is never safe to expose to untrusted input. Does the XML standard really mandate that a conforming parser must blindly download any DTD URL given to it from the real live internet? Somehow I doubt that. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Thu, 21 Feb 2013 11:35:23 +1300, Greg Ewing wrote: > Carl Meyer wrote: > > An XML parser that follows the XML standard is never safe to expose to > > untrusted input. > > Does the XML standard really mandate that a conforming parser > must blindly download any DTD URL given to it from the real > live internet? Somehow I doubt that. I don't believe it does. The DTD URL is, if I remember correctly, specified as an identifier. The fact that you can often also download the DTD from the location specified by the identifier is a secondary effect. But, it's been a *long* time since I looked at XML :) (Wikipedia says: "Programs for reading documents may not be required to read the external subset.", which would seem to confirm that.) --David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wed, Feb 20, 2013 at 5:45 PM, R. David Murray wrote: > (Wikipedia says: "Programs for reading documents may not be required to > read the external subset.", which would seem to confirm that.) Validating parsers are required to read the external subset; this doesn't apply to the parsers distributed for Python today. Even when loading external resources, I don't think there's anything in the XML specification that says how they have to be loaded, or how to deal with an error when they are (and refusing to load because of resource limits is reasonably just another error with respect to the parser). While I'd hate to make XML processing more painful than it often is, there's no injunction not to be reasonable. Security concerns and resource limits are cross-cutting concerns, so it's not wrong to provide safe defaults. Doing so *will* be backward incompatible, and I'm not sure there's a good way to gauge the extent of the breakage. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On 02/20/2013 03:35 PM, Greg Ewing wrote: > Carl Meyer wrote: >> An XML parser that follows the XML standard is never safe to expose to >> untrusted input. > > Does the XML standard really mandate that a conforming parser > must blindly download any DTD URL given to it from the real > live internet? Somehow I doubt that. For a validating parser, the spec does mandate that. It permits non-validating parsers (browsers are the only example given) to simply note the existence of an external entity reference and "retrieve it for display only on demand." [1] But this isn't particularly relevant; the quoted statement is true even if you ignore the external reference issues entirely and consider only entity-expansion DoS. Some level of non-conformance to the spec is necessary to make parsing of untrusted XML safe. Carl [1] http://www.w3.org/TR/xml/#include-if-valid ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Am 20.02.2013 23:45, schrieb R. David Murray: > I don't believe it does. The DTD URL is, if I remember correctly, > specified as an identifier. The fact that you can often also download the > DTD from the location specified by the identifier is a secondary effect. > > But, it's been a *long* time since I looked at XML :) A DTD may have an identifier and a resource locator (local file or URL). It depends which kind of DTD is used (internal, external public or external system), e.g. . For external DTDs a parser may chose to cache a DTD or map DTD identifier to its own set of DTDs. As far as I know a parser doesn't have to download a DTD unless it runs in validation mode. Just xml.sax and xml.dom.pulldom download DTDs, see https://pypi.python.org/pypi/defusedxml#python-xml-libraries DTD retrieval is not as severe as external entity expansion. With external entities like an attacker is actually able to download files and circumvent firewalls if the application returns parts of the XML file back. Most XML parsers expand entities and lots of them even expand external entities. Daniel Veillard (libxml2) has explained that entity expansion is required for XPath() and IIRC for features like XSL, too. Nowadays most XML parsers and libraries have options to disable certain features. Python's standard library doesn't have options for some features or ignores other settings silently. Everything is documented at https://pypi.python.org/pypi/defusedxml, too. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wed, 20 Feb 2013 22:55:57 +0100 Christian Heimes wrote: > Am 20.02.2013 21:17, schrieb Maciej Fijalkowski: > > On Wed, Feb 20, 2013 at 8:24 PM, Christian Heimes > > wrote: > >> Am 20.02.2013 17:25, schrieb Benjamin Peterson: > >>> Are these going to become patches for Python, too? > >> > >> I'm working on it. The patches need to be discussed as they break > >> backward compatibility and AFAIK XML standards, too. > > > > That's not very good. XML parsers are supposed to parse XML according > > to standards. Is the goal to have them actually do that, or just > > address DDOS issues? > > But the standard is flawed. It is not flawed as long as you are operating in a sandbox (read: controlled environment). > It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A > single 1 kB XML document can kill virtually any machine, even servers > with more than hundred GB RAM. Assuming an attacker can inject arbitrary XML. Not every XML document is loaded from the Internet. Not everyone is a security nuts. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: > > It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A > > single 1 kB XML document can kill virtually any machine, even servers > > with more than hundred GB RAM. > > > > > Assuming an attacker can inject arbitrary XML. Not every XML document > is loaded from the Internet. > > Even documents not loaded from the internet can be at risk. Often times security breaches are the result of a chain of actions. You can say "I'm not loading this XML from the internet, so therefore I am safe" but then you have another flaw (for example) where you unpack a zip file without verifying there are not absolute paths and suddenly your xml file has been replaces with a malicious one. > Not everyone is a security nuts. > > This is precisely why things should be safe by default and allow unsafe actions to be turned on optionally. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Am 20.02.2013 23:56, schrieb Fred Drake: > While I'd hate to make XML processing more painful than it often is, there's > no injunction not to be reasonable. Security concerns and resource limits > are cross-cutting concerns, so it's not wrong to provide safe defaults. > > Doing so *will* be backward incompatible, and I'm not sure there's a good > way to gauge the extent of the breakage. We could walk a different path but that would keep Python's XML libraries in an insecure mode by default. My latest patch to expat and pyexpat supports global default values. The global defaults are used when a new parser is created with pyexpat.ParserCreate(). It's also possible to disable the new limitations in expat by default. We can add a function to the XML package tree that enables all restrictions: * limit expansion depths of nested entities * limit total amount of expanded chars * disable external entity expansion * optionally force expat to ignore and reset all DTD information 3rd party users have to disable secure settings explicitly for the current interpreter (although expat limits are process wide and shared across subinterpreters). try: import xml.security except ImportError: # old Python pass else: xml.security.harden_xml_parser() I guess most programs either process untrusted XML input or large XML documents that require expansion and DTD validation. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Am 21.02.2013 00:08, schrieb Antoine Pitrou: > Not everyone is a security nuts. But, but, but ... it's fun to be paranoid! You get so many new potential enemies. :) Jerry Fletcher ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wed, 20 Feb 2013 18:21:22 -0500 Donald Stufft wrote: > On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: > > > It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A > > > single 1 kB XML document can kill virtually any machine, even servers > > > with more than hundred GB RAM. > > > > > > > Assuming an attacker can inject arbitrary XML. Not every XML document > > is loaded from the Internet. > > Even documents not loaded from the internet can be at risk. Often times > security breaches are the result of a chain of actions. You can say "I'm > not loading this XML from the internet, so therefore I am safe" but then > you have another flaw (for example) where you unpack a zip file > without verifying there are not absolute paths and suddenly your xml file has > been replaces with a malicious one. Assuming your ZIP file is coming from the untrusted Internet, indeed. Again, this is the same assumption that you are grabbing some important data from someone you can't trust. Just because you are living in a Web-centric world doesn't mean everyone does. There are a lot of use cases which are not impacted by your security rules. Bugfix releases shouldn't break those use cases, which means the security features should be mostly opt-in for 2.7 and 3.3. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Wed, Feb 20, 2013 at 5:30 AM, M.-A. Lemburg wrote: > The wording in the PEP alienates the egg format by defining > an incompatible new standard for the location of the metadata > file: This isn't a problem, because there's not really a use case at the moment for eggs to include a PEP 426-format metadata file, and if they ever do, it ought to be called METADATA, so that pkg_resources will know to read it if there are no egg-format dependencies listed. Setuptools also doesn't care what format PKG-INFO is in, as it only ever reads the "Version:" field, and that only in the case of in-development source packages. > It's easy to upgrade distribute and distutils to write > metadata 1.2 format, simply by changing the version in the > PKG-INFO files. As soon as distutils does it, setuptools will do it, because setuptools delegates metadata writing to distutils. So there's no "alienation" here. What will need to happen at some point is for pkg_resources to implement support for PEP 426-style version requirements, which I haven't tried to fully wrap my head around as yet. I'm hoping that there are simple textual substitutions (e.g. regexes) that can be done to convert them to pkg_resources requirements. If need be, I'll swipe whatever's needed from distlib. ;-) In the meantime, eggs aren't actually going anywhere, since wheels aren't actually trying to replace all of their use cases. And since the new metadata and binary formats don't actually add much new functionality over what eggs already do, eggs wouldn't lose much by not being able to use the same metadata, anyway. ;-) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wednesday, February 20, 2013 at 6:23 PM, Christian Heimes wrote: > We can add a function to the XML package tree that enables all restrictions: > > * limit expansion depths of nested entities > * limit total amount of expanded chars > * disable external entity expansion > * optionally force expat to ignore and reset all DTD information > > 3rd party users have to disable secure settings explicitly for the > current interpreter (although expat limits are process wide and shared > across subinterpreters). > > try: > import xml.security > except ImportError: > # old Python > pass > else: > xml.security.harden_xml_parser() We've learned nothing from Ruby and their YAML problems. Things need to be safe by default and the unsafe things explicitly enabled. Even *smart* developers do things wrong (e.g. YAML.load instead of YAML.safe_load) and protecting developers by default should be the path forward. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 6:42 AM, Nick Coghlan wrote: > Nothing in the PEP is particularly original - almost all of it is > either stolen from other build and packaging systems, or is designed > to provide a *discoverable* alternative to existing > setuptools/distribute/pip practices (specifically, the new extension > mechanism means things like entry points can be defined in the > standard metadata files without python-dev needing to get involved). FWIW, I actually think this is a step in the wrong direction relative to eggs; the ability to add new metadata files is a useful feature for application frameworks. For example, the EggTranslations project uses egg metadata to implement resource localization plugins. It lets you have an application with plugins that either contain their own translations, contain multiple translations for another plugin, a single language translation for an assortment of plugins, etc. These kinds of runtime-discovery use cases haven't seen much attention in the metadata standard discussion. On one level, that's fine, because it makes sense that distribution-provided metadata should be parseable by all tools, and that at build/download/install time the performance and ease-of-use favor a single file approach. That does not mean, however, that the presence of other files is bad or should be deprecated. IMO, metadata that see significant runtime use independent of the core metadata *should* appear in their own files, even if redundant. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Feb 20, 2013, at 6:22 PM, Antoine Pitrou wrote: > On Wed, 20 Feb 2013 18:21:22 -0500 > Donald Stufft wrote: >> On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A single 1 kB XML document can kill virtually any machine, even servers with more than hundred GB RAM. >>> >>> Assuming an attacker can inject arbitrary XML. Not every XML document >>> is loaded from the Internet. >> >> Even documents not loaded from the internet can be at risk. Often times >> security breaches are the result of a chain of actions. You can say "I'm >> not loading this XML from the internet, so therefore I am safe" but then >> you have another flaw (for example) where you unpack a zip file >> without verifying there are not absolute paths and suddenly your xml file has >> been replaces with a malicious one. > > Assuming your ZIP file is coming from the untrusted Internet, indeed. > Again, this is the same assumption that you are grabbing some important > data from someone you can't trust. > > Just because you are living in a Web-centric world doesn't mean > everyone does. There are a lot of use cases which are not impacted by > your security rules. Bugfix releases shouldn't break those use cases, > which means the security features should be mostly opt-in for 2.7 and > 3.3. > > Regards > > Antoine. Any type of input is a potential attack vector; this isn't web centric, it's a systemic flaw in the spec that allows any application that's loading XML to be bombed into oblivion. People need to trust that the standard library is reliable and sane-by-default. What we have right now isn't > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/20/2013 06:22 PM, Antoine Pitrou wrote: > On Wed, 20 Feb 2013 18:21:22 -0500 Donald Stufft > wrote: >> On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A single 1 kB XML document can kill virtually any machine, even servers with more than hundred GB RAM. >>> >>> Assuming an attacker can inject arbitrary XML. Not every XML >>> document is loaded from the Internet. >> >> Even documents not loaded from the internet can be at risk. Often >> times security breaches are the result of a chain of actions. You >> can say "I'm not loading this XML from the internet, so therefore I >> am safe" but then you have another flaw (for example) where you >> unpack a zip file without verifying there are not absolute paths and >> suddenly your xml file has been replaces with a malicious one. > > Assuming your ZIP file is coming from the untrusted Internet, indeed. > Again, this is the same assumption that you are grabbing some > important data from someone you can't trust. > > Just because you are living in a Web-centric world doesn't mean > everyone does. There are a lot of use cases which are not impacted by > your security rules. Bugfix releases shouldn't break those use cases, > which means the security features should be mostly opt-in for 2.7 and > 3.3. Two words: "hash randomization". If it applies to one, it applies to the other. Tres. - -- === Tres Seaver +1 540-429-0999 [email protected] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlElYScACgkQ+gerLs4ltQ4QgwCfctL8/FmnboJWozyPcSE1xbb2 wwIAoNVc2hoQci9G2M6g/keNNsN5RR0O =Q9IX -END PGP SIGNATURE- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wednesday, February 20, 2013 at 6:22 PM, Antoine Pitrou wrote: > On Wed, 20 Feb 2013 18:21:22 -0500 > Donald Stufft mailto:[email protected])> > wrote: > > On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: > > > > It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A > > > > single 1 kB XML document can kill virtually any machine, even servers > > > > with more than hundred GB RAM. > > > > > > > > > > > > > Assuming an attacker can inject arbitrary XML. Not every XML document > > > is loaded from the Internet. > > > > > > > > > Even documents not loaded from the internet can be at risk. Often times > > security breaches are the result of a chain of actions. You can say "I'm > > not loading this XML from the internet, so therefore I am safe" but then > > you have another flaw (for example) where you unpack a zip file > > without verifying there are not absolute paths and suddenly your xml file > > has > > been replaces with a malicious one. > > > > > Assuming your ZIP file is coming from the untrusted Internet, indeed. > Again, this is the same assumption that you are grabbing some important > data from someone you can't trust. > > No software you run on your computer grabs data from someone you don't trust and it all validates that even though you trust them they haven't been exploited? Like I said these sort of things are often caused by chaining several unrelated things together. > > Just because you are living in a Web-centric world doesn't mean > everyone does. There are a lot of use cases which are not impacted by > your security rules. Bugfix releases shouldn't break those use cases, > which means the security features should be mostly opt-in for 2.7 and > 3.3. > > Regards > > Antoine. > ___ > Python-Dev mailing list > [email protected] (mailto:[email protected]) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Postponing acceptance of PEP 426
The feedback I have received (both on-list and in response to some off-list queries to specific people) tells me that PEP 426 isn't quite ready for acceptance yet. Things I'll be working on or facilitating over the next few weeks: - documenting an overall transition plan to put the new metadata format in context - clear acceptance criteria in PEP 426, primarily a list of specific projects and individuals that I have requested endorsement from before I am willing to accept the PEP - adding entry-points to PEP 426 - some more adjustments to PEP 426 for improved pkg_resources compatibility - some adjustments to PEP 426 for easier translation to JSON-compatible structured data - PEP for a standardised approach to metadata caching in installation databases - draft PEP for adding distlib to Python 3.4 - draft PEP for automated conversion of PEP 426 metadata files to JSON-compatible structured data This postpone does mean the initial wheel rollout in various projects will be based on metadata 1.2 + the Provides-Extra field (and associated environment marker change), rather than being cleanly based on metadata 2.0. I can live with that - that part of the PEP is *not* open to change, but there are other legitimate concerns that need to be addressed before I declare the PEP accepted. Regards, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On 21/02/13 10:22, Antoine Pitrou wrote: On Wed, 20 Feb 2013 18:21:22 -0500 Donald Stufft wrote: On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A single 1 kB XML document can kill virtually any machine, even servers with more than hundred GB RAM. Assuming an attacker can inject arbitrary XML. Not every XML document is loaded from the Internet. Even documents not loaded from the internet can be at risk. Often times security breaches are the result of a chain of actions. You can say "I'm not loading this XML from the internet, so therefore I am safe" but then you have another flaw (for example) where you unpack a zip file without verifying there are not absolute paths and suddenly your xml file has been replaces with a malicious one. Assuming your ZIP file is coming from the untrusted Internet, indeed. Again, this is the same assumption that you are grabbing some important data from someone you can't trust. It's easy to forget that malware existed long before the Internet. The internet is just a transmission vector, it is not the source of malicious files. The source of malicious files is *other people*, and unless you never use XML files you didn't generate yourself, you cannot completely trust the source. You might trust your colleagues to not *intentionally* pass you a malicious XML file, but they may still do so accidentally. The risk seems small, these days, but remember that for decades the sole transmission vector for viruses and other malware was *people you trusted* not to deliberately give you a virus. Just because you are living in a Web-centric world doesn't mean everyone does. There are a lot of use cases which are not impacted by your security rules. Bugfix releases shouldn't break those use cases, which means the security features should be mostly opt-in for 2.7 and 3.3. I think that is reasonable. Insecure by default or not, code should not suddenly stop working because I've upgraded from 2.7.3 to 2.7.4. -- Steven ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Thu, Feb 21, 2013 at 9:49 AM, Tres Seaver wrote: > Two words: "hash randomization". If it applies to one, it applies to > the other. Agreed. Christian's suggested approach sounds sane to me: - make it possible to enable safer behaviour globally in at least 2.7 and 3.3 (and perhaps in 2.6 and 3.2 security releases as well) - make the safer behaviour the default in 3.4 - make it possible to selectively disable the safeguards in all versions A *possible* alternative in to step 1 is loud warnings in the docs directing people to defusedxml, but I prefer the idea of actually making the safeguards available directly in the standard library. Regards, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
M.-A. Lemburg egenix.com> writes: > The suggestion to have the metadata available on PyPI doesn't > have anything to do with security. > > It's about being able to determine compatibility and select the > right distribution file for download. The metadata also helps in > creating dependency graphs, which are useful for a lot of things. FYI, distlib already uses metadata in this way. Note that: 1. Dependency information is locked away in setup.py files, e.g in kwargs to the setup() call. 2. PyPI does not provide access to this metadata. So I have had to write code to extract what information I could out of setup.py setup() and into metadata that can be used. I have, for now, hosted it on my own server, but it can be used for e.g. determining complete dependency graphs without ever actually downloading any distributions. Distlib provides access to it: $ python finddeps.py pyramid 12 dists, elapsed time: 4.35 secs Download order: download setuptools (0.6c11) [for pyramid, zope.interface, zope.deprecation] download PasteDeploy (1.5.0) [for pyramid] download repoze.lru (0.6) [for pyramid] download venusian (1.0a7) [for pyramid] download MarkupSafe (0.15) [for Mako] download translationstring (1.1) [for pyramid] download Chameleon (2.11) [for pyramid] download WebOb (1.2.3) [for pyramid] download zope.interface (4.0.3) [for pyramid] download Mako (0.7.3) [for pyramid] download zope.deprecation (4.0.2) [for pyramid] download pyramid (1.4) In the above, nothing was actually downloaded except the metadata. It's just indicating an intuitive ordering for download based on a topological sort of the dependency graph - for pyramid, in this case. Regards, Vinay Sajip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
Chris Jerdonek gmail.com> writes: > Maybe this is already stated somewhere, but is there a plan for when > distlib will be brought into the repository? Is there a reason not to > do it now? It seems it would have more visibility that way (e.g. > people could see it as part of the development version of the online > docs and it would be in check-ins as are PEP edits), and its status > relative to Python would be clearer. There's no firm plan other than it being on the 3.4 wishlist, but IMO there's no rush to do it right now. AFAICT interested parties can be in the loop even though the project is actively developed on BitBucket rather than hg.python.org: remember that the current stakeholders in terms of packaging tools are not all in python-dev and may find it easier to contribute on BitBucket. (I often get comments from Daniel Holth in response to distlib commits I push to BitBucket.) I periodically sync the BitBucket repo with hg.python.org/distlib. The docs are also readily accessible on readthedocs.org (the first two Google results for distlib are those docs and the project on BitBucket). While not integrated with docs.python.org, I don't think that the distlib docs suffer inordinately from being on readthedocs.org - for one thing, I've enabled Disqus comments, allowing feedback on the docs - not that I've had any ;-) Closer integration should perhaps happen, but not until the project has matured a little. Regards, Vinay Sajip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
Paul Moore gmail.com> writes: > Understood - that's why I suggested that distlib reach a point where > it's stable as an external package and supported on (some) older > versions. I'm hoping for an experience more like unittest2 than > packaging/distutils2. Currently, distlib runs on Python 2.6, 2.7 and 3.2+, so there shouldn't be any concerns about support on those older Python versions. Regards, Vinay Sajip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wed, Feb 20, 2013 at 7:38 PM, Nick Coghlan wrote: > Christian's suggested approach sounds sane to me: Definitely. A strong +1 from me, FWIW these days. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Feb 21, 2013, at 10:38 AM, Nick Coghlan wrote: >- make it possible to enable safer behaviour globally in at least 2.7 >and 3.3 (and perhaps in 2.6 and 3.2 security releases as well) I want to be fairly conservative with 2.6.9. -Barry ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/20/2013 09:08 PM, Barry Warsaw wrote: > On Feb 21, 2013, at 10:38 AM, Nick Coghlan wrote: > >> - make it possible to enable safer behaviour globally in at least >> 2.7 and 3.3 (and perhaps in 2.6 and 3.2 security releases as well) > > I want to be fairly conservative with 2.6.9. I believe that the same rationale should apply as that for adding hash randomization in 2.6.8: this is at least as bad a vulnerability, with many more vectors of attack. Tres - -- === Tres Seaver +1 540-429-0999 [email protected] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlElo/cACgkQ+gerLs4ltQ4urQCg2Kyr6CKZPp35fAK1G4OtzYc+ XD8An0fJZw5DHRxg1JPe9AzcLqpvRZc5 =hmpM -END PGP SIGNATURE- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On Feb 20, 2013, at 11:35 PM, Tres Seaver wrote: >I believe that the same rationale should apply as that for adding hash >randomization in 2.6.8: this is at least as bad a vulnerability, with >many more vectors of attack. Except that I really want to EOL 2.6 in October as per schedule, and I really don't want a 2.6.10. So if we get the API changes wrong in 2.6.9 there won't be much of an opportunity to correct it. Also, in 2.6, hash randomization is opt-in so the default didn't change. Cheers, - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) iQIcBAEBCAAGBQJRJaheAAoJEBJutWOnSwa/A2sQAK0HnPQCG87vEVj7hpfp1h7o 8mVDpf8VyZtfVIhwJLmb106DCe3mWb+UZLf4ks16zSQfy1JVxNl179VESqDgOD30 RTC0/esArDzpVbCOKf0mepyYflQEnrA0FR/URAJVoqjGDlPSPr6vneWX2fOPGwn7 VsgzHzyqHs5k+JLTn9piDKLfW6LXWQOYl+oPF/T60SnYZTs8y6E00n9i2kLPaWy+ UMSnAC3jQMtgfhzPnXgPrlzVh4/ojYYnBVdhZYKGroBx4JGt7UDph1koFck7CRTD yJuVQ4QzsbtEF0/SyJVORWTh2N3jxEA7qWFIDQ8kif+Tqqfz/bujSud8SEK7dLuZ IPinvL2K9lD722RJjhmcMyfsrCJxafF2YbkdWyNr2zMlx6FO/oQWCbDVPBaZlu8+ cZyzF+JJiIM+ljlqL9dy5w156JxfQ5eh5h5ocZ/OiUisiKY93zT9j2hOM82bbo+/ QIWmMf2wPl277awbvw3+9GDi4xdgK0GJB/4BVOYBXy0/b2q2n5oHyHACJCoUtDyv AVCT+Hfw5Nu1ZIm3AU329gFeBPjEvl8+YsbiQf4hqsp40fnY7GzhRFb8HTwCZVmu amOWyeTg9f7dtF/sOwEpc/UGcCjJdBklrbndpNZ0f9gZF+FwncxbJThrTztlxaDY 7eT2EYRJsnLW5XiZTilj =HXz8 -END PGP SIGNATURE- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Maciej Fijalkowski, 20.02.2013 21:17: > On Wed, Feb 20, 2013 at 8:24 PM, Christian Heimes wrote: >> Am 20.02.2013 17:25, schrieb Benjamin Peterson: >>> Are these going to become patches for Python, too? >> >> I'm working on it. The patches need to be discussed as they break >> backward compatibility and AFAIK XML standards, too. > > That's not very good. XML parsers are supposed to parse XML according > to standards. I think we can shorten this discussion to "this is a serious problem that needs to be fixed". If that involves taking the freedom that the XML standard leaves about processing DTDs, then I think we shouldn't be throwing any high-level bike shedding at it. Consulting the standard actually helps. Stefan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Wed, 20 Feb 2013 18:45:10 -0500 Donald Stufft wrote: > > No software you run on your computer grabs data from someone you don't trust > and it all validates that even though you trust them they haven't been > exploited? What the hell do you mean exactly? There are other reasons to validate data than just "security". > Like I said these sort of things are often caused by chaining several > unrelated > things together. At this points this sounds like fear-mongering. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Thu, 21 Feb 2013 11:37:47 +1100 Steven D'Aprano wrote: > > It's easy to forget that malware existed long before the Internet. The > internet is just a transmission vector, it is not the source of malicious > files. The source of malicious files is *other people*, and unless you never > use XML files you didn't generate yourself, you cannot completely trust the > source. You might trust your colleagues to not *intentionally* pass you a > malicious XML file, but they may still do so accidentally. That's in theory very nice, but in practice security in everyday computing hasn't really been a concern before the massification of Internet access. (yes, there have been viruses on mainstream platforms such as the Amiga, but it was pretty minor compared to nowadays, and nobody cared about potential DoS attacks for example) So, as for XML files, we are talking about a DoS vulnerability. It will take more than a single file to make a DoS attack really annoying, which means the attacker must pollute the source of those XML files in a systemic way. It's not "a single XML file will smuggle confidential data out of the building". Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Thu, 21 Feb 2013 10:38:07 +1000 Nick Coghlan wrote: > On Thu, Feb 21, 2013 at 9:49 AM, Tres Seaver wrote: > > Two words: "hash randomization". If it applies to one, it applies to > > the other. > > Agreed. Christian's suggested approach sounds sane to me: > > - make it possible to enable safer behaviour globally in at least 2.7 > and 3.3 (and perhaps in 2.6 and 3.2 security releases as well) > - make the safer behaviour the default in 3.4 > - make it possible to selectively disable the safeguards in all versions +1 from me. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/21/2013 01:53 AM, Antoine Pitrou wrote: > On Thu, 21 Feb 2013 11:37:47 +1100 Steven D'Aprano > wrote: >> >> It's easy to forget that malware existed long before the Internet. >> The internet is just a transmission vector, it is not the source of >> malicious files. The source of malicious files is *other people*, >> and unless you never use XML files you didn't generate yourself, you >> cannot completely trust the source. You might trust your colleagues >> to not *intentionally* pass you a malicious XML file, but they may >> still do so accidentally. > > That's in theory very nice, but in practice security in everyday > computing hasn't really been a concern before the massification of > Internet access. > > (yes, there have been viruses on mainstream platforms such as the > Amiga, but it was pretty minor compared to nowadays, and nobody cared > about potential DoS attacks for example) > > So, as for XML files, we are talking about a DoS vulnerability. It > will take more than a single file to make a DoS attack really > annoying, which means the attacker must pollute the source of those > XML files in a systemic way. It's not "a single XML file will smuggle > confidential data out of the building". Antoine, A single, small,, malicious XML file can kill a machine (not just the process parsing it) by sucking all available RAM. We are talking hard lockup, reboot-to-fix-it sorts of DOC here. Tres. - -- === Tres Seaver +1 540-429-0999 [email protected] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlElzMQACgkQ+gerLs4ltQ7fDQCgmvvurNi5VtWA+4mqcz4tpUhR rNUAnRtpcKMFCM3z8qRKNfinAE9ly9fX =y+eM -END PGP SIGNATURE- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Thu, 21 Feb 2013 02:29:08 -0500 Tres Seaver wrote: > > Antoine, > > A single, small,, malicious XML file can kill a machine (not just the > process parsing it) by sucking all available RAM. We are talking hard > lockup, reboot-to-fix-it sorts of DOC here. Sure, but in many instances, rebooting a machine is not business-threatening. You will have a couple of minutes' downtime and that's all. Which is why the attack must be repeated many times to be a major annoyance. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
