Re: [Python-Dev] [PEP 437] A DSL for specifying signatures, annotations and argument converters
On 03/16/2013 02:17 AM, Stefan Krah wrote: Both PEPs were discussed at PyCon. The state of affairs is that a compromise is being worked upon and will be published by Larry in a revised PEP. I've pushed an update to PEP 436, the Argument Clinic PEP. It's now live on python.org. //arry/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] 2.7.4 is inevitable
I am going to cut the 2.7.4 release branch next weekend (March 23, 24). Things which are breaking the buildbots at the point will be backed out. Owners of current release blockers will be poked, but nothing is going to hold up the release. The show must go on. Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Status of XML fixes
Hello, I like to give an update on the XML vulnerability fixes. Brett has asked me a couple of days ago but I haven't had time to answer. I was/am busy with my daily job. Any attempt to fix the XML issues *will* change the behavior of the library and result into an incompatibility with older releases. Benjamin doesn't want to change the behavior of our XML libraries. IIRC Georg and Barry are +0. I think that we should keep the current and unsafe settings as default and add a simmple API to enable limitations and protections. What's available? - https://bitbucket.org/tiran/defusedexpat contains everything we need to fix the issues in the stdlib. All modifications to C code are available for all relevant Python versions. They have been tested on Linux and Windows, too. * modified expat library with checks and workarounds for entity expansion attacks. All fixes can be enabled or disabled by default at compile time. The default settings can also be configured globally (process wide, may be an issue for subinterpreters) and overwritten on the expat parser instance. * patched copies of pyexpat and _elementtree C extensions from Python 2.6, 2.7, 3.1, 3.2, 3.3 and 3.4 (a separate copy of each version). The patches provide the functions and attributes to modifiy the global and instance settings. * defusedexpat.py contains the patches for sax and dom parsers to disable external entity parsing. * http://bugs.python.org/issue17239 contains an old patch for the issues with a bunch of tests for each issue. What needs to be done? -- * agree on default settings: secure by default or backwards compatible by default? * review of the changes to expat, pyexpat and _elementtree. Antoine, Brett and Fred Drake have done some reviews. * design and implement an API to enable the protective restrictions. * documentation * perhaps more tests * finish the CVE reports In the mean time ... https://pypi.python.org/pypi/defusedxml provides documentation, examples and fixes for all Python versions w/o any C extension. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of XML fixes
I like to give an update on the XML vulnerability fixes. Brett has asked > me a couple of days ago but I haven't had time to answer. I was/am busy > with my daily job. > > Any attempt to fix the XML issues *will* change the behavior of the > library and result into an incompatibility with older releases. Benjamin > doesn't want to change the behavior of our XML libraries. IIRC Georg and > Barry are +0. I think that we should keep the current and unsafe > settings as default and add a simmple API to enable limitations and > protections. > > IMHO Benjamin is right, given that this attack has been known to exist since 2003. Moreover, as it appears that no changes whatsoever are going to make it into 2.7, I don't see why patching of 3.1, 3.2 and 3.3 is needed. As for 3.4, it can't hurt to add an opt-in option for a safe mode to the affected libraries. * review of the changes to expat, pyexpat and _elementtree. Antoine, > Brett and Fred Drake have done some reviews. > > I'll gladly review the _elementtree changes and can help with the expat & pyexpat changes as well. Until now I had the impression that the patches aren't ready for review yet. If they are, that's great. Do you have a patch in the issue tracker (so it can be reviewed with Rietveld)? ISTM the current form is just a file (say _elementtree.c) in your Bitbucket repo. Should that be just diffed with the trunk file to see the changes? Eli ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of XML fixes
Eli Bendersky, 17.03.2013 19:25: > IMHO Benjamin is right, given that this attack has been known to exist > since 2003. Moreover, as it appears that no changes whatsoever are going to > make it into 2.7, I don't see why patching of 3.1, 3.2 and 3.3 is needed. > As for 3.4, it can't hurt to add an opt-in option for a safe mode to the > affected libraries. Why keep the libraries vulnerable for another year (3.4 final is expected for early 2014), if there is something we can do about them now? The fact that the attacks have been known for a decade doesn't mean an attacker will need another ten years to exploit them. Stefan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of XML fixes
On Sun, 17 Mar 2013 20:00:19 +0100 Stefan Behnel wrote: > Eli Bendersky, 17.03.2013 19:25: > > IMHO Benjamin is right, given that this attack has been known to exist > > since 2003. Moreover, as it appears that no changes whatsoever are going to > > make it into 2.7, I don't see why patching of 3.1, 3.2 and 3.3 is needed. > > As for 3.4, it can't hurt to add an opt-in option for a safe mode to the > > affected libraries. > > Why keep the libraries vulnerable for another year (3.4 final is expected > for early 2014), if there is something we can do about them now? Well, Christian said that his stdlib patch wasn't ready yet. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of XML fixes
On Sun, Mar 17, 2013 at 12:00 PM, Stefan Behnel wrote: > Eli Bendersky, 17.03.2013 19:25: > > IMHO Benjamin is right, given that this attack has been known to exist > > since 2003. Moreover, as it appears that no changes whatsoever are going > to > > make it into 2.7, I don't see why patching of 3.1, 3.2 and 3.3 is needed. > > As for 3.4, it can't hurt to add an opt-in option for a safe mode to the > > affected libraries. > > Why keep the libraries vulnerable for another year (3.4 final is expected > for early 2014), if there is something we can do about them now? The fact > that the attacks have been known for a decade doesn't mean an attacker will > need another ten years to exploit them. > I'm using a conditional argument here. *If* we don't deem the changes important enough to go into 2.7, *then* they aren't important enough to go into 3.1 and 3.2; 3.3 is a question. That's because 2.7 is arguably more important in this respect, having no direct upgrade path, whereas for 3.x users the fix will be available with 3.4 anyway. Eli ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] peps: New DSL syntax and slightly changed semantics for the Argument Clinic DSL.
[PEP 436 revised syntax]
While I like the syntax better and appreciate the option to condense the
function declaration I still fear that the amount of implicitness will
distract from what is important: programming in C.
This applies especially if people start declaring converters using the
[python] feature.
So I hope that at least converters can be declared statically in a header
file, like I suggested in PEP 437.
A couple of comments:
> As of CPython 3.3, builtin functions nearly always parse their arguments
> with one of two functions: the original ``PyArg_ParseTuple()``, [1]_ and
> the more modern ``PyArg_ParseTupleAndKeywords()``. [2]_ The former
> only handles positional parameters; the latter also accommodates keyword
> and keyword-only parameters, and is preferred for new code.
What is the source for this? I seem to remember a discussion on python-ideas
(but cannot find it now) where some developers preferred non-keyword functions
for some use cases.
For example it's strange to write div(x=10, y=3), or worse, div(y=3, x=10).
Using positional-only arguments prevents this "feature".
> /*[clinic]
> os.stat as os_stat_fn -> stat result
>
>path: path_t(allow_fd=1)
>Path to be examined; can be string, bytes, or open-file-descriptor int.
I do not see where the C initialization or the cleanup are specified. Are
they part of the converter specification?
> /*[clinic]
> curses.window.addch
>
>[
>x: int
> X-coordinate.
>
>y: int
> Y-coordinate.
>]
The parameters appear to be in the wrong order.
> The return annotation is also optional. If skipped, the arrow ("``->``")
> must also be omitted.
Why is it optional? Aren't type annotations important?
> Clinic will ship with a number of built-in converters; new converters can
> also be added dynamically.
How are the converters specified? Inside the preprocessor source? Are
initialization
and cleanup part of the specification, e.g. is a converter represented by a
class?
I would prefer if the converters were in a header file, like I suggested in
PEP 437. Any tool can read such a file and custom converters can be redeclared
above their definition.
> The default value is dynamically assigned, "live" in the generated C code,
> and although it's specified as a Python value, it's translated into a native
> C value in the generated C code. Few default values are permitted, owing to
> this manual translation step.
I think there should be a table that lists which values are converted and what
the result of the conversion is.
> ``[``
>Establishes the start of an optional "group" of parameters.
>Note that "groups" may nest inside other "groups".
>See `Functions With Positional-Only Parameters`_ below.
I don't quite understand the terminology: Functions with the ``/`` are also
"positional-only". Why not reserve this syntax exclusively for the legacy
left-and-right optional case?
> ``/``
>This hints to Argument Clinic that this function is performance-sensitive,
>and that it's acceptable to forego supporting keyword parameters when
> parsing.
>(In early implementations of Clinic, this will switch Clinic from
> generating
>code using ``PyArg_ParseTupleAndKeywords`` to using ``PyArg_ParseTuple``.
>The hope is that in the future there will be no appreciable speed
> difference,
>rendering this syntax irrelevant and deprecated but harmless.)
Here I would use "positional-only" and mention that the slash plays essentially
the same role as the vertical bar in the existing syntax. If this isn't the
intention, then I simply did not understand the paragraph.
> types
>
>A list of strings representing acceptable Python types for this object.
>There are also four strings which represent Python protocols:
I don't quite follow: Aren't input types always specified by the converter
function?
> Argument Clinic also permits embedding Python code inside C files, which
> is executed in-place when Argument Clinic processes the file. Embedded code
> looks like this:
The example in posixmodule.c takes up a lot of space and from the perspective
of auditing the effects it's a little like following a longjmp.
Stefan Krah
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of XML fixes
On Mar 17, 2013, at 05:37 PM, Christian Heimes wrote: >Any attempt to fix the XML issues *will* change the behavior of the >library and result into an incompatibility with older releases. Benjamin >doesn't want to change the behavior of our XML libraries. IIRC Georg and >Barry are +0. I think that we should keep the current and unsafe >settings as default and add a simmple API to enable limitations and >protections. I strongly believe that the decision must be the same for all stable versions. We can't impose the madness of version checks on people for them to know what to do. -Barry ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of XML fixes
On 3/17/2013 8:48 PM, Barry Warsaw wrote: On Mar 17, 2013, at 05:37 PM, Christian Heimes wrote: Any attempt to fix the XML issues *will* change the behavior of the library and result into an incompatibility with older releases. Benjamin doesn't want to change the behavior of our XML libraries. IIRC Georg and Barry are +0. I think that we should keep the current and unsafe settings as default and add a simmple API to enable limitations and protections. I strongly believe that the decision must be the same for all stable versions. We can't impose the madness of version checks on people for them to know what to do. try: newSimpleXMLAPI() newapi = True except Exception: newapi = False ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] peps: New DSL syntax and slightly changed semantics for the Argument Clinic DSL.
On Sun, Mar 17, 2013 at 3:26 PM, Stefan Krah wrote:
> [PEP 436 revised syntax]
>
> While I like the syntax better and appreciate the option to condense the
> function declaration I still fear that the amount of implicitness will
> distract from what is important: programming in C.
>
> This applies especially if people start declaring converters using the
> [python] feature.
>
> So I hope that at least converters can be declared statically in a header
> file, like I suggested in PEP 437.
>
>
> A couple of comments:
>
>
>> As of CPython 3.3, builtin functions nearly always parse their arguments
>> with one of two functions: the original ``PyArg_ParseTuple()``, [1]_ and
>> the more modern ``PyArg_ParseTupleAndKeywords()``. [2]_ The former
>> only handles positional parameters; the latter also accommodates keyword
>> and keyword-only parameters, and is preferred for new code.
>
> What is the source for this? I seem to remember a discussion on python-ideas
> (but cannot find it now) where some developers preferred non-keyword functions
> for some use cases.
>
> For example it's strange to write div(x=10, y=3), or worse, div(y=3, x=10).
> Using positional-only arguments prevents this "feature".
IIRC objection was about functions like abs(5).
If function has single and obvious argument why you need to name that parameter?
The issue has related to documentation for existing one-argument functions only.
>
>
>> /*[clinic]
>> os.stat as os_stat_fn -> stat result
>>
>>path: path_t(allow_fd=1)
>>Path to be examined; can be string, bytes, or open-file-descriptor
>> int.
>
> I do not see where the C initialization or the cleanup are specified. Are
> they part of the converter specification?
>
>
>> /*[clinic]
>> curses.window.addch
>>
>>[
>>x: int
>> X-coordinate.
>>
>>y: int
>> Y-coordinate.
>>]
>
> The parameters appear to be in the wrong order.
>
>
>> The return annotation is also optional. If skipped, the arrow ("``->``")
>> must also be omitted.
>
> Why is it optional? Aren't type annotations important?
>
>
>> Clinic will ship with a number of built-in converters; new converters can
>> also be added dynamically.
>
> How are the converters specified? Inside the preprocessor source? Are
> initialization
> and cleanup part of the specification, e.g. is a converter represented by a
> class?
>
> I would prefer if the converters were in a header file, like I suggested in
> PEP 437. Any tool can read such a file and custom converters can be redeclared
> above their definition.
>
>
>> The default value is dynamically assigned, "live" in the generated C code,
>> and although it's specified as a Python value, it's translated into a native
>> C value in the generated C code. Few default values are permitted, owing to
>> this manual translation step.
>
> I think there should be a table that lists which values are converted and what
> the result of the conversion is.
>
>
>> ``[``
>>Establishes the start of an optional "group" of parameters.
>>Note that "groups" may nest inside other "groups".
>>See `Functions With Positional-Only Parameters`_ below.
>
> I don't quite understand the terminology: Functions with the ``/`` are also
> "positional-only". Why not reserve this syntax exclusively for the legacy
> left-and-right optional case?
>
>
>> ``/``
>>This hints to Argument Clinic that this function is performance-sensitive,
>>and that it's acceptable to forego supporting keyword parameters when
>> parsing.
>>(In early implementations of Clinic, this will switch Clinic from
>> generating
>>code using ``PyArg_ParseTupleAndKeywords`` to using ``PyArg_ParseTuple``.
>>The hope is that in the future there will be no appreciable speed
>> difference,
>>rendering this syntax irrelevant and deprecated but harmless.)
>
> Here I would use "positional-only" and mention that the slash plays
> essentially
> the same role as the vertical bar in the existing syntax. If this isn't the
> intention, then I simply did not understand the paragraph.
>
>
>> types
>>
>>A list of strings representing acceptable Python types for this object.
>>There are also four strings which represent Python protocols:
>
> I don't quite follow: Aren't input types always specified by the converter
> function?
>
>
>> Argument Clinic also permits embedding Python code inside C files, which
>> is executed in-place when Argument Clinic processes the file. Embedded code
>> looks like this:
>
> The example in posixmodule.c takes up a lot of space and from the perspective
> of auditing the effects it's a little like following a longjmp.
>
>
>
> Stefan Krah
>
>
>
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com
--
Thanks,
Andrew Svetlov
___
P
Re: [Python-Dev] Status of XML fixes
On Mar 17, 2013, at 09:16 PM, Glenn Linderman wrote: >try: >newSimpleXMLAPI() >newapi = True >except Exception: >newapi = False try: True except NameError: True = 1 False = 0 -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of XML fixes
2013/3/17 Barry Warsaw : > On Mar 17, 2013, at 09:16 PM, Glenn Linderman wrote: > >>try: >>newSimpleXMLAPI() >>newapi = True >>except Exception: >>newapi = False > > try: > True > except NameError: > True = 1 > False = 0 > > -Barry I understand why your bedtime is 21:30. :) -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
