Re: [Python-Dev] Use of Cython

[Ronald Oussoren via Python-Dev]

> On 5 Aug 2018, at 03:15, Nick Coghlan  wrote:
> 
> On 5 August 2018 at 00:46, Stefan Behnel  wrote:
>> Antoine Pitrou schrieb am 04.08.2018 um 15:57:
>>> Actually, I think testing the C API is precisely the kind of area where
>>> you don't want to involve a third-party, especially not a moving target
>>> (Cython is actively maintained and generated code will vary after each
>>> new Cython release).  Besides, Cython itself calls the C API, which
>>> means you might end up involuntarily testing the C API against itself.
>>> 
>>> If anything, testing the C API using ctypes or cffi would probably be
>>> more reasonable... assuming we get ctypes / cffi to compile everywhere,
>>> which currently isn't the case.
>> 
>> I agree that you would rather not want to let Cython (or another tool)
>> generate the specific code that tests a specific C-API call, but you could
>> still use Cython to get around writing the setup, validation and unittest
>> boilerplate code in C. Basically, a test could then look something like
>> this (probably works, although I didn't test it):
>> 
>>from cpython.object cimport PyObject
>>from cpython.list cimport PyList_Append
>> 
>>def test_PyList_Append_on_empty_list():
>># setup code
>>l = []
>>assert len(l) == 0
>>value = "abc"
>>pyobj_value =  value
>>refcount_before = pyobj_value.ob_refcnt
>> 
>># conservative test call, translates to the expected C code,
>># although with exception propagation if it returns -1:
>>errcode = PyList_Append(l, value)
>> 
>># validation
>>assert errcode == 0
>>assert len(l) == 1
>>assert l[0] is value
>>assert pyobj_value.ob_refcnt == refcount_before + 1
>> 
>> 
>> If you don't want the exception handling, you can define your own
>> declaration of PyList_Append() that does not have it. But personally, I'd
>> rather use try-except in my test code than manually taking care of cleaning
>> up (unexpected) exceptions.
> 
> Exactly, that's the kind of thing I had in mind. At the moment,
> writing a new dedicated C API test requires designing 4 things:
> 
> 1. The test case itself (what action to take, which assertions to make about 
> it)
> 2. The C code to make the API call you want to test
> 3. The Python->C interface for the test case from 1 to pass test
> values in to the code from 2
> 4. The C->Python interface to get state of interest from 2 back to the
> test case from 1
> 
> If we were able to use Cython to handle 3 & 4 rather than having to
> hand craft it for every test, then I believe it would significantly
> lower the barrier to testing the C API directly rather than only
> testing it indirectly through the CPython implementation.
> 
> Having such a test suite available would then hopefully make it easier
> for other implementations to provide robust emulations of the public C
> API.
> 
> ctypes & cffi likely wouldn't help as much in the case, since they
> don't eliminate the need to come up with custom code for parts 3 & 4,
> they just let you write that logic in Python rather than C.

I’m not sure if I understand this, ctypes and cffi are used to access C APIs 
without writing C code including the CPython API (see for example 
>). 

The code code below should be mostly equivalent to the Cython example posted 
earlier:

import unittest
import ctypes
from ctypes import pythonapi

class PyObject(ctypes.Structure):
_fields_ = (
('ob_refcnt', ctypes.c_ssize_t),
)

pythonapi.PyList_Append.argtypes = [ctypes.py_object, ctypes.py_object]

def refcount(v):
return PyObject.from_address(id(v)).ob_refcnt


def test_PyList_Append_on_empty_list():
   # setup code
   l = []
   assert len(l) == 0
   value = "abc"


   refcount_before = refcount(value)

   errcode = pythonapi.PyList_Append(l, value)

   assert errcode == 0
   assert len(l) == 1
   assert l[0] is value
   assert refcount(value) == refcount_before + 1

I write “mostly” because I rarely use ctypes and am not 100% sure that I use 
the API correctly.

A problem with using ctypes is that this tests the ABI and to the API, which 
for example means you cannot test C macros this way.

Ronald

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use of Cython

[Nick Coghlan]

On 5 August 2018 at 18:06, Ronald Oussoren  wrote:
> I’m not sure if I understand this, ctypes and cffi are used to access C APIs
> without writing C code including the CPython API (see for example
> ).
>
> The code code below should be mostly equivalent to the Cython example posted
> earlier:
>
> import unittest
> import ctypes
> from ctypes import pythonapi
>
> class PyObject(ctypes.Structure):
> _fields_ = (
> ('ob_refcnt', ctypes.c_ssize_t),
> )
>
> pythonapi.PyList_Append.argtypes = [ctypes.py_object, ctypes.py_object]
>
> def refcount(v):
> return PyObject.from_address(id(v)).ob_refcnt

The quoted code is what I was referring to in:

ctypes & cffi likely wouldn't help as much in the case, since they
don't eliminate the need to come up with custom code for parts 3 & 4,
they just let you write that logic in Python rather than C.


Cython has more machinery for accessing the CPython C API correctly
already built in to it, whereas ctypes has no type safety at all,
while cffi doesn't special case CPython's C API in particular.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] [RELEASED] Python 3.4.9 and Python 3.5.6 are now available

[Michael]

On 03/08/2018 03:22, Larry Hastings wrote:



On 08/02/2018 07:17 AM, Victor Stinner wrote:

3.4.9 and 3.5.6 have no more known security vulnerabilities :-)


Well, not to be a complete pill, but...

   https://bugs.python.org/issue17180
   https://bugs.python.org/issue17239
   https://bugs.python.org/issue19050

Sadly, just because they're languishing on bpo doesn't mean they 
aren't valid security vulnerabilities.


+1 - Sadly, not fixed after 5 years - Why? Because it isn't sexy, or 
fear for breaking things?


Breaking things could be valid - when it is a feature/design change, but 
the whole point of security fixes is because we believe the security 
vulnerability is breakage. Not fixing it keeps everything that depends 
on it (intentional or not) also broken. Any app that depends on 'broken' 
behavior needs to be fixed - rather than let a known vulnerability go 
from 0-day to 1825-day vulnerability (or is it 2000 already?)


Only read the discussion for 17180 - but it seems anything old does not 
get fixed because it did not get fixed years ago.


my two cents!

On a side note: I have been trying to test python on different 
"enterprise" distros of linux and am amazed to see Python2-2.7.5 as the 
'standard'. Rather disheartening for the all the good work that gets 
done. i.e., I am amazed that CVE's like the ones fixed in 3.4.9 and 
3.5.6 (and maybe already/later in 2.7.X) do not motivate distributions 
to update to current levels.


oh my - up to 4 cents! :)

Thanks for the work - I'll get to packaging them for AIX.



//arry/



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/aixtools%40felt.demon.nl



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] AIX and python tests

[Michael]

As I have time, I'll dig into these.

I have a couple of PR already 'out there', which I hope someone will be 
looking at when/as he/she/they have time. My time will also be intermittent.


My next test - and I hope not too difficult - would be the test_utf8. 
The test:


FAIL: test_cmd_line (test.test_utf8_mode.UTF8ModeTests) fails - and I am 
wondering if it is as simple as AIX default mode is ISO8559-1 and the 
test looks to be comparing UTF8 with the locale_default. If that is the 
case, obviously this test will never succeed - asis. Am I understanding 
the test properly. If yes, then I'll see what I can come up with for a 
patch to the test for AIX. If no, I'll need some hand holding to help me 
understand the test A bigger challenge, and I think a major issue with 
many of the test failures is test_ssl. Here I already know I'll need so 
assistance. I am quite lost. I know AIX at an expert level, but I do not 
know python (especially python internals, macros, etc..) and after about 
3 levels I am lost. I also find it hard to get 'artifacts' from the 
tests to know what is expected. Looking forward to assistance from 
various people - in understanding the tests, and probably better python 
coding criticism. Michael


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] [RELEASED] Python 3.4.9 and Python 3.5.6 are now available

[MRAB]

On 2018-08-05 19:57, Michael wrote:

On 03/08/2018 03:22, Larry Hastings wrote:



On 08/02/2018 07:17 AM, Victor Stinner wrote:

3.4.9 and 3.5.6 have no more known security vulnerabilities :-)


Well, not to be a complete pill, but...

https://bugs.python.org/issue17180
https://bugs.python.org/issue17239
https://bugs.python.org/issue19050

Sadly, just because they're languishing on bpo doesn't mean they 
aren't valid security vulnerabilities.


+1 - Sadly, not fixed after 5 years - Why? Because it isn't sexy, or 
fear for breaking things?


[snip]Re https://bugs.python.org/issue19050, on Windows 10, Python 3.6 
and Python 3.7 both work OK and Python 3.5 complains about a bad file 
descriptor.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] The curious case of 255 function arguments

[Stephen McDowell]

Hello Python Gurus,

TL;DR: 3.7 released functions having greater than 255 arguments.  Despite
explicit checks for this in 2.x, no such limit is actually imposed -- why?

In the 3.7 release notes "Other Language Changes" section (
https://docs.python.org/3.7/whatsnew/3.7.html#other-language-changes), the
first bullet point denotes

> More than 255 arguments can now be passed to a function, and a function
can now have more than 255 parameters. (Contributed by Serhiy Storchaka in
bpo-12844  and bpo-18896
.)

Now lets get something straight: unless I want to exclusively support
Python 3.7 or higher, I must make sure I obey the <255 rule.  Use *args //
**kwargs, etc.  I'm totally ok with that, 2020 is already here in my mind ;)

Curiosity is the reason I'm reaching out.  Upon further investigation and
some discussion with like-minded Python enthusiasts, the code being patched
by Serhiy Storchaka is present in e.g., Python 2.7 (
https://github.com/python/cpython/blob/2.7/Python/ast.c#L2013-L2016)

if (nargs + nkeywords + ngens > 255) {
  ast_error(n, "more than 255 arguments");
  return NULL;
}

Despite that code, as demonstrated with the supplemental output in the post
script, *no 2.x versions fail with >255 arguments*.  In contrast, 3.x where
x<7 all do fail (as expected) with a SyntaxError.  To test this, I tried
every minor release of python (excluding v1, arbitrarily choosing the
latest patch release of a minor version) with the following snippet via the
-c flag

/path/to/pythonX.Y -c 'exec("def foo(" + ", ".join(["a" + str(i) for i
in range(1, 300)]) + "): pass")'

Which tries to construct a function

def foo(a0, a1, ..., a299): pass

I've looked at the C code for a while and it is entirely non-obvious what
would lead to python *2* *allowing* >255 arguments.  Anybody happen to know
how / why the python *2* versions *succeed*?

Thank you for reading, this is not a problem, just a burning desire for
closure (even if anecdotal) as to how this can be.  I deeply love python,
and am not complaining!  I stumbled across this and found it truly
confounding, and thought the gurus here may happen to recall what changed
in 3.x that lead the the error condition actually being asserted :)

Sincerely,

Stephen McDowell

P.S. On a Fedora 25 box using GCC 6.4.1, I lovingly scripted the
installation of all the python versions just to see if it truly was a 2.x /
3.x divide.  The results of running `python -V` followed by the `python -c
'exec("def foo...")'` described above, with some extra prints for clarity
are as follows (script hackily thrown together in ~30minutes not included,
so as not to make your eyes bleed):


Python 2.0.1
==> Greater than 255 Arguments supported

Python 2.1.3
==> Greater than 255 Arguments supported

Python 2.2.3
==> Greater than 255 Arguments supported

Python 2.3.7
==> Greater than 255 Arguments supported

Python 2.4.6
==> Greater than 255 Arguments supported

Python 2.5.6
==> Greater than 255 Arguments supported

Python 2.6.9
==> Greater than 255 Arguments supported

Python 2.7.15
==> Greater than 255 Arguments supported

Python 3.0.1
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 1
SyntaxError: more than 255 arguments

Python 3.1.5
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 1
SyntaxError: more than 255 arguments

Python 3.2.6
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 1
SyntaxError: more than 255 arguments

Python 3.3.7
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 1
SyntaxError: more than 255 arguments

Python 3.4.9
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 1
SyntaxError: more than 255 arguments

Python 3.5.6
Traceback (most recent call last):
  File "",