[Python-Dev] How is obmalloc safe with "Invalid read of size 4" ?

2015-03-24 Thread Karl Pickett 
We are having random, rare, nonreproducible segfaults/hangs with python2 on
ubuntu 14.04 in EC2.  I've managed to attach GDB to some hung ones and
there looks like clear memory corruption in the 'interned' hash table,
causing lookdict_string() to spin forever because all remaining slots have
a garbage 'key' pointer.  This happens just loading the 'site' module
dependencies, like 're' or 'codecs', before any of our code even gets run.

So we then tried running it under valgrind, and we got a lot of nasty
errors.  Even after reading the Misc/README.valgrind, which talks about
*uninitialized* reads being ok, I still don't see how reading from *freed*
memory would ever be safe, and why the suppression file thinks thats ok:

$ valgrind   ./pymd79/bin/python -c ""
==14651== Memcheck, a memory error detector
==14651== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==14651== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
==14651== Command: ./pymd79/bin/python -c
==14651==
==14651== Invalid read of size 4
==14651==at 0x461E40: Py_ADDRESS_IN_RANGE (obmalloc.c:1911)
==14651==by 0x461EA3: PyObject_Free (obmalloc.c:994)
==14651==by 0x4789AB: tupledealloc (tupleobject.c:235)
==14651==by 0x5225BA: code_dealloc (codeobject.c:309)
==14651==by 0x4CFFC3: load_source_module (import.c:1100)
==14651==by 0x4D0E16: import_submodule (import.c:2700)
==14651==by 0x4D1E19: PyImport_ImportModuleLevel (import.c:2515)
==14651==by 0x4AE49A: builtin___import__ (bltinmodule.c:49)
==14651==by 0x422C89: PyObject_Call (abstract.c:2529)
==14651==by 0x4B12E5: PyEval_EvalFrameEx (ceval.c:3902)
==14651==by 0x4B6A47: PyEval_EvalCodeEx (ceval.c:3265)
==14651==by 0x4B6B71: PyEval_EvalCode (ceval.c:667)
==14651==  Address 0x5bcd020 is 2,256 bytes inside a block of size 2,801
free'd
==14651==at 0x4C28577: free (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==14651==by 0x4DB2B0: PyMarshal_ReadLastObjectFromFile (marshal.c:1145)
==14651==by 0x4CFE71: load_source_module (import.c:801)

- Karl
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How is obmalloc safe with "Invalid read of size 4" ?

2015-03-24 Thread Hrvoje Niksic 

On 03/24/2015 03:28 PM, Karl Pickett wrote:

So we then tried running it under valgrind, and we got a lot of nasty
errors.  Even after reading the Misc/README.valgrind, which talks about
*uninitialized* reads being ok, I still don't see how reading from
*freed* memory would ever be safe, and why the suppression file thinks
thats ok:


PyObject_Free() is not reading *freed* memory, it is reading memory 
outside (right before) the allocated range. This is, of course, 
undefined behavior as far as C is concerned and an invalid read in the 
eyes of valgrind. Still, it's the kind of thing you can get away with if 
you are writing a heavily optimized allocator (and if your name starts 
with "Tim" and ends with "Peters").


README.valgrind explains in quite some detail why this is done. In 
short, it allows for a very fast check whether the memory passed to 
PyObject_Free() was originally allocated by system malloc or by Python's 
pool allocator.


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PiCxx

2015-03-24 Thread π 
Hello PythonDevvers,

I apologise in advance. This is slightly off topic. This will be my only post 
on the subject.

PyCXX (http://sourceforge.net/projects/cxx/ 
) accomplishes roughly the same as 
Boost::Python (C++ wrapper for CPython), only without requiring Boost.

The original code has become tangled and bloated.  
Over the winter I've rewritten it using C++11.  New C++11 features have allowed 
for more direct/compact/concise/readable code.

I've called my rewrite PiCxx and put it up here: https://github.com/p-i-/PiCxx 


PiCxx only supports Python3 currently, as that is all I need for my own use.  
It wouldn't be too much work to add 2.x support.  Also I've only tested it on 
my OSX system (although it is designed to be cross-platform).  It offers an 
alternative to Boost::Python that doesn't require Boost.

Improvements, suggestions, bug fixes, etc are all welcome.

Well, that's all. I originally subscribed to this list because I thought I 
might need some help navigating some of the dark corners of the CPython API, 
but thanks to a handful of Samurai on SO and IRC I seem to have scraped through.

I will unsubscribe in due course; it's been charming to spend awhile in the 
belly of the snake, and humbling to witness how hard you guys work.

Happy spring everyone,

π___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PiCxx

2015-03-24 Thread Terry Reedy 

On 3/24/2015 11:53 AM, π wrote:

Hello PythonDevvers,

I apologise in advance. This is slightly off topic. This will be my only
post on the subject.


pydev is about development *of* Python the language and CPython the 
implementation.  python-list is about development *with* Python.  This 
post would be on-topic for python-list and would have fit better there.



PyCXX (http://sourceforge.net/projects/cxx/) accomplishes roughly the
same as Boost::Python (C++ wrapper for CPython), only without requiring
Boost.

The original code has become tangled and bloated.
Over the winter I've rewritten it using C++11.  New C++11 features have
allowed for more direct/compact/concise/readable code.

I've called my rewrite PiCxx and put it up here:
https://github.com/p-i-/PiCxx
Improvements, suggestions, bug fixes, etc are all welcome.


Announcements and requests like this are fairly routine on python-list. 
 Sometimes they get responses and discussion on the list, sometimes not.


One can participate on python-list via news/web mirror 
gmane.comp.python.general at news.gmane.org.


--
Terry Jan Reedy


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] libffi embedded in CPython

2015-03-24 Thread Paul Moore 
On 12 March 2015 at 17:44, Paul Moore  wrote:
> On 12 March 2015 at 17:26, Brett Cannon  wrote:
>>> I'm all for ditching our 'libffi_msvc' in favor of adding libffi as
>>> another 'external' for the Windows build.  I have managed to get
>>> _ctypes to build on Windows using vanilla libffi sources, prepared
>>> using their configure script from within Git Bash and built with our
>>> usual Windows build system (properly patched).  Unfortunately, making
>>> things usable will take some work on ctypes itself, which I'm not
>>> qualified to do. I'm happy to pass on my procedure and patches for
>>> getting to the point of successful compilation to anyone who feels up
>>> to fixing the things that are broken.
>>
>>
>> So it seems possible to use upstream libffi but will require some work.
>
> I'd be willing to contemplate helping out on the Windows side of
> things, if nobody else steps up (with the proviso that I have little
> free time, and I'm saying this without much idea of what's involved
> :-)) If Zachary can give a bit more detail on what the work on ctypes
> is, and/or put what he has somewhere that I could have a look at, that
> might help.

One thing that seems to be an issue. On Windows, ctypes detects if the
FFI call used the wrong number of arguments off the stack, and raises
a ValueError if it does. The tests rely on that behaviour. But it's
based on ffi_call() returning a value, which upstream libffi doesn't
do. As far as I can tell (not that the libffi docs are exactly
comprehensive...) there's no way of getting that information from
upstream libffi.

What does Unix ctypes do when faced with a call being made with the
wrong number of arguments? On Windows, using upstream libffi and
omitting the existing check, it seems to crash the Python process,
which obviously isn't good. But the test that fails is
Windows-specific, and short of going through all the tests looking for
one that checks passing the wrong number of arguments and isn't
platform-specific, I don't know how Unix handles this.

Can anyone on Unix tell me if a ctypes call with the wrong number of
arguments returns ValueError on Unix? Something like strcmp() (with no
args) should do as a test, I guess...

If there's a way Unix handles this, I can see about replicating it on
Windows. But if there isn't, I fear we could always need a patched
libffi to maintain the interface we currently have...

Thanks,
Paul
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com