Porting c extension - PyBuffer_New() deprecated in python3. What's the replacement?
Hi
I need to convert the following existing c extension code to support
Python 3.
// --- existing code --
// PyBuffer_New() deprecated in python3
if (!(pyBuf = PyBuffer_New(len)))
{
return NULL;
}
// should use memoryview object in python3
if (PyObject_AsWriteBuffer(pyBuf, &cbuf, &len))
{
Py_DECREF(pyBuf);
return NULL ;
}
// fill in cbuf
...
return pyBuf ;
//---
I'm somewhat confounded in finding an equivalent (PyBuffer_New()) for
creating a buffer of size len that has continuous memory in the c
extension function for python3. cbuf is manipulated/filled in using c,
after which the created pyBuf is then returned. So far, I haven't found
much in the way of examples/doc for porting the deprecated
Python-/C-level buffer API calls to the new C-level buffer
API/memoryview object model.
Any guidance or direction to existing doc/example is much appreciated.
TIA.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Porting c extension - PyBuffer_New() deprecated in python3. What's the replacement?
Stefan,
Thank-you for the reply. I hadn't considered cpython, unfortunately the
extension is too large a project to port at the moment. I ended up
replacing the PyBuffer_New() segment with malloc() and passing back an
object from PyByteArray_FromStringAndSize(). It seems to work.
mrh.
On 2014-01-11 01:10, Stefan Behnel wrote:
Mark Heieis, 11.01.2014 06:47:
I need to convert the following existing c extension code to support Python 3.
// --- existing code --
// PyBuffer_New() deprecated in python3
if (!(pyBuf = PyBuffer_New(len)))
{
return NULL;
}
// should use memoryview object in python3
if (PyObject_AsWriteBuffer(pyBuf, &cbuf, &len))
{
Py_DECREF(pyBuf);
return NULL ;
}
// fill in cbuf
...
return pyBuf ;
//---
I'm somewhat confounded in finding an equivalent (PyBuffer_New()) for
creating a buffer of size len that has continuous memory in the c extension
function for python3. cbuf is manipulated/filled in using c, after which
the created pyBuf is then returned. So far, I haven't found much in the way
of examples/doc for porting the deprecated Python-/C-level buffer API calls
to the new C-level buffer API/memoryview object model.
Any guidance or direction to existing doc/example is much appreciated.
If the extension isn't huge, you should consider rewriting it in Cython.
That can usually be done quite quickly - the main thing is to figure out
what the verbose C code actually does and write it down in much simpler
Python code. And it will make it easy to make the code portable and fast.
Also likely safer and more generic and versatile, because Cython covers
away a lot of the annoying boilerplate, ref-counting issues, type
conversions, etc.
For your specific problem at hand, you could use Cython's memory views:
http://docs.cython.org/src/userguide/memoryviews.html
They allow you to convert the input value to a 1-dim char buffer (or
whatever you need, but you mentioned the old Py2 buffer interface, which
can't do much more) by saying
cdef char[:] my_memview = some_python_object
If you need to pass the unpacked buffer into C code, you can get the
address as "&my_memview[0]" (i.e. the address of the first item in the
buffer). Memory views themselves support fast slicing and indexing, so you
can efficiently work with them using the normal Python slicing/indexing syntax.
In case what you actually receive are not arbitrary buffers but simple byte
strings or bytearray instances, you can even use the normal byte string
coercion in Cython and simply say
cdef char* c_string = some_python_byte_string_object
and then use that pointer to pass it on into C.
I've written a string processing tutorial for Cython here:
http://docs.cython.org/src/tutorial/strings.html
These things may take a moment to learn, especially if you are used to
doing everything in excessive manual detail in C code, but once you are
through that, you should get things done much more quickly than when trying
to do them by hand.
Stefan
--
https://mail.python.org/mailman/listinfo/python-list
autoconf tools and python3 3m 3dm
Hi,
I've been migrating a python2 package+extension to python3. The problem
I'm running into is with ./configure and which version it picks up or
doesn't in this case.
The default is python2 and works just fine as expected.
However, when ./configure PYTHON=python3 is run, the problems occur.
"./configure" picks python3 correctly and propagates that through to all
of the makefiles created. Good so far. It defines all of the appropriate
libs and include with the '3.3' extension.
The yum installed python3, however, is denoted by a suffix 'dm',
yielding an extension of '3.3dm' (Fedora 20)
Building and installing from source, python has a suffix 'm', so the
extension is '3.3m;
In ./configure, PYTHON_VERSION is set by "$PYTHON -c 'import sys;
print(sys.version[:3])'" which reports "3.3". PYTHON_VERSION is then
used as the suffix to find all of the appropriate python directories,
libraries, site-packages and includes. The same result is had if
python3.3m or python3.3dm is used.
Unfortunately, this isn't enough, as the python site packages are
installed with naming ending in 'm' or 'dm', resulting in all of the
makefiles failing because they can't find the libraries, include files,
or site packages.
I've kludged a solution by manually creating soft links in the lib and
include directories which almost solves the problem:
for example in /usr/lib64:
lrwxrwxrwx. 1 root root 19 2013-12-18 15:02 libpython2.7.so ->
libpython2.7.so.1.0
-r-xr-xr-x. 1 root root 1800480 2013-11-12 08:47 libpython2.7.so.1.0
lrwxrwxrwx 1 root root 21 2014-01-20 16:11 libpython3.3dm.so ->
libpython3.3dm.so.1.0
-rwxr-xr-x 1 root root 3057640 2013-11-07 02:03 libpython3.3dm.so.1.0
lrwxrwxrwx 1 root root 20 2014-01-20 16:11 libpython3.3m.so ->
libpython3.3m.so.1.0
-rwxr-xr-x 1 root root 2498992 2013-11-07 02:03 libpython3.3m.so.1.0
lrwxrwxrwx 1 root root 20 2014-01-20 16:37 libpython3.3.so ->
libpython3.3m.so.1.0
-rwxr-xr-x 1 root root6704 2013-11-07 02:03 libpython3.so
This approach doesn't feel right to me. I don't think pythonX.Y-config
would work either as one would need to know in advance specifically
which one to call and there'd be extra work to extract the full version
info, etc. ("$python3-config --includes" yields
-I/usr/include/python3.3m -I/usr/include/python3.3m)
Has anyone else come up against this? If so, what was the solution?
BTW - anyone know what the 'm' and 'dm' suffixes indicate?
TIA.
--
https://mail.python.org/mailman/listinfo/python-list
