On 2022-09-29 23:41, Jen Kris wrote:
I just solved this C API problem, and I’m posting the answer to help
anyone else who might need it.
The errors were:
(1) we must call Py_INCREF on each object when it’s created.
Some functions return an object that has already been incref'ed ("new
reference"). This occurs when it has either created a new object (the
refcount will be 1) or has returned a pointer to an existing object (the
refcount will be > 1 because it has been incref'ed).
Other functions return an object that hasn't been incref'ed. This occurs
when you're looking up something, for example, looking at a member of a
list or the value of an attribute.
(2) in C_API_2 (see below) we don’t cast value_1 as I did before with
PyObject * value_ptr = (PyObject * )value_1. Instead we use PyObject
* value_ptr = PyLong_FromLong(value_1);
(3) The command string to PyObject_CallFunctionObjArgs must be null
terminated.
Always read the docs carefully!
Here’s the revised code:
First we load the modules, and increment the reference to each object:
int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);
Possible null pointers here.
Py_INCREF(pName_random);
Py_INCREF(pMod_random);
Py_INCREF will fail on a null pointer, which you haven't check for yet.
If they aren't null, then they're new references (incref'ed) that you've
incref'ed.
if (pMod_random == 0x0){
PyErr_Print();
Leaks here because of the refcount.
return 1;}
PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
"randrange");
Possible null pointers here too.
Py_INCREF(pAttr_seed);
Py_INCREF(pAttr_randrange);
Same issue as above.
return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;
return 0;
}
Next we call a program to initialize the random number generator with
random.seed(), and increment the reference to its return value
p_seed_calc:
int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * value_ptr = PyLong_FromLong(value_1);
New reference.
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
value_ptr, NULL);
// _________
if (p_seed_calc == 0x0){
PyErr_Print();
Leak.
return 1;}
Py_INCREF(p_seed_calc);
p_seed_calc will be a new reference to Py_None, incref'ed again, so leak.
return 0;
}
Now we call another program to get a random number:
int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1)
{
PyObject * value_ptr = PyLong_FromLong(value_1);
PyObject * p_randrange_calc =
PyObject_CallFunctionObjArgs(pAttr_randrange, value_ptr, NULL);
if (p_randrange_calc == 0x0){
Leak.
PyErr_Print();
return 1;}
//Prepare return values
long return_val = PyLong_AsLong(p_randrange_calc);
Leak.
return return_val;
}
That returns 28, which is what I get from the Python command line.
Thanks again to MRAB for helpful comments.
Jen
Getting refcounting right can be difficult!
Sep 29, 2022, 15:31 by pyt...@mrabarnett.plus.com:
On 2022-09-29 21:47, Jen Kris wrote:
To update my previous email, I found the problem, but I have a
new problem.
Previously I cast PyObject * value_ptr = (PyObject * )value_1
but that's not correct. Instead I used PyObject * value_ptr =
PyLong_FromLong(value_1) and that works. HOWEVER, while
PyObject_CallFunctionObjArgs does work now, it returns -1,
which is not the right answer for random.seed. I use "long
return_val = PyLong_AsLong(p_seed_calc);" to convert it to a long.
random.seed returns None, so when you call
PyObject_CallFunctionObjArgs it returns a new reference to Py_None.
If you then pass to PyLong_AsLong a reference to something that's
not a PyLong, it'll set an error and return -1.
So my question is why do I get -1 as return value? When I
query p_seed calc : get:
(gdb) p p_seed_calc
$2 = (PyObject *) 0x7ffff69be120 <_Py_NoneStruct>
Exactly. It's Py_None, not a PyLong.
Thanks again.
Jen
Sep 29, 2022, 13:02 by python-list@python.org:
Thanks very much to @MRAB for taking time to answer. I changed my
code to conform to your answer (as best I understand your comments
on references), but I still get the same error. My comments
continue below the new code immediately below.
int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);
Py_INCREF(pName_random);
Py_INCREF(pMod_random);
if (pMod_random == 0x0){
PyErr_Print();
return 1;}
PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random,
"seed");
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
"randrange");
Py_INCREF(pAttr_seed);
Py_INCREF(pAttr_randrange);
return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;
return 0;
}
int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * value_ptr = (PyObject * )value_1;
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
value_ptr, NULL);
if (p_seed_calc == 0x0){
PyErr_Print();
return 1;}
//Prepare return values
long return_val = PyLong_AsLong(p_seed_calc);
return return_val;
}
So I incremented the reference to all objects in Get_LibModules,
but I still get the same segfault at
PyObject_CallFunctionObjArgs. Unfortunately, reference counting
is not well documented so I’m not clear what’s wrong.
Sep 29, 2022, 10:06 by pyt...@mrabarnett.plus.com:
On 2022-09-29 16:54, Jen Kris via Python-list wrote:
Recently I completed a project where I used
PyObject_CallFunctionObjArgs extensively with the NLTK
library from a program written in NASM, with no problems.
Now I am on a new project where I call the Python random
library. I use the same setup as before, but I am getting
a segfault with random.seed.
At the start of the NASM program I call a C API program
that gets PyObject pointers to “seed” and “randrange” in
the same way as I did before:
int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);
Both PyUnicode_FromString and PyImport_Import return new
references or null pointers.
if (pMod_random == 0x0){
PyErr_Print();
You're leaking a reference here (pName_random).
return 1;}
PyObject * pAttr_seed =
PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange =
PyObject_GetAttrString(pMod_random, "randrange");
return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;
You're leaking 2 references here (pName_random and pMod_random).
return 0;
}
Later in the same program I call a C API program to call
random.seed:
int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * p_seed_calc =
PyObject_CallFunctionObjArgs(pAttr_seed, value_1);
It's expecting all of the arguments to be PyObject*, but
value_1 is Py_ssize_t instead of PyObject* (a pointer to a
_Python_ int).
The argument list must end with a null pointer.
It returns a new reference or a null pointer.
if (p_seed_calc == 0x0){
PyErr_Print();
return 1;}
//Prepare return values
long return_val = PyLong_AsLong(p_seed_calc);
You're leaking a reference here (p_seed_calc).
return return_val;
}
The first program correctly imports “random” and gets
pointers to “seed” and “randrange.” I verified that the
same pointer is correctly passed into C_API_2, and the
seed value (1234) is passed as Py_ssize_t value_1. But I
get this segfault:
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff64858d5 in _Py_INCREF (op=0x4d2) at
../Include/object.h:459
459 ../Include/object.h: No such file or directory.
So I tried Py_INCREF in the first program:
Py_INCREF(pMod_random);
Py_INCREF(pAttr_seed);
Then I moved Py_INCREF(pAttr_seed) to the second program.
Same segfault.
Finally, I initialized “random” and “seed” in the second
program, where they are used. Same segfault.
The segfault refers to Py_INCREF, so this seems to do with
reference counting, but Py_INCREF didn’t solve it.
I’m using Python 3.8 on Ubuntu.
Thanks for any ideas on how to solve this.
Jen
-- https://mail.python.org/mailman/listinfo/python-list
-- https://mail.python.org/mailman/listinfo/python-list
--
https://mail.python.org/mailman/listinfo/python-list
--
https://mail.python.org/mailman/listinfo/python-list
