Carl Banks wrote:
On May 4, 12:03 pm, Gary Herron <[EMAIL PROTECTED]> wrote:Alexander Schmolck wrote:Gary Herron <[EMAIL PROTECTED]> writes:But... It's not!A simple test shows that. I've attached a tiny test program that shows thisextremely clearly. Please run it and watch it fail.In [7]: run ~/tmp/t.py final count: 2000000 should be: 2000000(I took the liberty to correct your test to actually do what I said, namelyuse a numpy.array; just replace ``count = 0`` with ``import numpy; count = numpy.array(0)``).The test was meant to simulate the OP's problem, but even with your suggestion of using numpy, it *still* fails!Ok, so numpy scalars don't support += atomically. Thank you for your skepticism in discovering this.
You're welcome.
However, what I said was not wholly untrue: code in C extensions is protected by the GIL and thus not interruptable, unless it either releases the GIL, or calls back into Python code (which is apparently what numpy scalars do).
Yikes. Thanks for that.However, the upshot of all of this is that one must maintain extreme skepticism. Unless you know both your Python code and any extension modules you call, and you know them at a level necessary to find such details, you must conclude that any operation, no matter how atomic it my look, may in fact not be atomic.
Gary Herron
To illustrate, a small extension module is included below. Just to
make things interesting, I added a long busy loop in between reading
and setting the counter, just to give the other thread maximum
opportunity to cut in.
If you were to compile it and run the test, you would find that it
works perfectly.
===========================
/* atomic.c */
#include <Python.h>
#include <structmember.h>
typedef struct {
PyObject_HEAD
long value;
} AtomicCounterObject;
static int init(AtomicCounterObject* self, PyObject* args,
PyObject* kwargs) {
self->value = 0;
return 0;
}
static PyObject* iadd(AtomicCounterObject* self, PyObject* inc) {
long incval = PyInt_AsLong(inc);
long store, i;
static int bigarray[100000];
if (incval == -1 && PyErr_Occurred()) return 0;
store = self->value;
/* Give the thread plenty of time to interrupt */
for (i = 0; i < 100000; i++) bigarray[i]++;
self->value = store + incval;
return (PyObject*)self;
}
static PyObject* toint(AtomicCounterObject* self) {
return PyInt_FromLong(self->value);
}
static PyNumberMethods ac_as_number = {
0, /*nb_add*/
0, /*nb_subtract*/
0, /*nb_multiply*/
0, /*nb_divide*/
0, /*nb_remainder*/
0, /*nb_divmod*/
0, /*nb_power*/
0, /*nb_negative*/
0, /*nb_positive*/
0, /*nb_absolute*/
0, /*nb_nonzero*/
0, /*nb_invert*/
0, /*nb_lshift*/
0, /*nb_rshift*/
0, /*nb_and*/
0, /*nb_xor*/
0, /*nb_or*/
0, /*nb_coerce*/
(unaryfunc)toint, /*nb_int*/
0, /*nb_long*/
0, /*nb_float*/
0, /*nb_oct*/
0, /*nb_hex*/
(binaryfunc)iadd, /*nb_inplace_add*/
0, /*nb_inplace_subtract*/
0, /*nb_inplace_multiply*/
0, /*nb_inplace_divide*/
0, /*nb_inplace_remainder*/
0, /*nb_inplace_power*/
0, /*nb_inplace_lshift*/
0, /*nb_inplace_rshift*/
0, /*nb_inplace_and*/
0, /*nb_inplace_xor*/
0, /*nb_inplace_or*/
0, /* nb_floor_divide */
0, /* nb_true_divide */
0, /* nb_inplace_floor_divide */
0, /* nb_inplace_true_divide */
0, /* nb_index */
};
static PyTypeObject AtomicCounterType = {
PyObject_HEAD_INIT(NULL)
0, /*ob_size*/
"AtomicCounter", /*tp_name*/
sizeof(AtomicCounterObject), /*tp_basicsize*/
0, /*tp_itemsize*/
0, /*tp_dealloc*/
0, /*tp_print*/
0, /*tp_getattr*/
0, /*tp_setattr*/
0, /*tp_compare*/
0, /*tp_repr*/
&ac_as_number, /*tp_as_number*/
0, /*tp_as_sequence*/
0, /*tp_as_mapping*/
0, /*tp_hash */
0, /*tp_call*/
0, /*tp_str*/
0, /*tp_getattro*/
0, /*tp_setattro*/
0, /*tp_as_buffer*/
Py_TPFLAGS_DEFAULT, /*tp_flags*/
0, /*tp_doc */
0, /*tp_traverse*/
0, /*tp_clear*/
0, /*tp_richcompare*/
0, /*tp_weaklistoffset*/
0, /*tp_iter*/
0, /*tp_iternext*/
0, /*tp_methods*/
0, /*tp_members*/
0, /*tp_getset*/
0, /*tp_base*/
0, /*tp_dict*/
0, /*tp_descr_get*/
0, /*tp_descr_set*/
0, /*tp_dictoffset*/
(initproc)init, /*tp_init*/
0,
PyType_GenericNew,
};
static PyMethodDef methods[] = {
{0} /* sentinel */
};
PyMODINIT_FUNC initatomic(void) {
PyObject* m;
if (PyType_Ready(&AtomicCounterType) < 0)
return;
m = Py_InitModule("atomic", methods);
if (m == NULL)
return;
Py_INCREF(&AtomicCounterType);
PyModule_AddObject(m, "AtomicCounter", (PyObject
*)&AtomicCounterType);
}
========================
# actest.py
import threading
N = 200000
import atomic; count = atomic.AtomicCounter()
class timer(threading.Thread):
def run(self):
global count
for i in xrange(N):
count += 1
def test():
thread1=timer()
thread2=timer()
thread1.start()
thread2.start()
thread1.join()
thread2.join()
print 'final count:', count
print ' should be:', 2*N
if __name__=='__main__':
test()
========================
Carl Banks
--
http://mail.python.org/mailman/listinfo/python-list
-- http://mail.python.org/mailman/listinfo/python-list
