On 2015-10-14 18:41, James DeVincentis wrote:
I’ve got a bit of a problem with copy.deepcopy and using
multiprocessing.Queue.
I have an HTTPAPI that gets exposed to add objects to a
multiprocessing.Qeue. Source code here:
https://github.com/jmdevince/cifpy3/blob/master/lib/cif/api/handler.py#L283
The trouble is, even using deepcopy, my debugging shows that it keeps
re-using the same address space on every iteration of the loop (and/or
fork since it uses the Forking TCP server mixin).
This is reflected when the reference to that address space gets removed
from the Queue. Source code:
https://github.com/jmdevince/cifpy3/blob/master/lib/cif/worker/worker.py
So I submit a few objects very rapidly to the HTTP server. It places
them in the queue. For some reason on the second iteration copy.deepcopy
stops using new address spaces for deepcopied objects. As such the queue
processes that object only once (even though Queue.put() is called
repeatedly for the same address space).
I’ve tried ‘del’ the object before reusing it but it still reallocates
to the same address space. I’m at a complete loss.
2015-10-14 17:03:30 APIHTTP WARNING Storing object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb2e8>
2015-10-14 17:03:30 THREAD #1-1 WARNING Got Observable with Object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbdb978>
2015-10-14 17:03:30 APIHTTP WARNING Storing object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>
2015-10-14 17:03:30 THREAD #2-1 WARNING Got Observable with Object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbdab70>
2015-10-14 17:03:30 APIHTTP WARNING Storing object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>
2015-10-14 17:03:30 APIHTTP WARNING Storing object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>
2015-10-14 17:03:30 APIHTTP WARNING Storing object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>
2015-10-14 17:03:30 APIHTTP WARNING Storing object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>
2015-10-14 17:03:30 APIHTTP WARNING Storing object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>
2015-10-14 17:03:30 APIHTTP WARNING Storing object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>
2015-10-14 17:03:30 APIHTTP WARNING Storing object:
<cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>
Could it just be re-using space that it has recently freed?
--
https://mail.python.org/mailman/listinfo/python-list
