Anyone have any ideas? I feel like this could be a bug with the garbage collector across multiprocessing.
From: James DeVincentis [mailto:ad...@hexhost.net] Sent: Wednesday, October 14, 2015 12:41 PM To: 'python-list@python.org' Subject: Problem with copy.deepcopy and multiprocessing.Queue I've got a bit of a problem with copy.deepcopy and using multiprocessing.Queue. I have an HTTPAPI that gets exposed to add objects to a multiprocessing.Qeue. Source code here: https://github.com/jmdevince/cifpy3/blob/master/lib/cif/api/handler.py#L283 The trouble is, even using deepcopy, my debugging shows that it keeps re-using the same address space on every iteration of the loop (and/or fork since it uses the Forking TCP server mixin). This is reflected when the reference to that address space gets removed from the Queue. Source code: https://github.com/jmdevince/cifpy3/blob/master/lib/cif/worker/worker.py So I submit a few objects very rapidly to the HTTP server. It places them in the queue. For some reason on the second iteration copy.deepcopy stops using new address spaces for deepcopied objects. As such the queue processes that object only once (even though Queue.put() is called repeatedly for the same address space). I've tried 'del' the object before reusing it but it still reallocates to the same address space. I'm at a complete loss. 2015-10-14 17:03:30 APIHTTP WARNING Storing object: <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb2e8> 2015-10-14 17:03:30 THREAD #1-1 WARNING Got Observable with Object: <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbdb978> 2015-10-14 17:03:30 APIHTTP WARNING Storing object: <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0> 2015-10-14 17:03:30 THREAD #2-1 WARNING Got Observable with Object: <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbdab70> 2015-10-14 17:03:30 APIHTTP WARNING Storing object: <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0> 2015-10-14 17:03:30 APIHTTP WARNING Storing object: <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0> 2015-10-14 17:03:30 APIHTTP WARNING Storing object: <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0> 2015-10-14 17:03:30 APIHTTP WARNING Storing object: <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0> 2015-10-14 17:03:30 APIHTTP WARNING Storing object: <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0> 2015-10-14 17:03:30 APIHTTP WARNING Storing object: <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0> 2015-10-14 17:03:30 APIHTTP WARNING Storing object: <cif.types.observables.fqdn.Fqdn object at 0x7fb26dbcb0f0>
-- https://mail.python.org/mailman/listinfo/python-list
