I think I tracked this down and resolved it. It appears taking an object from a multiprocess.Queue and placing it into a queue.Queue is a no-no even if the queue.Queue isn’t shared across processes.
I have a series of workers (multiprocessing) that share a multiprocess.Queue across all processes. Each worker has it’s own local queue (Queue.queue) that is shared within only it’s own threads. It appears that when taking an object from a multiprocessing queue and moving it into a Queue.queue it breaks things. I don’t know what or how, but it does. As soon as I switched the local queue to a multiprocess.Queue it fixed the issue. Kind of odd to me. Not sure if anyone wants to look into it. > On Oct 15, 2015, at 5:42 PM, Ian Kelly <ian.g.ke...@gmail.com> wrote: > > On Thu, Oct 15, 2015 at 4:02 PM, James DeVincentis <ad...@hexhost.net> wrote: > > > > Anyone have any ideas? I feel like this could be a bug with the garbage > > collector across multiprocessing. > > I'll second MRAB's response from yesterday: could it just be reusing space > that it has recently freed? > > As a debugging measure, what happens if you try to keep the objects in > memory, e.g. by adding them to some long-lived list? Do you start seeing > different addresses? -- https://mail.python.org/mailman/listinfo/python-list
