The problem isn’t with multiprocess.Queue. The problem is that
socketserver.ForkingMixIn calls os._exit() after the request is handled by the
forked process. Which I did not know until I dug into the internals of socket
server.ForkingMixIn.
This causes the Finalize() hooks inside of multiproces
On Sun, Oct 18, 2015 at 2:46 AM, James DeVincentis
wrote:
>
> I see, looks like I’ll have to use Queue.close()
>
> Didn’t think it would be necessary since I was assuming it would be
garbage collected. Sigh. Bug, fixed.
I'm not really following what the issue is here -- it sounds like it runs
pre
>> 2015-10-16 00:25:05 worker #3DEBUGLocal Queue Size: 0
>> 2015-10-16 00:25:05 worker #4DEBUGLocal Queue Size: 0
>> 2015-10-16 00:25:05 worker #2 DEBUGLocal Queue Size: 0
>> 2015-10-16 00:25:05 MAIN DEBUGGlobal Queue Size: 572
>>
>
queue
>>>> 2015-10-16 00:23:08 Manager #2 ERRORPut
>>>> into local queue
>>>> 2015-10-16 00:23:08 Manager #2 ERROR Waiting for item from global
>>>> queue
>>>> — Nothing else gets logged by the manager threads at all, even i
gt;>> 2015-10-16 00:25:00 worker #1DEBUGLocal Queue Size: 0
>>> 2015-10-16 00:25:00 worker #3DEBUGLocal Queue Size: 0
>>> 2015-10-16 00:25:00 worker #4DEBUGLocal Queue Size: 0
>>> 2015-10-16 00:25:00 worker #2DEBUG Local Queue Size: 0
-16 00:25:05 worker #1DEBUGLocal Queue Size: 0
>> 2015-10-16 00:25:05 worker #3DEBUGLocal Queue Size: 0
>> 2015-10-16 00:25:05 worker #4DEBUGLocal Queue Size: 0
>> 2015-10-16 00:25:05 worker #2DEBUGLocal Queue Size: 0
>> 2015-10-16 00:25:05 MAIN
deas? I feel like this could be a bug with the garbage
>> collector across multiprocessing.
>>
>> From: James DeVincentis [mailto:ad...@hexhost.net]
>> Sent: Wednesday, October 14, 2015 12:41 PM
>> To: 'python-list@python.org'
>> Subject: Problem with copy
2:41 PM
> To: 'python-list@python.org'
> Subject: Problem with copy.deepcopy and multiprocessing.Queue
>
> I’ve got a bit of a problem with copy.deepcopy and using
> multiprocessing.Queue.
>
> I have an HTTPAPI that gets exposed to add objects to a multiprocess
I take that back. It’s not entirely fixed.
Something else strange is going on here. More debugging needed.
> On Oct 15, 2015, at 6:36 PM, James DeVincentis wrote:
>
> I think I tracked this down and resolved it.
>
> It appears taking an object from a multiprocess.Queue and placing it into
I think I tracked this down and resolved it.
It appears taking an object from a multiprocess.Queue and placing it into a
queue.Queue is a no-no even if the queue.Queue isn’t shared across processes.
I have a series of workers (multiprocessing) that share a multiprocess.Queue
across all process
On Thu, Oct 15, 2015 at 4:02 PM, James DeVincentis
wrote:
>
> Anyone have any ideas? I feel like this could be a bug with the garbage
collector across multiprocessing.
I'll second MRAB's response from yesterday: could it just be reusing space
that it has recently freed?
As a debugging measure, w
Anyone have any ideas? I feel like this could be a bug with the garbage
collector across multiprocessing.
From: James DeVincentis [mailto:ad...@hexhost.net]
Sent: Wednesday, October 14, 2015 12:41 PM
To: 'python-list@python.org'
Subject: Problem with copy.deepcopy and multiproces
On 2015-10-14 18:41, James DeVincentis wrote:
I’ve got a bit of a problem with copy.deepcopy and using
multiprocessing.Queue.
I have an HTTPAPI that gets exposed to add objects to a
multiprocessing.Qeue. Source code here:
https://github.com/jmdevince/cifpy3/blob/master/lib/cif/api/handler.py#L28
I've got a bit of a problem with copy.deepcopy and using
multiprocessing.Queue.
I have an HTTPAPI that gets exposed to add objects to a
multiprocessing.Qeue. Source code here:
https://github.com/jmdevince/cifpy3/blob/master/lib/cif/api/handler.py#L283
The trouble is, even using deepcopy, m
14 matches
Mail list logo
