[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

2018-08-14 Thread Carlo Rosati
New submission from Carlo Rosati : Hello, When I run the attached code, I encounter a segmentation fault. Thanks, Carlo -- files: 3.py messages: 323546 nosy: carlorosati priority: normal severity: normal status: open title: Segfault/TimeoutError: itertools.tee of

[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

2018-08-15 Thread Carlo Rosati
Carlo Rosati added the comment: I figured out that the problem is itertools.tee does not use a multiprocessing.Manager proxied object for shared state. I was able to create a workaround tee as follows. def multiprocessing_tee(iterable, n=2): """Write a multiprocessing saf

[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

2018-08-15 Thread Carlo Rosati
Carlo Rosati added the comment: Okay I needed to do .pop(0) instead of .pop(-1) which is probably O(N) -- ___ Python tracker <https://bugs.python.org/issue34

[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

2018-08-15 Thread Carlo Rosati
Carlo Rosati added the comment: You'll also need to lock when modifying the manager's list. Does anyone know how to do this using the multiprocessing.Queues without deadlocking? -- ___ Python tracker <https://bugs.python.o

[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

2018-08-16 Thread Carlo Rosati
Carlo Rosati added the comment: I've actually written a few workarounds that should be considered a multiprocessing specific tee function. I need feedback/critique on these. Hopefully we can all agree on one solution that's the best. It is unfortunate that the multiprocessing ma

[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

2018-08-16 Thread Carlo Rosati
Carlo Rosati added the comment: `for i in itertools.count()` in the first implementation I posted should be `while True`. I was using that for debugging. -- ___ Python tracker <https://bugs.python.org/issue34

[issue34410] Segfault/TimeoutError: itertools.tee of multiprocessing.pool.imap_unordered

2018-08-21 Thread Carlo Rosati
Carlo Rosati added the comment: If what you've said is correct, would it make the most sense to create a Manager method which returns a Proxy to a tee'd iterator? -- ___ Python tracker <https://bugs.python.o