[issue30919] Shared Array Memory Allocation Regression

2017-07-13 Thread Dimitar Tasev
New submission from Dimitar Tasev: Hello, I have noticed a significant performance regression when allocating a large shared array in Python 3.x versus Python 2.7. The affected module seems to be `multiprocessing`. The function I used for benchmarking: from timeit import timeit timeit

[issue30919] Shared Array Memory Allocation Regression

2017-07-14 Thread Dimitar Tasev
Dimitar Tasev added the comment: If I understand correctly, there is no way to force the old behaviour in Python 3.5, i.e. to use an anonymous memory mapping in multiprocessing.heap.Arena so that the memory can be shared between the processes instead of writing to a shared file? The data

[issue30919] Shared Array Memory Allocation Regression

2017-07-14 Thread Dimitar Tasev
Dimitar Tasev added the comment: I have looked into your advice of changing multiprocessing.heap.Arena.__init__, I have removed the code that allocated the file and reverted to the old behaviour. I have done some brief runs and it seems to bring back the old behaviour which is allocating the

[issue30919] Shared Array Memory Allocation Regression

2017-07-14 Thread Dimitar Tasev
Dimitar Tasev added the comment: Thank you, that is indeed the solution I ended up with, along with a large explanation of why it was necessary. Do you think that it's worth updating the `multiprocessing` documentation to make users aware of that beha