[issue30919] Shared Array Memory Allocation Regression

Changes by Antoine Pitrou : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___

[issue30919] Shared Array Memory Allocation Regression

Antoine Pitrou added the comment: New changeset 3051f0b78e53d1b771b49375dc139ca13f9fd76e by Antoine Pitrou in branch 'master': bpo-30919: shared memory allocation performance regression in multiprocessing (#2708) https://github.com/python/cpython/commit/3051f0b78e53d1b771b49375dc139ca13f9fd76e

[issue30919] Shared Array Memory Allocation Regression

Antoine Pitrou added the comment: Serhiy, I've changed the strategy in PR 2708: now the partition free space is tested before deciding to locate the file there or not. -- ___ Python tracker ___

[issue30919] Shared Array Memory Allocation Regression

Antoine Pitrou added the comment: Again, setting TMPDIR is a poor workaround as it will impact any library creating temporary files. I haven't found anyone complaining about limited shared array size on 2.7, so likely it's not a common concern. Right now, anyone creating a huge shared array

[issue30919] Shared Array Memory Allocation Regression

Serhiy Storchaka added the comment: This can break Python 3 applications that work with shared arrays larger than the size of physical memory. If do this change, it should be optional. Setting TMPDIR=/dev/shm restores performance of 2.7 version even on versions that we will decide to not patch

[issue30919] Shared Array Memory Allocation Regression

Antoine Pitrou added the comment: The main issue here is to restore performance of 2.7 version. Setting TMPDIR sounds too general and might have impact on other libraries. I think it's ok if shared array size is limited by virtual memory size on the computer, since the whole point is to expo

[issue30919] Shared Array Memory Allocation Regression

Serhiy Storchaka added the comment: Yes, /dev/shm is ok on my computer. But there is yet one drawback of this patch. Currently the size of shared array is limited by the free space on the file system for temporary files. If it isn't enough you can easy set TMPDIR to other path. In 2.7 and with

[issue30919] Shared Array Memory Allocation Regression

Antoine Pitrou added the comment: > The size of file system mounted on "/run/user/${uid}" is limited (only 200 > MiB on my computer) Hmm, you're right. Is /dev/shm ok on your computer? -- ___ Python tracker

[issue30919] Shared Array Memory Allocation Regression

Serhiy Storchaka added the comment: The size of file system mounted on "/run/user/${uid}" is limited (only 200 MiB on my computer). get_temp_dir() will be successful, but this may be not enough for allocating large shared array. And it is harder to control by user, because the location is not

[issue30919] Shared Array Memory Allocation Regression

Antoine Pitrou added the comment: Gareth, Dimitar, you may want to take a look at https://github.com/python/cpython/pull/2708 -- stage: needs patch -> patch review ___ Python tracker __

[issue30919] Shared Array Memory Allocation Regression

Changes by Antoine Pitrou : -- pull_requests: +2774 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue30919] Shared Array Memory Allocation Regression

Antoine Pitrou added the comment: The original changeset in Richard's repository is https://hg.python.org/sandbox/sbt/rev/6c8554a7d068. Unless Richard answers otherwise, I think it's likely the performance degradation was an oversight. Given the code we're talking about is POSIX-specific, tru

[issue30919] Shared Array Memory Allocation Regression

Gareth Rees added the comment: I propose: 1. Ask Richard Oudkerk why in changeset 3b82e0d83bf9 the temporary file is zero-filled and not truncated. Perhaps there's some file system where this is necessary? (I tested HFS+ which doesn't support sparse files, and zero-filling seems not to be nec

[issue30919] Shared Array Memory Allocation Regression

Dimitar Tasev added the comment: Thank you, that is indeed the solution I ended up with, along with a large explanation of why it was necessary. Do you think that it's worth updating the `multiprocessing` documentation to make users aware of that behaviour? -- __

[issue30919] Shared Array Memory Allocation Regression

Gareth Rees added the comment: I see now that the default start method is 'fork' (except on Windows), so calling set_start_method is unnecessary. Note that you don't have to edit multiprocessing/heap.py, you can "monkey-patch" it in the program that needs the anonymous mapping: from multi

[issue30919] Shared Array Memory Allocation Regression

Dimitar Tasev added the comment: I have looked into your advice of changing multiprocessing.heap.Arena.__init__, I have removed the code that allocated the file and reverted to the old behaviour. I have done some brief runs and it seems to bring back the old behaviour which is allocating the

[issue30919] Shared Array Memory Allocation Regression

Gareth Rees added the comment: Nonetheless this is bound to be a nasty performance for many people doing big data processing with NumPy/SciPy/Pandas and multiprocessing and moving from 2 to 3, so even if it can't be fixed, the documentation ought to warn about the problem and explain how to wo

[issue30919] Shared Array Memory Allocation Regression

Gareth Rees added the comment: If you need the 2.7 behaviour (anonymous mappings) in 3.5 then you can still do it, with some effort. I think the approach that requires the smallest amount of work would be to ensure that subprocesses are started using fork(), by calling multiprocessing.set_star

[issue30919] Shared Array Memory Allocation Regression

Dimitar Tasev added the comment: If I understand correctly, there is no way to force the old behaviour in Python 3.5, i.e. to use an anonymous memory mapping in multiprocessing.heap.Arena so that the memory can be shared between the processes instead of writing to a shared file? The data size

[issue30919] Shared Array Memory Allocation Regression

Gareth Rees added the comment: Note that some filesystems (e.g. HFS+) don't support sparse files, so creating a large Arena will still be slow on these filesystems even if the file is created using ftruncate(). (This could be fixed, for the "fork" start method only, by using anonymous maps in

[issue30919] Shared Array Memory Allocation Regression

Gareth Rees added the comment: In Python 2.7, multiprocessing.heap.Arena uses an anonymous memory mapping on Unix. Anonymous memory mappings can be shared between processes but only via fork(). But Python 3 supports other ways of starting subprocesses (see issue 8713 [1]) and so an anonymous

[issue30919] Shared Array Memory Allocation Regression

Serhiy Storchaka added the comment: It can be significantly sped up if replace the writing loop with if size: os.lseek(self.fd, size, 0) os.write(self.fd, b'') os.lseek(self.fd, 0, 0) or just with os.truncate(self.fd, size) (not sure the latter always works).

[issue30919] Shared Array Memory Allocation Regression

Serhiy Storchaka added the comment: This is because instantiating new Arena is much slower in Python 3. ./python -m timeit -r1 -s 'from multiprocessing.heap import Arena' 'Arena(2**26)' Python 3.7: 1 loop, best of 1: 1.18 sec per loop Python 2.7: 10 loops, best of 1: 9.19 usec per loop Ma

[issue30919] Shared Array Memory Allocation Regression

New submission from Dimitar Tasev: Hello, I have noticed a significant performance regression when allocating a large shared array in Python 3.x versus Python 2.7. The affected module seems to be `multiprocessing`. The function I used for benchmarking: from timeit import timeit timeit('shared