Olivier Grisel added the comment: I have implemented a custom subclass of the multiprocessing Pool to be able plug custom pickling strategy for this specific use case in joblib:
https://github.com/joblib/joblib/blob/master/joblib/pool.py#L327 In particular it can: - detect mmap-backed numpy - transform large memory backed numpy arrays into numpy.memmap instances prior to pickling using the /dev/shm partition when available or TMPDIR otherwise. Here is some doc: https://github.com/joblib/joblib/blob/master/doc/parallel_numpy.rst I could submit the part that makes it possible to customize the picklers of multiprocessing.pool.Pool instance to the standard library if people are interested. The numpy specific stuff would stay in third party projects such as joblib but at least that would make it easier for people to plug their own optimizations without having to override half of the multiprocessing class hierarchy. ---------- nosy: +Olivier.Grisel _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue17560> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
