New submission from Windson Yang <wiwind...@gmail.com>:
I'm using macOX and I got huge memory usage when using generator with multiprocess. (see file) I think this is because (https://github.com/python/cpython/blob/master/Lib/multiprocessing/pool.py#L383) if not hasattr(iterable, '__len__'): iterable = list(iterable) if chunksize is None: chunksize, extra = divmod(len(iterable), len(self._pool) * 4) if extra: chunksize += 1 When we convert an iterable to list(iterable), we lost the advantage of using the generator. I'm not sure how to fix it, maybe we can set a default value for an object don't have '__len__' attr, any ideas? ---------- files: test.py messages: 321788 nosy: Windson Yang, zach.ware priority: normal severity: normal status: open title: multiprocessing memory huge usage type: resource usage versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8 Added file: https://bugs.python.org/file47698/test.py _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue34134> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
