Filed an issue: https://bugs.python.org/issue29949
Thanks for your report, Jan. On Fri, Mar 31, 2017 at 3:04 AM, INADA Naoki <songofaca...@gmail.com> wrote: > Maybe, this commit make this regression. > > https://github.com/python/cpython/commit/4897300276d870f99459c82b937f0ac22450f0b6 > > Old: > minused = (so->used + other->used)*2 (L619) > > New: > minused = so->used + other->used (L620) > minused = (minused > 50000) ? minused * 2 : minused * 4; (L293) > > So size of small set is doubled. > > $ /usr/bin/python3 > Python 3.5.2+ (default, Sep 22 2016, 12:18:14) > [GCC 6.2.0 20160927] on linux > Type "help", "copyright", "credits" or "license" for more information. >>>> import sys >>>> s = set(range(10)) >>>> sys.getsizeof(frozenset(s)) > 736 >>>> > > $ python3 > Python 3.6.0 (default, Dec 30 2016, 20:49:54) > [GCC 6.2.0 20161005] on linux > Type "help", "copyright", "credits" or "license" for more information. >>>> import sys >>>> s = set(range(10)) >>>> sys.getsizeof(frozenset(s)) > 1248 >>>> > > > > On Fri, Mar 31, 2017 at 2:34 AM, INADA Naoki <songofaca...@gmail.com> wrote: >> I reproduced the issue. >> This is very usual, memory usage issue. Slashing is just a result of >> large memory usage. >> >> After 1st pass of optimization, RAM usage is 20GB+ on Python 3.5 and >> 30GB on Python 3.6. >> And Python 3.6 starts slashing in 2nd optimization pass. >> >> I enabled tracemalloc while 1st pass. Results is end of this mail. >> It seems frozenset() cause regression, but I'm not sure yet. >> I don't know what is contents of frozenset yet. (I know almost >> nothing about this application). >> >> Jan, do you know about what this is? >> Could you make script which just runs `transitive_closure(edges)` with >> edges similar to >> `log_reduction.py spaun`? >> >> I'll dig into it later, maybe next week. >> >> --- >> Python 3.6.1 >> 1191896 memory blocks: 22086104.2 KiB >> File >> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", >> line 85 >> reachables[vertex] = frozenset(reachables[vertex]) >> File >> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", >> line 410 >> self.dependents = transitive_closure(self.dg.forward) >> 602986 memory blocks: 51819.1 KiB >> File "<string>", line 14 >> File >> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", >> line 634 >> first_view=None, v_offset=0, v_size=0, v_base=None) >> >> Python 3.5.3 >> 1166804 memory blocks: 11116407.0 KiB >> File >> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", >> line 85 >> reachables[vertex] = frozenset(reachables[vertex]) >> File >> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", >> line 410 >> self.dependents = transitive_closure(self.dg.forward) >> 602989 memory blocks: 51819.3 KiB >> File "<string>", line 14 >> File >> "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py", >> line 634 >> first_view=None, v_offset=0, v_size=0, v_base=None) -- https://mail.python.org/mailman/listinfo/python-list
