[issue33597] Compact PyGC_Head

STINNER Victor added the comment: > bpo-33597: Add What's New for PyGC_Head (GH-8236) Thank you! -- ___ Python tracker ___ ___ Pyt

[issue33597] Compact PyGC_Head

INADA Naoki added the comment: New changeset d5c875bbf1ef9b24349d3243b1ffaab6a9d5a78e by INADA Naoki in branch 'master': bpo-33597: Add What's New for PyGC_Head (GH-8236) https://github.com/python/cpython/commit/d5c875bbf1ef9b24349d3243b1ffaab6a9d5a78e -- __

[issue33597] Compact PyGC_Head

Change by INADA Naoki : -- pull_requests: +7772 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.

[issue33597] Compact PyGC_Head

STINNER Victor added the comment: Would you mind to mention your optimization in What's New in Python 3.8? IHMO any enhancement of the memory footprint should be documented :-) -- ___ Python tracker ___

[issue33597] Compact PyGC_Head

Change by INADA Naoki : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___ P

[issue33597] Compact PyGC_Head

INADA Naoki added the comment: New changeset 5ac9e6eee5ed18172d70d28cf438df0be4e3b83d by INADA Naoki in branch 'master': bpo-33597: Reduce PyGC_Head size (GH-7043) https://github.com/python/cpython/commit/5ac9e6eee5ed18172d70d28cf438df0be4e3b83d -- _

[issue33597] Compact PyGC_Head

STINNER Victor added the comment: > Do you think buildbots for master branch are sound enough to commit this > change? Or should I wait one more week? CIs on master are stable again. Since PyGC_Head is a key feature of Python, I would suggest you to wait at least a second approval of another

[issue33597] Compact PyGC_Head

INADA Naoki added the comment: @Victor Thanks for review. Do you think buildbots for master branch are sound enough to commit this change? Or should I wait one more week? http://buildbot.python.org/all/#/grid?branch=master -- ___ Python tracker

[issue33597] Compact PyGC_Head

STINNER Victor added the comment: It seems like the PR 7043 has no big impact on performances. Sometimes, it's a little bit faster, sometimes it's a little bit slower. The trend is a little bit more in the "faster" side, but it's not very obvious. I approved PR 7043. It shouldn't break the w

[issue33597] Compact PyGC_Head

INADA Naoki added the comment: You're right. I'll revert the optimization completely... -- ___ Python tracker ___ ___ Python-bugs-

[issue33597] Compact PyGC_Head

Serhiy Storchaka added the comment: There are also problems with func_name and func_qualname. func_name can be an instance of the str subtype and has __dict__. func_qualname seems can be of any type. Even if they would be exact strings, excluding them from tp_traverse will break functions t

[issue33597] Compact PyGC_Head

INADA Naoki added the comment: Oops, this optimization broke trace module. I reverted a part of the optimization. Current benchmark is: $ ./python-patched -m perf timeit --compare-to ./python-master -s "import gc, doctest, ftplib, asyncio, email, http.client, pydoc, pdb, fractions, decimal,

[issue33597] Compact PyGC_Head

INADA Naoki added the comment: https://github.com/python/cpython/pull/7043/commits/053111f321d792fb26f8be10abba24a980f3590f I added one micro optimization. Although it is not relating to two-word-gc directly, it makes gc_collect faster than master on the microbench (without importing tkinter.

[issue33597] Compact PyGC_Head

Antoine Pitrou added the comment: Here is a micro-benchmark of GC overhead: * before: $ ./python -m timeit -s "import gc, doctest, ftplib, asyncio, email, http.client, pydoc, pdb, fractions, decimal, difflib, textwrap, statistics, shutil, shelve, lzma, concurrent.futures, telnetlib, smtpd,

[issue33597] Compact PyGC_Head

STINNER Victor added the comment: "Hopefully, PyGC_Head seems to be excluded from PyGC_Head, and so it seems like the PR 7043 doesn't break the stable *ABI*." Oops, I mean: PyGC_Head seems to be excluded *from the Py_LIMITED_API*. -- ___ Python tr

[issue33597] Compact PyGC_Head

INADA Naoki added the comment: On Wed, May 30, 2018 at 7:14 PM STINNER Victor wrote: > STINNER Victor added the comment: > I asked if this change breaks the stable ABI. Steve Dower replied: > "Looks like it breaks the 3.7 ABI, which is certainly not allowed at this time. But it’s not a limi

[issue33597] Compact PyGC_Head

STINNER Victor added the comment: I asked if this change breaks the stable ABI. Steve Dower replied: "Looks like it breaks the 3.7 ABI, which is certainly not allowed at this time. But it’s not a limited API structure, so no problem for 3.8." https://mail.python.org/pipermail/python-dev/2018-

[issue33597] Compact PyGC_Head

INADA Naoki added the comment: > Is the PR already ready for review? I think so. -- ___ Python tracker ___ ___ Python-bugs-list ma

[issue33597] Compact PyGC_Head

Serhiy Storchaka added the comment: Even with smaller benefit the idea looks worth to me. Is the PR already ready for review? -- ___ Python tracker ___ __

[issue33597] Compact PyGC_Head

STINNER Victor added the comment: > As I said, the code just defers to "perf". And you should have tested that > :-) --track-memory and --tracemalloc have no unit tests, it's in the perf TODO list ;-) Well, it was just a remark. I'm looking for new contributors for perf! -- __

[issue33597] Compact PyGC_Head

Antoine Pitrou added the comment: As I said, the code just defers to "perf". And you should have tested that :-) I'm not really interested in checking it. All I know is that the very old code (inherited from Unladen Swallow) did memory tracking correctly. And I have no reason to disbelieve

[issue33597] Compact PyGC_Head

STINNER Victor added the comment: > Why wouldn't it? It certainly gives believable numbers (see above). > And it seems to defer to your own "perf" tool anyway. I wrote the code and I never seriously tested it, that's why I have doubts :-) Maybe it works, I just suggest to double check the co

[issue33597] Compact PyGC_Head

Antoine Pitrou added the comment: > I'm not sure that the code tracking the memory usage in performance works Why wouldn't it? It certainly gives believable numbers (see above). And it seems to defer to your own "perf" tool anyway. > perf has two options: --track-memory and --tracemalloc, s

[issue33597] Compact PyGC_Head

STINNER Victor added the comment: > Ok, I ran a subset of the benchmarks to record their memory footprint and got > these results: I'm not sure that the code tracking the memory usage in performance works :-) It may be worth to double check the code. By the way, perf has a --tracemalloc opt

[issue33597] Compact PyGC_Head

Antoine Pitrou added the comment: Ok, I ran a subset of the benchmarks to record their memory footprint and got these results: master-mem.perf === Performance version: 0.6.2 Python version: 3.8.0a0 (64-bit) revision 73cbe7a Report on Linux-4.15.0-22-generic-x86_64-with-glibc2.9 N

[issue33597] Compact PyGC_Head

Antoine Pitrou added the comment: > Are you sure that all memory allocators align at least on 8 bytes (to give up > 3 unused bits)? If they don't then a simple double array will end up unaligned. It's not impossible but extremely unlikely. -- __

[issue33597] Compact PyGC_Head

STINNER Victor added the comment: Are you sure that all memory allocators align at least on 8 bytes (to give up 3 unused bits)? I don't know the answer, it's an open question. -- nosy: +vstinner ___ Python tracker

[issue33597] Compact PyGC_Head

Change by Eitan Adler : -- nosy: +eitan.adler ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.py

[issue33597] Compact PyGC_Head

Yury Selivanov added the comment: This is such a great idea. +1 from me. -- nosy: +yselivanov ___ Python tracker ___ ___ Python-bugs

[issue33597] Compact PyGC_Head

INADA Naoki added the comment: In Doc folder: make clean make PYTHON=../python venv /usr/bin/time make html master: 113.15user 0.41system 1:55.46elapsed 98%CPU (0avgtext+0avgdata 205472maxresident)k 18800inputs+223544outputs (1major+66066minor)pagefaults 0swaps 111.07user 0.44sy

[issue33597] Compact PyGC_Head

INADA Naoki added the comment: @Serhiy php implemented similar idea recently. https://react-etc.net/entry/improvements-to-garbage-collection-gc-php-7-3-boosts-performance-in-benchmark In short, each tracked object have only "index" of GC struct, not "pointer". GC struct is in array and it can

[issue33597] Compact PyGC_Head

Antoine Pitrou added the comment: Le 22/05/2018 à 17:31, INADA Naoki a écrit : > > INADA Naoki added the comment: > > $ ./python-gc -c 'import asyncio,sys; sys._debugmallocstats()' Thanks. You can also collect peak memory stats during the benchmark suite, though the numbers may be approxima

[issue33597] Compact PyGC_Head

Serhiy Storchaka added the comment: This is an interesting idea. The other problem with the garbage collecting is that it modifies the memory of all collectable objects. This leads to deduplicating virtually all memory blocks after the fork, even if these objects are not used in the child. I

[issue33597] Compact PyGC_Head

INADA Naoki added the comment: $ ./python-gc -c 'import asyncio,sys; sys._debugmallocstats()' master: # bytes in allocated blocks=4,011,368 # bytes in available blocks= 136,640 50 unused pools * 4096 bytes = 204,800 # bytes lost to po

[issue33597] Compact PyGC_Head

Antoine Pitrou added the comment: Interesting. Do you have any comparisons on memory footprint too? -- ___ Python tracker ___ ___ P

[issue33597] Compact PyGC_Head

INADA Naoki added the comment: $ ./python -m perf compare_to master.json twogc.json -G --min-speed=2 Slower (3): - scimark_monte_carlo: 268 ms +- 9 ms -> 278 ms +- 8 ms: 1.04x slower (+4%) - fannkuch: 1.03 sec +- 0.02 sec -> 1.06 sec +- 0.02 sec: 1.03x slower (+3%) - spectral_norm: 285 ms +- 9

[issue33597] Compact PyGC_Head

Change by Serhiy Storchaka : -- nosy: +serhiy.storchaka ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https:

[issue33597] Compact PyGC_Head

Change by INADA Naoki : -- keywords: +patch pull_requests: +6682 stage: -> patch review ___ Python tracker ___ ___ Python-bugs-list

[issue33597] Compact PyGC_Head

New submission from INADA Naoki : Currently, PyGC_Head takes three words; gc_prev, gc_next, and gc_refcnt. gc_refcnt is used when collecting, for trial deletion. gc_prev is used for tracking and untracking. So if we can avoid tracking/untracking while trial deletion, gc_prev and gc_refcnt can