New submission from John Brearley <brear...@bell.net>:
While running a tensorflow script in the IDLEX GUI that runs for 8 million
steps and produce 2 lines stdout per step, my PC used all 16GB RAM and crashed
the python process, not to mention messed up other apps, like Firefox & Norton
AntiVirus. While the RAM was recovered, Firefox started responding, but Norton
Antivirus didn’t, so the PC had to be rebooted.
The issue is easily reproduced with the short print loop that dumps 20K lines
of stdout, at 171 characters / line on the IDLEX GUI window. When the script is
run in the IDLEX GUI, the Windows Task Manager shows the python process start
at 19MB RAM consumption, then grows to 569MB RAM consumption. If I run the
script a second time in the same IDLEX GUI window, it grows to 1.1GB RAM
consumption.
So 20K lines off output at 171 characters / line (“i: nnnnn” prefix + 2 * 80
byte string + newline) = 3.4M total characters stored in the scrollback buffer.
The delta memory consumed was 569MB – 19MB = 550MB. The RAM consumed /
character is 550MB / 3.4M = 161 bytes / character. This seems excessively
inefficient.
I now understand how the tensorflow script would stop after 550K iterations and
the 550K lines of stdout in the IDLEX GUI would consume all 16GB RAM on my PC.
BTW, when I run the same test script in the WinPython command prompt window, it
only consumes 4MB RAM while it runs. However the scrollback buffer is limited
to 10K lines, wrapped at the 80 character mark, so much less data saved.
I haven’t found any options in IDLEX GUI to limit the scrollback buffer size.
My request is to review the scrollback memory storage algorithms. If nothing
can be done to improve them, then please add a circular buffer to limit the
memory consumption.
# Print loop to test memory consumption in Python IDLEX GUI.
s1 = "0123456789"
s2 = s1+s1+s1+s1+s1+s1+s1+s1
for i in range(20000):
print("i:", i, s2, s2)
I am using Python 3.6.4 on Windows 7 PC, Intel i7-4770S, 3.1GHz, 16GB RAM.
----------
assignee: terry.reedy
components: IDLE
messages: 313263
nosy: jbrearley, terry.reedy
priority: normal
severity: normal
status: open
title: IDLEX GUI consumes all RAM for scrollback buffer, uses 161Bytes /
character stored
type: resource usage
versions: Python 3.6
_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue33000>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
