New submission from James Westby:
Hi,
I'm looking in to the memory usage of our testsuite, which does a fair amount of
def setUp():
patcher = patch.multiple(...)
self.mock_whatever = patcher.start()
self.addCleanup(patcher.stop)
or other ways of creating a mock and assigning it to an instance variable on a
TestCase.
This means that by the end of a run, we have quite a lot of references to
MagicMocks.
This then becomes the majority of the memory usage of the python process: (from
meliae)
Total 1157176 objects, 1189 types, Total size = 327.1MiB (342972340 bytes)
Index Count % Size % Cum Max Kind
0 575750 49 198058000 57 57 344 MagicProxy
1 49955 4 52034888 15 72 196888 dict
2 124127 10 11881628 3 76 386477 str
3 12997 1 11749288 3 79 904 type
4 8225 0 9146200 2 82 1112 MagicMock
5 66310 5 5282392 1 84 80056 tuple
6 38161 3 4579320 1 85 120 function
7 1503 0 3972281 1 86 49488 module
8 28506 2 3648768 1 87 128 code
9 25768 2 2869680 0 88 69168 list
10 12649 1 2803196 0 89 66356 unicode
11 2251 0 2503112 0 89 1112 ClientHandler
12 2228 0 2477536 0 90 1112 _patch
13 28223 2 2257840 0 91 80 instancemethod
14 2014 0 2239568 0 91 1112 BoundMethodWeakref
15 2003 0 2227336 0 92 1112 DummyCache
16 24681 2 2221112 0 93 792 _CallList
17 18555 1 1632840 0 93 88 weakref
18 1457 0 1550248 0 94 1064 Morsel
19 46258 3 1110192 0 94 24 int
The fact that each MagicMock creates 72 MagicProxies means that it is a
significant chunk of memory, despite each being small.
John Arbash Meinel suggested setting __slots__ = ['name', 'parent'] on
MagicProxy to reduce the memory usage of this simple object.
This helps with the memory usage:
Total 1140377 objects, 1189 types, Total size = 169.5MiB (177755867 bytes)
Index Count % Size % Cum Max Kind
0 47744 4 51347840 28 28 196888 dict
1 574210 50 36749440 20 49 64 MagicProxy
2 122020 10 11769659 6 56 386477 str
3 12975 1 11729400 6 62 904 type
4 8203 0 9121736 5 67 1112 MagicMock
5 64125 5 5141368 2 70 80056 tuple
6 36024 3 4322880 2 73 120 function
7 1503 0 3972281 2 75 49488 module
8 28506 2 3648768 2 77 128 code
9 12643 1 2801540 1 79 66356 unicode
10 23634 2 2716064 1 80 69168 list
11 2251 0 2503112 1 82 1112 ClientHandler
12 28223 2 2257840 1 83 80 instancemethod
13 2014 0 2239568 1 84 1112 BoundMethodWeakref
14 2003 0 2227336 1 85 1112 DummyCache
15 24615 2 2214536 1 87 792 _CallList
16 18482 1 1626416 0 87 88 weakref
17 1457 0 1550248 0 88 1064 Morsel
18 46259 4 1110216 0 89 24 int
19 2496 0 858624 0 89 344 ModelState
I'm going to go through and drop references so that these can get garbage
collected, but making Mock less memory-intensive would probably be appreciated
by its users.
Reducing the memory usage of the tiny MagicProxies would be good, but also if
there is a way to reduce the number of them by not pre-assiging 72 of them for
every MagicMock, when each is very unlikely to be used, then that would be good
as well.
Thanks,
James
----------
components: Library (Lib)
messages: 226017
nosy: james-w
priority: normal
severity: normal
status: open
title: mock could be less memory-intensive
type: resource usage
versions: Python 2.7
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue22293>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
