Am 21.06.2016 um 15:18 schrieb Dr. David Alan Gilbert:
* Peter Lieven (p...@kamp.de) wrote:
Hi,
while upgrading from Qemu 2.2.0 to Qemu 2.5.1.1 I noticed that the RSS memory
usage has heavily increased.
We use hugepages so the RSS memory does not include VM memory. In Qemu 2.2.0 it
used to be ~30MB per vServer
and increased to up to 300 - 400MB for Qemu 2.5.1.1 (same with master). The
memory increases over time, but seems
not to grow indefinetly. I tried to bisect, but had no result so far that made
sense. I also tried valgrind / massif, but
valgrind does not see the allocation (at least at exit) and massif fails to
rund due to - so it pretends - heap corruption.
Any help or ideas how to debug further would be appreciated.
I think I'd try stripping devices off; can you get a similar difference
to happen with a guest with no USB, no hugepages, no VGA and a simple
locally stored IDE disk?
Will do. VGA I already ruled out. Hugepages I can try, but its easier
to monitor the RSS size if the vServer memory is outside the RSS.
If you're having trouble bisecting is it possible it's a change
in one of the libraries it's linked against?
Same libraries. If I compile qemu-2.2.0 and qemu/master on the exactly
same machine I see the difference.
There was someone asking the other day on #qemu who had a setup that
was apparently using much more RAM than expected and we didn't
manage to track it down but I can't remember the version being used.
What I currently suspect are the following changes:
- We changed the coroutine pool to a per thread model. I disabled the pool.
This seems
to cut the max used RSS to about 150MB which is still a lot more than
qemu-2.2.0
- I suspect that something (e.g. the object based device tree) is creating a
lot of small allocations
which create a massive overhead. I managed to get valgrind/massif running
with attached debugger
and took snapshots of the running VM. What I see is that kernel RSS size is
much, much bigger than
what massif sees as allocated memory. I talk of massif sees 5MB heap usage
and the RSS size is 50MB or similar.
- Changing the memory allocator to tcmalloc or jemalloc seems to relax the
issue altough its not gone.
- VGA memory seems to have been moved from VM memory into Heap. But thats a
fixed 16MB allocation.
I will try to cut down devices as you proposed.
Peter
