I just enabled LeakSanitizer (LSan) for ASan Mochitest runs on inbound, in bug
988041. LeakSanitizer is a special mode of AddressSanitizer that detects when
allocations are not freed by shutdown, and reports the allocation stack for any
such leaked blocks. We already have good coverage in debug builds for most
leaks, so LSan is mostly useful for catching things that fall between the
cracks, like raw buffers, because it tracks every allocation made with malloc.
It has already detected a number of pretty bad leaks, plus many smaller ones.
The basics
----------
A leak detected by LSan will show up as a failure like this in the test harness:
TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at SomeFrame, SomeOtherFrame
This indicates that leak sanitizer has detected a leak, and the first two
frames that are not allocating functions are SomeFrame and SomeOtherFrame.
If you then go to the brief or full log, you can then look for a report
starting with "ERROR: LeakSanitizer: detected memory leaks". Then the report
will have entries like "Direct leak of 34 byte(s) in 1 object(s) allocated
from:", followed by a call stack for the allocation.
A leak is either "direct" or "indirect". An indirect leak is probably not the
root cause of the leak, so these are not displayed in TBPL.
Existing leaks
--------------
We have a number of existing leaks that show up in Mochitest runs. These fit in
three categories:
1. Known, unfixed leaks in Gecko code.
2. Intentional leaks by Gecko.
3. Memory not freed on shutdown by libraries.
To deal with these leaks, there is a suppression file, located in the tree
(build/sanitizers/lsan_suppressions.txt). Each entry is in the form leak:FOO.
If any stack frame contains FOO, then the entire leak will be suppressed. LSan
reports the suppressions used, and how many bytes of leaks are suppressed by
each. (For some reason, this does not always work locally, but it does work on
TBPL if you look at the full log.) The suppressions are very coarse-grained, so
the danger of adding any suppression is that it will start accidentally
matching actual leaks.
Please do not add new suppressions that are in the first category. These leaks
should be fixed before landing. Please _do_ help fix the existing leaks being
suppressed, in bugs blocking bug 976414, which will allow us to remove
suppressions, and catch more leaks in the future. See also the list at the end
of this message.
For the second category, some data structures in Gecko are intentionally leaked
through shutdown. AddressSanitizer builds are non-DEBUG, so DEBUG-only cleanup
routines will not be run, which can cause leaks. Other objects are just not
tracked by the DEBUG leak checkers, so we always leak them. The best way to
deal with these leaks is to run your cleanup code under MOZ_ASAN as well as
under DEBUG. If that is too difficult, a suppression can be added.
The third category of leaks involve system libraries, and look like this:
Direct leak of 32 byte(s) in 1 object(s) allocated from:
#0 0x470ad1 in __interceptor_malloc
/builds/slave/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:74
#1 0x7f82ff1d87ea (/usr/lib/x86_64-linux-gnu/libXrandr.so.2+0x17ea)
These leaks only have two frames. These could be the sign of real leaks, but
they are probably just internal intentional shutdown leaks in system libraries,
so feel free to just add a suppression for them, to the alphabetized list at
the end of the suppressions file. I'll probably file a bug soon to just ignore
these simple leaks all the time.
Fixing LSan-identified leaks
----------------------------
For the leaks identified by LSan that I've fixed so far, I've looked at the
location where the allocation is, and then looked at where the pointer ends up.
Usually there is an error case that wasn't handled properly soon after.
Bisecting the test suite down to a particular test also will help with the
investigation.
If you want to run LSan locally, you need an AddressSanitizer build, which is
really only supported on Linux right now. OSX support is in progress. The
address sanitizer article on MDN has information about how to do this, plus
information about LSan at the end:
https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Firefox_and_Address_Sanitizer
For local testing, you will need to comment out the libc.so suppression, and
whatever suppression matches the particular leak you are looking at.
To do an LSan run on try, just do an ASan push like normal, on any tree that
includes bug 988041. To run with LSan locally, just run the mochitest as normal
with an ASan build, and it should automatically do an LSan run.
Known unfixed leaks
-------------------
Thanks to everybody who has already fixed leaks detected by LSan!
Here are the remaining leaks that are being suppressed and need to be fixed, if
anybody wants to help out. If you fix them, be sure to remove the corresponding
entry in the suppression file.
Bug 981220 - Pixman fails to free TLS memory.
Bug 979928 - WebRTC has a variety of leaks that need to be fixed.
Bug 981195 - Small leak in the JS parser, involving
TypeCompartment::fixObjectType.
Bug 982111 - WebM is leaking under nestegg_read_packet.
Bug 987385 - Plugin leaks in the IPC layer.
Bug 987925 - NSS: Small leak under PK11_ChangePW.
Bug 1021302 - Memory allocated for fds by sockets is leaking.
Bug 1021350 - Small leak under event_base_once
Bug 1021854 - nsFileStreamBase::DoOpen() leaking memory for fds.
Bug 1023583 - Leak of array buffer data involving web audio and XHR.
Bug 1023585 - Leak of array buffer data in
JSStructuredCloneWriter::transferOwnership().
Bug 1022010 - Small leak under _render_glyph_outline.
Bug 1023548 - NSS: Small leak under SECITEM_AllocItem_Util.
Further information
-------------------
Feel free to talk to me if you encounter any problems with LeakSanitizer.
Here are some other resources that may be useful:
MDN article on LeakSanitizer and AddressSanitizer:
https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Firefox_and_Address_Sanitizer#LeakSanitizer
(I'll put more of the information in this message into that article or a new
one for LSan.)
The official LeakSanitizer page:
https://code.google.com/p/address-sanitizer/wiki/LeakSanitizer
Chromium's Leak Sanitizer page:
http://www.chromium.org/developers/testing/leaksanitizer
Chrome has been running LeakSanitizer for about a year:
https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/foex1srrEwU
Chrome's LSan suppression file:
http://src.chromium.org/viewvc/chrome/trunk/src/tools/lsan/suppressions.txt
--Andrew
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
