I think the results so far point out the difficulty of trying to investigate this issue with dtrace. If the system has already reached a steady state of kernel memory utilization, tracking new kmem_alloc() calls isn't likely to reveal anything interesting, since those allocations are probably being freed rather quickly. If kernel memory utilization is growing over time, tracking new kmem_alloc() calls could be interesting, but only if you can also match up kmem_alloc()/kmem_free() pairs, to determine who is allocating memory but not releasing it. Without this matching, you will get a lot of noise, as you have already found. I think it will be tricky to get that right. If you are already at steady state, you will also need to reboot the system in order to track memory allocations since boot, and may also need to get your dtrace script running early in the boot process.
As johansen correctly pointed out, enabling kmem_audit will increase your memory utilization so you will need to verify that you have ample swap space allocated if you decide to go that route. You mentioned that you are running multiple zones, so you could also plan to check utilization as zones are brought up one at a time. If you still want to avoid the kmem_audit route, you will probably need to start looking at things from another angle. For example, reviewing your file systems, volume manager, network services, etc, and check to see if they provide hooks to check their memory utilization or if there are any known memory utilization issues. HTH, David ----- Original Message ----- From: johan...@sun.com Date: Tuesday, January 6, 2009 1:09 pm > > But I wonder can this be the problem? (claiming 55% of memory in > > kmem_alloc_xxx buffers, which totals at about 16GB! ) Didn't those > > "simple" exec (ps, perl, grep, prtdiag etc.) free the buffers/memory > > when they exit? > > Many of these allocations probably were freed. Remember, this script is > just showing you the processes that have allocated the most memory. > It's a starting point for drilling into the problem, if one exists. > > > What is more, as my understanding, the dtrace scripts we run only > > monitor those exec calling kmem_alloc at the time while the dtrace > > script is running, what about those exec already have kmem_alloc_xx > > buffers allocated and in use? > > Yes, that's correct. However, there's a tradeoff. Kmem_audit will keep > the last stack of the thread that alloc'd or free'd the buffer. > Unfortuantely, that consumes kernel memory. If you're worried about > your kernel memory being used up, using kmem_audit will consume more > memory. In my previous e-mail, I suggested kmem_audit as a last resort > when tracking down memory problems. Setting kmem_flags requires you to > reboot your system, wheras DTrace can be run on a production box at any > time. Using DTrace, you can see what threads are allocating and freeing > kernel memory, but only for the duration of the trace. Once the trace > is complete, the DTrace buffers are freed. It's up to you to decide > what approach is most appropriate. > > -j _______________________________________________ perf-discuss mailing list perf-discuss@opensolaris.org
