* On 2004.09.15, in <[EMAIL PROTECTED]>,
* "D Walsh" <[EMAIL PROTECTED]> wrote:
>
> I sat down in front of a Solaris 9 system, installed clamav as
> instructed and yes indeed there appears to be a problem with the
> implementation of free(), in 30 mins of sending e-mail from the EICAR
> test site memory did climb to 2.87gb and did not clear itself.
This does not by itself indicate a problem in Solaris's free().
There are two things to learn here. First, what your profile indicates
is that clamd has allocated a lot of memory, and the arrangement of
those segments is such that it cannot all be returned to the heap yet.
This C program illustrates, somewhat.
#include <stdlib.h>
#include <unistd.h>
#define PS_ARGS "-eopid,vsz,rss,comm" /* for solaris */
/* #define PS_ARGS "auwx"*/ /* for others */
main()
{
void *big, *small; /* two references to memory */
char cmd[256]; /* a command string */
sprintf(cmd, "ps " PS_ARGS " | grep %d | grep -v grep", getpid());
system(cmd); /* show basic usage */
big = malloc(1024*1024*1024); /* allocate a big lump */
system(cmd); /* show usage */
small = malloc(1024*1024); /* allocate a small lump */
system(cmd); /* usage is larger */
free(big); /* free big first */
system(cmd); /* usage *might* not decrease */
free(small); /* free small */
system(cmd); /* usage *might* decrease */
}
Unix memory management works in a particular way. All allocated memory
is taken from a free pool in the process's address space called the
heap. There's a pointer called the break (brk) that indicates what
belongs to the process, and what belongs to the system. Anything on one
side is the process's, and anything on the other side is the system's.
If you allocate memory in segments, the break moves to make space with
each allocation. (For small allocations, the malloc manager usually
tries to anticipate larger needs; it doesn't take only small chunks
from the system.) Free()ing memory just means telling malloc that it's
available for reuse. It doesn't necessarily mean it goes back to the
system -- the break needs to be reset for that to happen. So the malloc
manager needs enough smarts to rearrange memory so that the break can be
moved without releasing memory that's in use. In the demo code above,
freeing "big" doesn't necessarily mean that the 1 GB can be returned to
the heap, because "small" is in its way. The malloc manager needs to
relocate intervening segments -- i.e., "small" -- to make that work.
Relocating means doing a lot of memory copying, so the more complex your
segment list -- the more small malloc()s you've done -- the harder this
problem gets.
Most modern mallocs can probably handle the case above -- it's not too
hard, since there's plenty of space in the memory occupied by big for
small to relocate to, easily. But if you swap the free(big) and the
free(small), this gets harder. Not all mallocs can do that, and when
you look at how much memory copying is required for this, it's not
necessarily clear that you want to, unless you really need to to satify
a demand elsewhere in the system. So malloc managers get even smarter,
trying to predict needs and such. The effect is that you don't *always*
get back what you free() when you free() it, but you might get it back
later... and you might not.
Debugging malloc()s can help, if this bothers you. They are designed
to do a lot of relocation work so that all memory free()d is returned
to the heap, so that segmentation faults are triggered whenever free
memory is used, and you can use the debugging data from that to track
the problem. But on the flip side, they typically perform terribly, so
they're not generally suitable for production work.
...
The second thing to realize is that Solaris is actually being very
smart, not dumb. Its memory management system is quite sophisticated. (I
don't mean its malloc() here, so much as the kernel memory management
system -- although the malloc() is surely designed to be cooperative
with that.) Memory that is no longer needed by a process does not
necessarily get returned to the system's free list. Instead it's used
for a variety of caching functions, memory-mapping of filesystems
and swap, a bunch of similar things. If you write a small program to
allocate 100 MB, read a 100 MB file into that memory, free it, and then
repeat, you could well find that the disk is barely touched for the
second round. Sun have white papers on this stuff of you want to read
more about it -- I couldn't do it justice here, but it's really quite
nice.
The dark side of it all is that you can't at all trust "ps" on a Solaris
system, especially on Solaris 9 and up, to tell you how much memory a
process is truly using. But the bright side is that in most cases you
don't really need to, unless you can tell from other indicators that
your system is really stressed for memory. And then you need some more
advanced tools for diagnisis of your system issues, but it's not going
to help you find leaks in clamd. I'll try to dig up references if you
want them badly.
> This leads me to believe that the problem is occurring in Solaris 9
> due to changes in the OS itself and perhaps clamav should consider
> using a different avenue with this OS.
Solaris 9 is much more advanced than 8 in the memory management area,
particularly with respect to the things I mentioned above. Solaris 10
even moreso.
> Aside from that, all I can tell you is I'm not impressed with this OS
> and I guess I prefer the Darwin/FreeBSD environment because I'm used to
> it, know what tools are available and can muck around with some
> confidence without blowing things up.
Being used to it is really what it comes down to. If you study it more,
you'll find more reasons to be impressed.
All this is just to throw in with that camp that says these are no
indications of memory leaks. Now back to your regularly-scheduled virus
discussion.
--
-D. [EMAIL PROTECTED] NSIT::ENSS
-------------------------------------------------------
This SF.Net email is sponsored by: thawte's Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Clamav-users mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/clamav-users
