On Wed, Oct 13, 2010 at 10:41 AM, Panda-X <exilepa...@gmail.com> wrote:
>
>
> 2010/10/13 Rob Coops <rco...@gmail.com>
>
>
>>
>> On Wed, Oct 13, 2010 at 8:42 AM, Panda-X <exilepa...@gmail.com> wrote:
>>
>>> Hi List,
>>>
>>> My script is running on WinXP, Dos shell, which is a server program.
>>>
>>> It works fine, unless it will pops-up a "Out of Memory!" and stop running
>>> per each few days!
>>>
>>> Could someone tell me this error message is coming from Perl ? or from
>>> the
>>> Dos shell ?
>>>
>>> Thanks!
>>> ex
>>>
>>
>> Good question, most likely a program on your system is eating memory and
>> not letting go. Over time you will run out of memory and you will get this
>> message. The problem might be your script or well, anything else on your
>> system that is running at the same time.
>> Even though I do not doubt your programming skills I would suggest a
>> simple test. Don't run your script for a few days and see if you get the
>> same error. If so you are sure that it is not caused by your script, if you
>> don't get the error then your script is the problem and you have some fun
>> debugging to do.
>>
>> Normally perl is quite good in the cleaning up of unused resources memory
>> or otherwise, but as a developer you can make it very hard for perl to keep
>> the memory clean. You might be building an array, a hash or some other data
>> element that never stops growing, or that you copy data from A to B to C and
>> never really release any of them etc. Specially scripts that are supposed to
>> run constantly will need you as a developer to pay special attention to
>> keeping the memory foot print within reason...
>>
>> It might be a good idea to post the script to this list and ask people to
>> help you out, often a mistake like this is not hard to spot and usually
>> takes a few minor changes to help perl keep the memory footprint nice and
>> small.
>>
>> I'm sure that there are debug options in perl that can help you with this
>> as well I have never ended up using them as the few scripts that I did write
>> that had to live forever didn't cause me any memory problems (a few other
>> non-memory related once though) so I have no idea how to debug stuff like
>> that with perl.
>>
>> Regards,
>>
>> Rob
>>
>
>
> Thank you very much!
>
> The server script I am using is as a chat server, It carries message from
> one user to another.
> When the target user got their messages, the [array] will be deleted from
> the HoH. ( see below )
>
> I've turned on strict and warnings, as well $| = 1; everything besides
> the struct below, and the daemon object, everything is my $localvars.
>
> Overall to say, I have a heavy struct like this :
>
> $MESG = {
> usr1 => {
> [$From, $to, $Time, $Context],
> [$From, $to, $Time, $Context],
> ...
> },
> usr2 => {
> [....],
>
>
> [....],
> ...
> }
> }
>
> In the script, I am using modules :
> IO::Socket::INET::Daemon;
> Devel::Size qw(total_size);
>
> Then, I have this sub to check the struct ( an relative size )
>
> sub DumpSize {
> my $total_sz = 0 ;
> foreach my $usr ( sort keys %$MESG ) {
> my $size = 0 ;
> print "$usr: MESG=" ;
> print scalar ( @{$MESG->{$usr}} ) ;
> print " , LEN=";
> foreach ( @{$MESG->{$usr}} ) {
> $size += length $_ ;
> $total_sz += length $_ ;
> } print "$size , TTZ=$total_sz , MEM=";
> print total_size $MESG;
> print " TRF=";
> print $COUNT;
> print $/;
> }
>
> print "$/==== END $_[0] ";
> print TimeStamp ( time ) ;
> print " ======================$/$/";
> }
>
> This sub will turns out something like this :
>
> ==== END GET 2010/10/13 16:23:09 ======================
>
> usr1: MESG=2 , LEN=535 , TTZ=535 , MEM=1853 TRF=7223
> usr2: MESG=1 , LEN=175 , TTZ=710 , MEM=1853 TRF=7223
> usr3: MESG=2 , LEN=339 , TTZ=1049 , MEM=1853 TRF=7223
>
> ==== END GET 2010/10/13 16:23:27 ======================
>
> usr2: MESG=2 , LEN=535 , TTZ=535 , MEM=1275 TRF=7224
> usr3: MESG=1 , LEN=175 , TTZ=710 , MEM=1275 TRF=7224
>
> ==== END GET 2010/10/13 16:24:43 ======================
>
> usr2: MESG=1 , LEN=171 , TTZ=171 , MEM=2200 TRF=7227
> usr3: MESG=2 , LEN=535 , TTZ=706 , MEM=2200 TRF=7227
> usr5: MESG=2 , LEN=346 , TTZ=1052 , MEM=2200 TRF=7227
> usr6: MESG=1 , LEN=171 , TTZ=1223 , MEM=2200 TRF=7227
>
> This sub will be called every time a user Send or Get a mesg.
> MESG is how much messages in the inbox of the user
> LEN is the Totel text length for the user,
> TTZ is the total text length in server,
> MEM is the $MESG struct size
> TRF is how many rounds of Get / Send happened.
>
> From my observation, the Out of Memory error could happen anytime, where
> the "MEM" size was never over 10K.
>
> Any clues ?
>
>
So let me get this right have one structure $MESG that stores all messages
for all users for as long as they have not fetched the messages... so if a
user sends 1 message your structure will contain one line with usr1 => { [ ]
} and if this same user sends another message $message will contain usr1 =>
{ [ ], [ ] } and so on.
I assume that once a message is retrieved the message is removed from the
$MESG structure, if not that is an very obvious problem, which I am sure you
already thought about as you say your memory consumption of $MESG never went
over 10k which should not cause a problem at all. Still I would push as much
of the $MESG structure out to the file system (say after a user has not
picked up messages directed at him/her for more then 10 minutes as it is
likely that this user will not pick up the messages for another couple of
hours and if they do reading the messages back from file is not so horrible
unless we are talking about tens of thousands of users at a time...
Besides that it looks quite solid to me, and form the sound of it you have a
few clues about what you are doing with perl ;-) When you fire up the
windows task manager does your perl program also stay under the 10k or might
it be that the footprint of the interpreter is growing while the main
internal structure is not?
It could be that one of the other variables manages to hang on to life even
though they are all local, or and this is something I did do in the past
that your socket connections are not closed properly this would cause your
system to allocate a buffer etc for your socket every time you open one and
if you open a new one but don't correctly close the previous one this could
lead to an out of memory error as well.
My problem at the time turned out to be that on the Solaris version I was
working on the sockets only got closed after a long long wait period, this
was due to a configuration error on the Solaris build and the fact that it
expected a certain terminator from the client side to indicate it was done
talking.
Out of curiosity I had a look around for some interesting article about
profiling in perl and I cam across this one:
http://www.perl.com/pub/2004/06/25/profiling.html (which actually points to
Devel::NTYProf which is great but there is not much in terms of memory
profiling there)
Another interesting read I found that actually poses the exact same question
you have (though on the Linux platform)
http://stackoverflow.com/questions/1359771/perl-memory-usage-profiling-and-leak-detection
has
several suggestions where things might be going wrong, among which an
explanation how to test for memory leaks.
In the end finding a leak is just as hard in perl as it is in Java, C or
your bike tire it is just a mater of testing every bit of the thing till you
get lucky and see the air bubbles escaping.
Regards,
Rob
