Re: out of memory problem

[Rob Dixon]

Arantxa Otegi wrote:

I have memory problems programming with perl: "out of memory!"

I have to process a lot of xml files which are in different directories (more than 20000 files in 110 directories). The files are quite small (almost all of them are smaller than 100KB).

Here is some code:

#### code #####################################################################################

use strict;
use Expansion;

die "Usage: doc_expansion.pl sarrera_dir irteera_dir hizkuntza_kodea %synset_expansioa" if ! $#ARGV == 3;

my $sarrera_dir = $ARGV[0];
my $irteera_dir = $ARGV[1];
my $hizk = $ARGV[2];
my $portzentaia = $ARGV[3];

my %variantak = Expansion::irakurri_variantak($hizk);

my @dirs;
my $dir;
my @fitxategiak;
my $fitx;
my $irt_fitx;
my @fitx_zenb;
my $fitx_zenb;

$| = 1;

opendir(DIR,$sarrera_dir)|| die "ezin da ireki DIR: $sarrera_dir";
@dirs = readdir(DIR);
closedir(DIR);

foreach $dir (@dirs){
next if $dir =~ /\.$/;

opendir(DIR,"$sarrera_dir/$dir/cache_en") || die "DIR(2): $sarrera_dir/$dir/cache_en" ;

if(not -e "$irteera_dir/$dir"){
  mkdir("$irteera_dir/$dir",0777)|| die "DIR(2): $irteera_dir/$dir";
}
if(not -e "$irteera_dir/$dir/cache_en"){

mkdir("$irteera_dir/$dir/cache_en",0777)|| die "DIR(2): $irteera_dir/$dir/cache_en";

}

@fitxategiak = grep {/\.wsd$/} readdir(DIR);
closedir(DIR);
foreach $fitx (@fitxategiak){
  print "$dir/cache_en/$fitx ....... ";

  @fitx_zenb = split(/\./,$fitx);
  $fitx_zenb = $fitx_zenb[0];
  $irt_fitx = "$irteera_dir/$dir/cache_en/$fitx_zenb.exp.$hizk";

Expansion::fitx_expansioa("$sarrera_dir/$dir/cache_en/$fitx",$irt_fitx,$portzentaia,%variantak);

  print "$irt_fitx--$fitx_zenb.exp.$hizk\n";
}
}

-------------------------------------------------------------------------------------------

package Expansion;
use strict;
use XML::DOM;

$| = 1;

sub fitx_expansioa{
my $sarfitx;
my $irteera_fitx;
my %variantak;
my $portzentaia;
($sarfitx,$irteera_fitx,$portzentaia,%variantak) = @_;


my $dom_parser = new XML::DOM::Parser;
my $dok = $dom_parser->parsefile($sarfitx);

//XML process
...........

my $irteeraXML = $dok->toString;
$dok->dispose;
open(I,">$irteera_fitx") or die "Ezin da ireki $irteera_fitx: $!";
print I $irteeraXML;
close I;
}

##### end code ##############################################################################

Can anybody help me to fix the problem?

Nothing stands out as being a problem, but it would be interesting to know if
taking out the part you mark 'XML process' makes a difference. Please try this
and, if the problem goes away, try putting the code back little by little until
it returns.

Rob

--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
http://learn.perl.org/