Hi all,
Please note the Perl script that am trying to write at end the of this email,
after the dashes.
I have two files that contains thousands and millions of records of two database
tables that are supposed to be replicated and same "similar" amount of records
but, unfortunately, they're not.
This is what's happening:
TARGET1.LOG:
2
4
6
SOURCE1.LOG:
1
2
3
4
5
6
I place all of TARGET1.LOG into an array, then I read SOURCE1.LOG one line at a
time and do a grep against the array. If there is no match, then that means the
record does not exist and is thus not replicated and I will want to print this
into a file or maybe store it into an unmatch array temporarily and write the
whole array into a file later on.
The script does what it intends to do for small-sized file. But when I get a
file with 8million+ records that is 600MB in size. It consumes a lot of the CPU
system resource. Stripping the spaces manually via sed cuts the size down to
60MB, but it still consumes a lot of resource. So now I want to know how can I
convert this script to grep the string from a file instead of from the array.
Doing so I think will lessen the use of most of the CPU resource.
Am guessing that I have to take out the array bit and just modify this line
here:
@match=grep{/$lookfor/[EMAIL PROTECTED];
But I am not sure how or what I should change it to. How do I tell grep to
search from a file instead of from an array? Do I need to open a FILEHANDLE for
grep to search from? Also, I want to do the sed stripping of the spaces from
within Perl instead of what am currently doing which is running the sed
manually first as a separate step and then running my Perl script.
Any help will be very much appreciated. Thanks in advance.
-----------------------------------------------------------------------------------
#!/usr/bin/perl
my $data_file = 'target1.log';
open DATA, "$data_file" or die "can't open $data_file $!";
my @array_of_data = <DATA>;
close (DATA);
print "The array MAX index is ==> $#array_of_data ...\n";
open (DATA, "< source1.log");
while( <DATA> )
{
#print $_;
chomp $_;
$lookfor=$_;
@match=grep{/$lookfor/[EMAIL PROTECTED];
if (@match) {
print "MATCH :: << $lookfor >> \n";
}
else
{
print "NO-MATCH :: << $lookfor >> \n";
}
}
close (DATA);
exit;
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
<http://learn.perl.org/> <http://learn.perl.org/first-response>
