Ok... so ..... I'm a bit nuts.
I added a fifth and sixth variation in the series "how best to replace one string with another in a text file" -- which seems to be a great example of the many ways to code even simple problems in Perl.
For this round of testing, I quadrupled the size of the test text file to 64,508 lines / 11.9MB to better show runtime differences between the variations. Originally, just to be able to demonstrate that each test script produced the exact same output, I made each script write out a different file. All read the same input file, "testin.txt", and then each writes a different (i.e., "testoutN.txt" where N = 1 .. 6). I found that this change produced significant run time variations in most of the scripts -- which I believe is related to Windows file system buffering / caching. I decided to leave it in as it's a better reflection of "real world" behavior. But leaving it in does require analyzing the results statistically rather than as single runtime measures. So I ended up running each variation 51 times (with nothing else running on my PC) and then summarizing the results using a few simple statistical measures in Excel.
The 5th variation was suggested by Chris Wagner (wagnerc) "geared far more toward performance/coolness than simplicity. ;)". [Read on to see if he was right.] It is the only variation that does not use Perl's built in substitution operator. Instead, it uses index to find the target string. In my original implementation, the script updated the $text variable (slurped from the input file) every time index found a match (using substr as a lvalue) -- equivalent to the substitution operator. I also added $Bill Luebkert's suggestion for slurping a file into a scalar string variable by locally undefing Perl's line separator variable.
my $oldlen = length($oldval);
my $newlen = length($newval);
open (my $infile, '<', 'testin.txt') or die "could not open infile $!";
my $text; { local $/ = undef; $text = <$infile>; }
close($infile);
my $idx = 0;
while (($idx = index($text, $oldval, $idx)) > -1) {
substr($text,$idx,$oldlen) = $newval;
$idx += $newlen;
}
open (my $outfile, '>', 'testout.txt') or die "could not open outfile $!";
print {$outfile} $text;
close($outfile);
This produced 10 second plus runtimes -- up to 20+ times longer than any of the other variations. I then created a second version (5a) that instead of updating the $text string, writes the results directly to the output file - requiring significantly more logic - which I did not exhaustively test -- but does produce the same output as all of the other variations.
my $oldlen = length($oldval);
open (my $infile, '<', 'testin.txt') or die "could not open infile $!";
my $text; { local $/ = undef; $text = <$infile>; }
close($infile);
open (my $outfile, '>', 'testout5a.txt') or die "could not open outfile $!";
my $startpos = 0;
while ((my $idx = index($text, $oldval, $startpos)) >= $startpos ) {
if ($idx > $startpos) {
print {$outfile} substr($text, $startpos, $idx - $startpos);
}
print {$outfile} $newval;
$startpos = $idx + $oldlen;
}
if ($startpos < length($text)) {
print {$outfile} substr($text,$startpos);
}
close($outfile);
This version produced very fast runtimes -- which seems to show that there is considerable overhead in using substr as an lvalue to perform a string replacement.
The sixth variation is a comparison against the fourth variation (which uses File::Slurp) and the fifth variation (which "rolls its own" substitution code). This uses $Bill Luebkert's suggested method to slurp a file into a scalar string variable and Perl's substitution operator. As an aside, $Bill had also suggested that the "s" qualifier was required for the substitution operator. According to the perl docs, "s" is "treat string as single line". I ran it both with and without the "s" qualifier -- which made no difference in the results -- so I left it off. The "g" qualifier seems to be sufficient.
open (my $infile, '<', 'testin.txt') or die "could not open infile $!";
my $text; { local $/ = undef; $text = <$infile>; }
close($infile);
$text =~ s/$oldval/$newval/g;
open (my $outfile, '>', 'testout6.txt') or die "could not open outfile $!";
print {$outfile} $text;
close($outfile);
I also updated the fourth variation (using File::Slurp) to specify the "binmode => :raw" option for both the read_file and write_file calls -- as a few people thought this might produce faster runtimes. I don't believe that it had any measurable effect on runtime but I left it in.
my $text = read_file($input_file, binmode => ':raw');
$text =~ s/$oldval/$newval/g;
write_file($output_file, {binmode => ':raw'}, \$text);
And after reading the perl docs on the qualifiers for the substitution operator, I updated the second variation (while loop, line at a time) to add the "o" qualifier, "compile pattern only once", which I'm not sure had any measureable effect on runtimes -- but I left it in.
while (my $line = <$infile>) {
$line =~ s/$oldval/$newval/go;
print {$outfile} $line;
}
So the 6 variations are as follows.
Variation 1 -- one line map statement
Variation 2 -- while loop, line at a time, explicit variables
Variation 3 -- for loop, no intermediate variable (uses $_)
Variation 4 -- File::Slurp into scalar string variable, substitute, file_write, using binmode
Variation 5 -- simple slurp into scalar string variable, "roll your own" substitution directly to output file
Variation 6 -- simple slurp into scalar string variable, substitute, one line write to output file
Shown below are the run time statistics from 51 executions -- included as an HTML table -- which may not be viewable to everyone -- followed by a simple listing by variation using tabs -- which should be viewable to all.
In summary, through the 75th percentile, Chris Wagner's "roll your own" suggestion (variation 5) and variation 6 produced identical results. But variation 6 also produced 12 (out of 51) runtime results that ranged from 2.2 seconds to 3.2 seconds (10x+ slower than the other 39 resuls) -- whereas variation 5 produced consistently low results -- with only the last (worst) two results coming in at ~0.8 seconds. Running these same tests at 30 and 40 iterations showed similar behavior / results for these two variations -- so I don't think it's a "fluke". But I can't explain the difference other than variation 5 is not updating a 12MB string buffer in memory -- whereas version 6 does.
Variation 2 can produce very good results -- but showed wide variation in the results.
Variation 6 shows that it's better to slurp a file into a text string to perform substitutions -- than into a array to perform line at a time substitutions -- which variations 1, 3, and 4 do. And variations 5 and 6 show that it's better to slurp a file into a text variable by undef'ing Perl's line separator variable than to use the File::Slurp package.
But variation 5 is significantly more complex than the other variations -- and therefore more susceptible to errors.
Regards,
... Dewey
Run time statistics
[html table]
| Stat |
Var 1
|
Var 2
|
Var 3
|
Var 4
|
Var 5
|
Var 6
|
| Min |
1.0615
|
0.2924
|
0.6109
|
0.7711
|
0.2238
|
0.2193
|
| 25th Percentile |
1.8527
|
0.3200
|
0.7010
|
2.4285
|
0.2351
|
0.2286
|
| Median |
2.2833
|
1.9228
|
1.6524
|
3.2647
|
0.2418
|
0.2391
|
| 75th Percentile |
3.0093
|
2.3083
|
2.6238
|
3.8105
|
0.2509
|
0.2538
|
| StdDevP |
0.9023
|
1.0150
|
1.1688
|
1.0080
|
0.1061
|
0.9566
|
| VarP |
0.8142
|
1.0303
|
1.3660
|
1.0161
|
0.0112
|
0.9150 |
Individual results
Variation 1
1.0615 Min runtime
1.8527 25th percentile
2.2833 Median (50th percentile)
3.0093 75th percentile
0.9023 Std Deviation (StdDevP)
0.8142 Variance (VarP)
Variation 2
0.2924 Min runtime
0.3200 25th percentile
1.9228 Median (50th percentile)
2.3083 75th percentile
1.0150 Std Deviation
1.0303 Variance
Variation 3
0.6109 Min runtime
0.7010 25th percentile
1.6524 Median (50th percentile)
2.6238 75th percentile
1.1688 Std Deviation
1.3660 Variance
Variation 4
0.7711 Min runtime
2.4285 25th percentile
3.2647 Median (50th percentile)
3.8105 75th percentile
1.0080 Std Deviation
1.0161 Variance
Variation 5
0.2238 Min runtime
0.2351 25th percentile
0.2418 Median (50th percentile)
0.2509 75th percentile
0.1061 Std Deviation
0.0112 Variance
Variation 6
0.2193 Min runtime
0.2286 25th percentile
0.2391 Median (50th percentile)
0.2538 75th percentile
0.9566 Std Deviation
0.9150 Variance
_______________________________________________ Perl-Win32-Users mailing list [email protected] To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
