I added your fourth variation -- but given the use of "File::Slurp" to read the file, I substituted "print {$outfile} $text;" with the "write_file" function from this same package. I also eliminated the binmode => raw option which really isn't appropriate for a text file -- and didn't cause any difference in the runtimes. Using write_file also eliminated the explicit open and close file statements on the output file.
my $text = read_file($input_file);
$text =~ s/$oldval/$newval/g;
write_file($output_file, \$text);
I then ran all four test scripts 15 times each. Average run times are as follows.
Test Average Run Time
testit1 0.1855 (map one liner)
testit2 0.0813 (while loop - line at a time)
testit3 0.1358 (for loop)
testit4 0.0896 (file slurp - read file, substitute, write file)
------
Overall 0.1230
So the while loop maintains its advantage by a very slight margin (.0083 seconds) which is probably well within the margin of error -- and certainly isn't noticeable.
Functional versus procedural versus OO is a matter of programming style and aesthetics -- and I agree that the key goals are almost always simplicity, correctness, reliability, and maintainability -- and all three of these styles / idioms can achieve these goals (and also produce gibberish, hard to identify bugs, ...).
Betraying my age a bit, I remember the old days of "Doctor Dobb's Journal" -- when it had the subtitle "running fast and light without overbyte" (before "Moore's Law" effectively made it irrevelant - mostly). From the perspective of this old adage, the simple while loop, line at a time variation is best as it runs quickly with a very light memory requirement independent of the file size. And even under today's typical PC environment with 512MB to 1GB of memory (and 80GB disks in laptops, with 300GB+ common in desktops), where slurping a 3MB file poses no issues, there's still the potential problem of a user running such a script against a *much* larger file. With all of the other variations, you run the risk of a casual user running it against a 500MB file, or multi-gigabyte file -- which would force the OS to page (oh no ;-) -- though hopefully not crash (though I'm not sure what would happen on a 512MB PC slurping a 3GB file). And I suspect that it would fail on a file greater than 3-4GB due to 32 bit memory process limitations. Yes, with File::Slurp, you can read and write chunks at a time -- but this adds complexity. With the while loop, line at a time variation, file size is irrelevant -- other than as related to runtime.
Regards,
... Dewey
| Lyle Kopnicky <[EMAIL PROTECTED]>
03/01/2006 01:46 PM |
|
D D Allen wrote:
I think what's going on is that for variations 1 and 3, Perl is "slurping" the input file into an array, performing the substitutions on the array elements, and then writing out the array elements. For variation 2, there's no array involved. It's simply "read a line, perform the substitution on that line, write a line". And I think the explanation for the time differences between 1 and 3 is that there is a certain amount of overhead associated with the map function -- that's not present with the for loop.
Thanks for your informative test. When we make decisions about how to code something, it is important to consider what factors are most important in our situation. Sometimes, speed will be critical. At other times, conceptual clarity of the code, for ease of maintenance, will be of highest concern. Most of the time we choose a balance. Personally, I find that if speed is not critical (and if it is, you should be using something faster than current Perl implementations), I want to emphasize conceptual clarity.
I feel that in the long run, it is best to write clearer code, unless the performance penalty is overwhelming. Understand that future versions of Perl may be optimized to handle some things more efficiently, e.g. map. If more people used it, there would be more incentive for Wall et al to optimize for it.
One issue that arises in designing for clarity in the code is that different developers are used to different idioms. So, what is clear and simple to me may seem obtuse to a future maintainer, and vice-versa. Many of the practices in the Best Practices book seem strange to many Perl developers, but Conway suggests that if we would all get used to them, we'd be happier in the long run. It is in that spirit that I prefer the three-line map solution to the while loop:
my @lines = <$infile>;
map { s/$oldval/$newval/g; $_ } @lines;
print {$outfile} @lines;
It conceptually separates I/O from computations. It is also more declarative - saying what you want rather than how to do it. Some programmers may find it less familiar than while loops, but I think that is only because they have been taught to use while loops. If you start a new programmer from scratch, my bet is they'll find the map solution conceptually simpler. Granted, I have a functional programming background, so I'm apt to say such things.
Anyway, despite the fact that the map solution is simpler than the while loop, there is yet a simpler solution to this problem, which I mentioned earlier. It involves sucking in the entire file and doing substitutions on it. There is no need to impose the conceptual framework of line-based processing. It might even be faster. I'm eager to find out.
Here is the variant I suggest you test:
use File::Slurp;
...
my $text = read_file($input_file, binmode => ':raw');
$text =~ s/$oldval/$newval/g;
print {$outfile} $text;
(I know Conway suggests using Perl6::Slurp, but I couldn't find a PPM for it, so I used File::Slurp. Admittedly, the binmode => ':raw' is a bit obtuse, and could be left out. I do think it will boost performance slightly, which I am curious about in the context of this discussion.)
--
Lyle Kopnicky
Software Project Engineer
Veicon Technology, Inc.
_______________________________________________ Perl-Win32-Users mailing list [email protected] To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
