Craig,
> Is there a loop I can use that will process the first 10,000
> files, then
> exit; then the next 10,000 files, then exit; and so on until
> all files
> have been processed? The exiting is so I can run a SQL Server job to
> insert the processed files into SQL Server.
Like every other poster I think the warning lights are going off here.
If you need to restart your program in order to get process your files
your grok algorithm probably has issues. However to answer your
question you can call yourself again via a system call.
<UNTESTED CODE>
my $file_list = "file_list.txt";
my $max_process = 10_000;
my @files;
unless ($ARGV[0]) {
# We were called without arguments so we are the parent.
create_file_list($file_list); # create a text file of 70_000
file names to process
while (-e $file_list) { # While the text file is still here
# Call ourself again with an argument to process the
next 10_000 files.
die "Sub call failed" unless system( "perl $0 grok")==0
}
}
# We were called with arguments so we grok the next 10K files
open IN, $file_list or die "Cached file list not found : $!\n";
@files=<IN>; # Slurp
close IN;
# Extract the first 10000 files;
my @to_process = splice ( @files, 0, $max_process);
# Update the cache
open OUT, ">$file_list" or die "Cannot overwrite $file_lists : $!";
print OUT, @files;
close OUT
for my $file (@to_process) {
chomp $file;
Process($file);
}
# Delete file and signal parent that we're done.
unlink $file_list unless @files
exit 0
**********************************************************************
Registered Office:
Marks and Spencer plc
Waterside House
35 North Wharf Road
London
W2 1NW
Registered No. 214436 in England and Wales.
Telephone (020) 7935 4422
Facsimile (020) 7487 2670
<<www.marksandspencer.com>>
Please note that electronic mail may be monitored.
This e-mail is confidential. If you received it by mistake, please let us know
and then delete it from your system; you should not copy, disclose, or
distribute its contents to anyone nor act in reliance on this e-mail, as this
is prohibited and may be unlawful.
_______________________________________________
Perl-Win32-Users mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
