On Thu, Mar 18, 2010 at 7:30 PM, Peter Scott <pe...@psdt.com> wrote:
> On Wed, 17 Mar 2010 20:19:01 +0530, raphael() wrote:
>
> > Hello,
> >
> > Is there a way to save/store downloaded data (using WWW::Mechanize) in
> > memory (temporarily)
> > rather than writing it to disk. Like store 10MB in memory and then flush
> > it to the hard disk when data reaches 10MB.
>
> Not easily. WWW::Mechanize is a LWP::UserAgent; look at the
> documentation for that module and see the :content_cb hook. Write a
> handler to concatenate data in memory until you reach 10MB and then flush
> to disk, writing the remainder there. Set read_size_hint to below 10MB.
>
> --
> Peter Scott
> http://www.perlmedic.com/ http://www.perldebugged.com/
> http://www.informit.com/store/product.aspx?isbn=0137001274
> http://www.oreillyschool.com/courses/perl1/
>
> --
> To unsubscribe, e-mail: beginners-unsubscr...@perl.org
> For additional commands, e-mail: beginners-h...@perl.org
> http://learn.perl.org/
>
>
>
*update*
I just can't get it :(
---------- CODE ----------
use strict;
use warnings;
use WWW::Mechanize;
use File::Basename;
--BLAH BLAH--
open(INFILE, '<', "$file")
or die "! INFILE\n";
my @urls = <INFILE>;
close( INFILE );
my $wmc = WWW::Mechanize->new
(
agent => 'Mozilla/5.0',
timeout => '20',
max_redirect => '2',
autocheck => '0',
);
for my $url ( @urls ) {
last if ( $url =~ m/__END__/ );
next unless ( $url =~ m/^(?:\s+)?http/ );
my $base = basename( $url );
$wmc->max_size( '512' );
$wmc->get( "$url" );
my ( $url, $extension ) = $wmc->content =~ m!(REGEX)-IS-(HERE)!;
$wmc->max_size( undef );
--SOME CODE HERE--
open( my $file2write, '>>', "$base.$extension" );
binmode $file2write;
my $data_in_memory;
my $bytes_received = 0;
$wmc->get(
"$url",
':content_cb' => sub {
my ($data, $response) = @_;
$bytes_received += length($data);
$data_in_memory .= $data;
if ( $bytes_received >= 1048576 ) {
$bytes_received = 0;
print {$file2write} "$data_in_memory";
$data_in_memory = undef;
}
});
print "\n";
close $file2write;
}
---------- END ----------
Now the above code I wrote using your advice works :)
The only problem is that when the download is ending if the last data
received
is less than one MB (which it naturally is) the end part is not written to
the file on HDD
since the sub doesn't enter if{} block.
So the whole file is now written to HDD in intervals which amount to the
time required to fetch 1MB of data (will increase to 10 MB).
Is there something like another loop or counter that can be added to this
so that it flushes out the last data received even if it is less thank 1MB?
Any help would be appreciated!
