[redirected to [EMAIL PROTECTED]
Camilo Gonzalez wrote:
> I need to read in from a temp file that is about 10 megs big in 1.5
> meg increments and write results to a database. I can't slurp up the
> whole temp file because I'm only allowed 2 megs of memory. I was
> hoping to read in only 1.5 megs per pass of the file but the read
> seems to be slurping up the whole thing. Do I need sysread?
>
> Anyway, here's my attempt:
>
>
> open (TEMP, "dataTemp.txt") or die "Can't open dataTemp for
> reading:$!"; my $offset = 0;
> while (read TEMP, my $tempbuf, 1500000, $offset)
Because you increment $offset by 1500000 at the end of the loop, you're
extending $tempbuf with each pass through the loop.
A hint: When you write big numbers in perl, you can use underscores to make
them more readable:
$size = 1_500_000; # less likely to add or drop a zero!
If you don't use $offset, you'll fix that problem. However, you'll introduce
another, because the logical record that spans the 1.5 Mb boundary won't be
processed correctly. You'll have to add logic to handle that.
Is the input data not delimited by a known string that you could set $/ to?
Are input rows variously delimited by \r, \n, and \f?
> {
> my @temp = split/[\r\n\f]/, $tempbuf;
>
> foreach my $temp(@temp)
> {
> my ($label,
> $genre,
> $catalogue_num,
> $upc,
> $artist,
> $title,
> $description,
> $format,
> $shop_price,
> $dist_price,
> $dist_1,
> $dist_2,
> $vendor,
> $cost,
> $release_date,
> $special_buy,
> $items_buy_unit,
> $items_sell_unit,
> $quant_break,
> $asset_acct,
> $buy,
> $buy_unit_measure,
> $expense_cos_account,
> $inactive_item,
> $income_acct,
> $inventory,
> $item_picture,
> $minimum_level,
> $selling_base_price,
> $price_level_b,
> $price_level_c,
> $price_level_e,
> $price_level_f,
> $sell,
> $sell_unit_measure,
> $quantity_break) = split/\t/, $temp;
>
> next if $genre eq "";
>
> my $inserts = "INSERT INTO bigdata VALUES (
> '$label',
> '$genre',
> '$catalogue_num',
> '$upc',
> '$artist',
> '$title',
> '$description',
> '$format',
> '$shop_price',
> '$dist_price',
> '$dist_1',
> '$dist_2',
> '$vendor',
> '$cost',
> '$release_date',
> '$special_buy',
> '$items_buy_unit',
> '$items_sell_unit',
> '$quant_break',
> '$asset_acct',
> '$buy',
> '$buy_unit_measure',
> '$expense_cos_account',
> '$inactive_item',
> '$income_acct',
> '$inventory',
> '$item_picture',
> '$minimum_level',
> '$selling_base_price',
> '$price_level_b',
> '$price_level_c',
> '$price_level_e',
> '$price_level_f',
> '$sell',
> '$sell_unit_measure',
> '$quantity_break')";
>
> my $sth = $dbh->do($inserts) or warn "Can't load <TEMP> into bigdata:
> $!:$@";
If the database driver you're using allows placeholders, you can simplify
this greatly. Also, using a prepared statement (if supported) can speed this
up as well.
>
>
>
>
> }
> $offset += 1500000;
> }
> close (TEMP);
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
