Re: Please help: perl run out of memory

2022-04-27 Thread David Emanuel da Costa Santiago
Às 11:33 de 17/04/22, wilson escreveu: hello the experts, can you help check my script for how to optimize it? currently it was going as "run out of memory". $ perl count.pl Out of memory! Killed My script: use strict; my %hash; my %stat; To be honest you don't need the %stat, however

Re: Please help: perl run out of memory

2022-04-26 Thread hw
On Sun, 2022-04-17 at 17:33 +0800, wilson wrote: > hello the experts, > > can you help check my script for how to optimize it? > currently it was going as "run out of memory". > > $ perl count.pl > Out of memory! > Killed I would use a database like Mariadb for this, not only to create a repor

Re: Please help: perl run out of memory

2022-04-22 Thread David Precious
On Thu, 21 Apr 2022 07:12:07 -0700 al...@coakmail.com wrote: > OP maybe need the streaming IO for reading files. Which is what they were already doing - they used: while () { ... } Which, under the hood, uses readline, to read a line at a time. (where "HD" is their global fileh

Re: Please help: perl run out of memory

2022-04-21 Thread alice
OP maybe need the streaming IO for reading files. Thanks On 2022-04-21 21:56, David Precious wrote: > On Thu, 21 Apr 2022 17:26:15 +0530 > "M.N Thanishka sree Manikandan" wrote: > >> Hi wilson >> Try this module file::slurp > > Given that the OP is running into memory issues processing an 80

Re: Please help: perl run out of memory

2022-04-21 Thread David Precious
On Thu, 21 Apr 2022 17:26:15 +0530 "M.N Thanishka sree Manikandan" wrote: > Hi wilson > Try this module file::slurp Given that the OP is running into memory issues processing an 80+ million line file, I don't think suggesting a CPAN module designed to read the entire contents of a file into mem

Re: Please help: perl run out of memory

2022-04-21 Thread M.N Thanishka sree Manikandan
Hi wilson Try this module file::slurp Regards, Manikandan On Sun, 17 Apr, 2022, 15:03 wilson, wrote: > hello the experts, > > can you help check my script for how to optimize it? > currently it was going as "run out of memory". > > $ perl count.pl > Out of memory! > Killed > > > My script: > u

Re: Please help: perl run out of memory

2022-04-21 Thread Adriel Peng
I am not sure, but can Tie::Hash etc be used by tying hash to a local file to reduce the memory use? regards.

Re: Please help: perl run out of memory

2022-04-18 Thread Rob Coops
Hi Wilson, Looking at the script I see some room for improvement. You currently declare %hash as a global variable, and keep it around forever. With tens of millions of rows that is quite a large structure to just have sitting around after you have build the %stat hash. So I would start by limitin

Re: Please help: perl run out of memory

2022-04-17 Thread David Mertens
I see nothing glaringly inefficient in the Perl. This would be fine on your system if you were dealing with 1 million items, but you could easily be pushing up against your system's limits with the generic data structures that Perl uses, especially since Perl is probably using 64-bit floats and int