Hi Richard, CCing the list.
On Friday 01 October 2010 18:32:32 Richard Green wrote: > Thanks so much Shlomi, I assumed the database was the probably the best > course of action. I haven't worke with C in ten years but was considering > it for this problem. Can you point me toward the solution you had in C? I didn't really have a solution for this problem in C, but for a completely different problem. Namely a solver for Freecell - http://fc-solve.berlios.de/ , which I've been working on since 2000 . It contains an implementation of a hash, but it's not a general-purpose hash and it's quite tied to the rest of the library. I just noted the fact that 20 million positions stored there exceeded my 3 GB of RAM, so you'll know that it would be hard to handle 88 million records in RAM using a hash. Regards, Shlomi Fish > Thanks -Rich > > On Fri, Oct 1, 2010 at 12:03 AM, Shlomi Fish <shlo...@iglu.org.il> wrote: > > Hi Richard, > > > > On Friday 01 October 2010 05:33:20 Richard Green wrote: > > > Hi everyone, could use some advice on a perl script I wrote using > > > hashes. > > > > I > > > > > have three files ( each file is a list of indexes) my program loads > > > these indexes into hashes and compares the differences and > > > similarities between them. With smaller files it runs fine. problem is > > > I now files have about > > > > 88 > > > > > million records and the script has been running for days. not sure the > > > > best > > > > > way to resolve the issue.(pieces of code samples below) One suggestion > > > > that > > > > > was given to me was to load the first file into a hash and then as I > > > open the next file I immediately do a comparison of the second hash > > > into the > > > > 1st > > > > > one for similarities and differences. I can't really get my head around > > > > how > > > > > to that. Is there a simple way to compare three very large hashes > > > without so much demand on memory? The program manages to loads the > > > large hashes without problems and as I mentioned previously smaller > > > files have no issues. Any suggestions folks have are muchly > > > appreciated. > > > Thanks R > > > > Well, your code seems fine except for the fact that you used the > > «$$my_hash_ref{$key}» notation instead of the clearer > > «$my_hash_ref->{$key}» > > (which shouldn't matter to perl). There may be some CPAN modules to do > > what you're doing more efficiently. However, I should note that on most > > systems perl will have a difficulty managing a hash with 88 million > > records, because > > perl variables and data structure have a lot of overhead. > > > > Just as an anecdote, I recently worked on a C program which manages a > > gigantic > > hash in memory and I could only manage about 20 million records before it > > exceeded the 3 GB RAM in my x86-64 Linux laptop and reached the swap. > > (It's a > > development branch of Freecell Solver - http://fc-solve.berlios.de/ if > > you're > > interested.) > > > > As a result, I strongly recommend that you instead delegate the hard work > > to a > > database: > > > > http://perl-begin.org/uses/databases/ > > > > Regards, > > > > Shlomi Fish > > > > -- > > ----------------------------------------------------------------- > > Shlomi Fish http://www.shlomifish.org/ > > Why I Love Perl - http://shlom.in/joy-of-perl > > > > <rindolf> She's a hot chick. But she smokes. > > <go|dfish> She can smoke as long as she's smokin'. > > > > Please reply to list if it's a mailing list post - http://shlom.in/reply > > . > > > > -- > > To unsubscribe, e-mail: beginners-unsubscr...@perl.org > > For additional commands, e-mail: beginners-h...@perl.org > > http://learn.perl.org/ -- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ Stop Using MSIE - http://www.shlomifish.org/no-ie/ <rindolf> She's a hot chick. But she smokes. <go|dfish> She can smoke as long as she's smokin'. Please reply to list if it's a mailing list post - http://shlom.in/reply . -- To unsubscribe, e-mail: beginners-unsubscr...@perl.org For additional commands, e-mail: beginners-h...@perl.org http://learn.perl.org/
