Re: Million hash comparisons

2008-11-20 Thread Jenda Krynicky
From: Nitin Kalra <[EMAIL PROTECTED]> > All I want to do is to compare 2 hashes having 10-20 > Million key/ value pairs, this means 100-200 Million > comparisons. Hashes are the best thing which come to > my mind but storing hashes in memory overloads the > system and lowers the performance drastic

Re: Million hash comparisons

2008-11-20 Thread Jenda Krynicky
From: Rob Dixon <[EMAIL PROTECTED]> > Jenda Krynicky wrote: > > From: Nitin Kalra <[EMAIL PROTECTED]> > >> > >> In a Perl script of mine I have to compare 2 8M-10M > >> files(each). Which mean 80-90M searches. As a normal > >> procedure (upto 1 M)I use hashes, but going beyond 1M > >> system perfor

Re: Million hash comparisons

2008-11-19 Thread Rob Dixon
Jenda Krynicky wrote: > From: Nitin Kalra <[EMAIL PROTECTED]> >> >> In a Perl script of mine I have to compare 2 8M-10M >> files(each). Which mean 80-90M searches. As a normal >> procedure (upto 1 M)I use hashes, but going beyond 1M >> system performance degrades drastically. > > You mean you comp

Re: Million hash comparisons

2008-11-19 Thread Jenda Krynicky
From: Nitin Kalra <[EMAIL PROTECTED]> > Hi Community, > > In a Perl script of mine I have to compare 2 8M-10M > files(each). Which mean 80-90M searches. As a normal > procedure (upto 1 M)I use hashes, but going beyond 1M > system performance degrades drastically. You mean you compute MD5 (or some

Re: Million hash comparisons

2008-11-19 Thread Dr.Ruud
Nitin Kalra schreef: > In a Perl script of mine I have to compare 2 8M-10M > files(each). Which mean 80-90M searches. As a normal > procedure (upto 1 M)I use hashes, but going beyond 1M > system performance degrades drastically. > > If anybody has faced something like this then plz > share the sa

Re: Million hash comparisons

2008-11-19 Thread Rob Dixon
Nitin Kalra wrote: > > In a Perl script of mine I have to compare 2 8M-10M > files(each). Which mean 80-90M searches. As a normal > procedure (upto 1 M)I use hashes, but going beyond 1M > system performance degrades drastically. > > If anybody has faced something like this then plz > share the sa

Re: Million hash comparisons

2008-11-19 Thread Rob Dixon
Mr. Shawn H. Corey wrote: > On Wed, 2008-11-19 at 18:30 +, Nitin Kalra wrote: >> >> In a Perl script of mine I have to compare 2 8M-10M >> files(each). Which mean 80-90M searches. As a normal >> procedure (upto 1 M)I use hashes, but going beyond 1M >> system performance degrades drastically. >>

Re: Million hash comparisons

2008-11-19 Thread Mr. Shawn H. Corey
On Wed, 2008-11-19 at 18:30 +, Nitin Kalra wrote: > Hi Community, > > In a Perl script of mine I have to compare 2 8M-10M > files(each). Which mean 80-90M searches. As a normal > procedure (upto 1 M)I use hashes, but going beyond 1M > system performance degrades drastically. > > If anybody ha