-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Martin wrote:
> I'd think he's talking about in memory SQLite Databases, this way you
> should be quite fast _and_ could dump all that to a persistent
> storage...
I was just talking about regular on disk SQLite databases. In terms of
priming the pum
Hi,
2008/12/24 :
> Hi Roger,
>
>> you may want to consider using SQLite
>
> Thank you for your suggestion about looking at SQLite. I haven't
> compared the performance of SQLite to Python dictionaries, but I'm
> skeptical that SQLite would be faster than in-memory Python dictionaries
> for the ty
r your help Roger!
Regards,
Malcolm
- Original message -
From: "Roger Binns"
To: python-list@python.org
Date: Wed, 24 Dec 2008 00:50:49 -0800
Subject: Re: Most efficient way to build very large dictionaries
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
pyt...@bdurham.com wrote:
&g
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
pyt...@bdurham.com wrote:
> Thank you for your suggestion about looking at SQLite. I haven't
> compared the performance of SQLite to Python dictionaries, but I'm
> skeptical that SQLite would be faster than in-memory Python dictionaries
> for the type
t need SQLite's
ability to work with data sets larger than my physical memory.
Regards,
Malcolm
- Original message -
From: "Roger Binns"
To: python-list@python.org
Date: Wed, 24 Dec 2008 00:19:56 -0800
Subject: Re: Most efficient way to build very large dictionaries
-BE
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
pyt...@bdurham.com wrote:
> Can I take advantage of this knowledge to optimize
You do the optimization last :-) The first thing you need to do is make
sure you have a way of validating you got the correct results. With 25M
entries it would be very e
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
pyt...@bdurham.com wrote:
> I would appreciate your thoughts on whether there are advantages to
> working with a pre-built dictionary and if so, what are the best ways to
> create a pre-loaded dictionary.
Based on this and your other thread, you may w
I'm working with some very large dictionaries (about 25M items
per dictionary) that get filled based on data parsed from text
based log files. I'm using Python dictionaries to track the
frequency of specific combinations of events, eg, I build a key
and then add various timing info from the current
