Re: Fastest way to store ints and floats on disk

2008-08-10 Thread Aahz
In article <[EMAIL PROTECTED]>, Paul Rudin <[EMAIL PROTECTED]> wrote: >Laszlo Nagy <[EMAIL PROTECTED]> writes: >> >> Permature optimalization is the root of all evil. (Who said that?) > >Knuth I think. "Premature optimization is the root of all evil in programming." --C.A.R. Hoare (often misatt

Re: Fastest way to store ints and floats on disk

2008-08-09 Thread castironpi
On Aug 9, 4:43 pm, John Machin <[EMAIL PROTECTED]> wrote: > On Aug 10, 4:58 am, castironpi <[EMAIL PROTECTED]> wrote: > > > > > On Aug 7, 2:27 pm, "M.-A. Lemburg" <[EMAIL PROTECTED]> wrote: > > > > On 2008-08-07 20:41, Laszlo Nagy wrote: > > > > >  Hi, > > > > > I'm working on a pivot table. I woul

Re: Fastest way to store ints and floats on disk

2008-08-09 Thread John Machin
On Aug 10, 4:58 am, castironpi <[EMAIL PROTECTED]> wrote: > On Aug 7, 2:27 pm, "M.-A. Lemburg" <[EMAIL PROTECTED]> wrote: > > > > > On 2008-08-07 20:41, Laszlo Nagy wrote: > > > > Hi, > > > > I'm working on a pivot table. I would like to write it in Python. I > > > know, I should be doing that in

Re: Fastest way to store ints and floats on disk

2008-08-09 Thread castironpi
On Aug 7, 2:27 pm, "M.-A. Lemburg" <[EMAIL PROTECTED]> wrote: > On 2008-08-07 20:41, Laszlo Nagy wrote: > > > > > > >  Hi, > > > I'm working on a pivot table. I would like to write it in Python. I > > know, I should be doing that in C, but I would like to create a cross > > platform version which c

Re: Fastest way to store ints and floats on disk

2008-08-09 Thread Matthew Woodcraft
Laszlo Nagy <[EMAIL PROTECTED]> writes: > The facts table cannot be kept in memory because it is too big. I need to > store it on disk, be able to read incrementally, and make statistics. In most > cases, the "statistic" will be simple sum of the measures, and counting the > number of facts affect

Re: Fastest way to store ints and floats on disk

2008-08-08 Thread Paul Rudin
Laszlo Nagy <[EMAIL PROTECTED]> writes: > Permature optimalization is the root of all evil. (Who said that?) Knuth I think. But note the "premature" bit - around here people sometimes give the impression that it goes "optimisation is the root of all evil". -- http://mail.python.org/mailman/li

Re: Fastest way to store ints and floats on disk

2008-08-08 Thread Laszlo Nagy
Hmm... I wrote an browser based analysis tool and used the working name pyvot... Is this for the public domain? I found Numeric to provide the best balance of memory footprint and speed. I also segregated data prep into a separate process to avoid excessive memory use at run time. Turns o

Re: Fastest way to store ints and floats on disk

2008-08-07 Thread Emile van Sebille
Laszlo Nagy wrote: Hi, I'm working on a pivot table. Hmm... I wrote an browser based analysis tool and used the working name pyvot... I found Numeric to provide the best balance of memory footprint and speed. I also segregated data prep into a separate process to avoid excessive memor

Re: Fastest way to store ints and floats on disk

2008-08-07 Thread castironpi
On Aug 7, 1:41 pm, Laszlo Nagy <[EMAIL PROTECTED]> wrote: >   Hi, > > I'm working on a pivot table. I would like to write it in Python. I > know, I should be doing that in C, but I would like to create a cross > platform version which can deal with smaller databases (not more than a > million facts

Re: Fastest way to store ints and floats on disk

2008-08-07 Thread M.-A. Lemburg
On 2008-08-07 20:41, Laszlo Nagy wrote: Hi, I'm working on a pivot table. I would like to write it in Python. I know, I should be doing that in C, but I would like to create a cross platform version which can deal with smaller databases (not more than a million facts). The data is first i