Re: In-place memory manager, mmap (was: Fastest way to store ints and floats on disk)

2008-08-24 Thread Mohamed Yousef
source-forge places no limits and is a very good place (specific project site / releases / usage counters) -- http://mail.python.org/mailman/listinfo/python-list

Re: In-place memory manager, mmap (was: Fastest way to store ints and floats on disk)

2008-08-24 Thread castironpi
On Aug 24, 9:52 am, Kris Kennaway <[EMAIL PROTECTED]> wrote: > castironpi wrote: > > Hi, > > > I've got an "in-place" memory manager that uses a disk-backed memory- > > mapped buffer.  Among its possibilities are: storing variable-length > > strings and structures for persistence and interprocess c

Re: In-place memory manager, mmap (was: Fastest way to store ints and floats on disk)

2008-08-24 Thread Kris Kennaway
castironpi wrote: Hi, I've got an "in-place" memory manager that uses a disk-backed memory- mapped buffer. Among its possibilities are: storing variable-length strings and structures for persistence and interprocess communication with mmap. It allocates segments of a generic buffer by length a

In-place memory manager, mmap (was: Fastest way to store ints and floats on disk)

2008-08-22 Thread castironpi
Hi, I've got an "in-place" memory manager that uses a disk-backed memory- mapped buffer. Among its possibilities are: storing variable-length strings and structures for persistence and interprocess communication with mmap. It allocates segments of a generic buffer by length and returns an offset

Re: Fastest way to store ints and floats on disk

2008-08-10 Thread Aahz
In article <[EMAIL PROTECTED]>, Paul Rudin <[EMAIL PROTECTED]> wrote: >Laszlo Nagy <[EMAIL PROTECTED]> writes: >> >> Permature optimalization is the root of all evil. (Who said that?) > >Knuth I think. "Premature optimization is the root of all evil in programming." --C.A.R. Hoare (often misatt

Re: Fastest way to store ints and floats on disk

2008-08-09 Thread castironpi
On Aug 9, 4:43 pm, John Machin <[EMAIL PROTECTED]> wrote: > On Aug 10, 4:58 am, castironpi <[EMAIL PROTECTED]> wrote: > > > > > On Aug 7, 2:27 pm, "M.-A. Lemburg" <[EMAIL PROTECTED]> wrote: > > > > On 2008-08-07 20:41, Laszlo Nagy wrote: > > > > >  Hi, > > > > > I'm working on a pivot table. I woul

Re: Fastest way to store ints and floats on disk

2008-08-09 Thread John Machin
On Aug 10, 4:58 am, castironpi <[EMAIL PROTECTED]> wrote: > On Aug 7, 2:27 pm, "M.-A. Lemburg" <[EMAIL PROTECTED]> wrote: > > > > > On 2008-08-07 20:41, Laszlo Nagy wrote: > > > > Hi, > > > > I'm working on a pivot table. I would like to write it in Python. I > > > know, I should be doing that in

Re: Fastest way to store ints and floats on disk

2008-08-09 Thread castironpi
On Aug 7, 2:27 pm, "M.-A. Lemburg" <[EMAIL PROTECTED]> wrote: > On 2008-08-07 20:41, Laszlo Nagy wrote: > > > > > > >  Hi, > > > I'm working on a pivot table. I would like to write it in Python. I > > know, I should be doing that in C, but I would like to create a cross > > platform version which c

Re: Fastest way to store ints and floats on disk

2008-08-09 Thread Matthew Woodcraft
Laszlo Nagy <[EMAIL PROTECTED]> writes: > The facts table cannot be kept in memory because it is too big. I need to > store it on disk, be able to read incrementally, and make statistics. In most > cases, the "statistic" will be simple sum of the measures, and counting the > number of facts affect

Re: numpy and filtering (was: Fastest way to store ints and floats on disk)

2008-08-08 Thread Robert Kern
Laszlo Nagy wrote: Attached there is an example program that only requires numpy. At the end I have two numpy array: rdims: [[3 1 1] [0 0 4] [1 3 0] [2 2 0] [3 3 3] [0 0 2]] rmeas: [[10.0 254.0] [4.0 200.0] [5.0 185.0] [5000.0 160.0] [15.0 260.0] [2.0 180.0]] I would l

numpy and filtering (was: Fastest way to store ints and floats on disk)

2008-08-08 Thread Laszlo Nagy
Attached there is an example program that only requires numpy. At the end I have two numpy array: rdims: [[3 1 1] [0 0 4] [1 3 0] [2 2 0] [3 3 3] [0 0 2]] rmeas: [[10.0 254.0] [4.0 200.0] [5.0 185.0] [5000.0 160.0] [15.0 260.0] [2.0 180.0]] I would like to use numpy to

Re: Fastest way to store ints and floats on disk

2008-08-08 Thread Paul Rudin
Laszlo Nagy <[EMAIL PROTECTED]> writes: > Permature optimalization is the root of all evil. (Who said that?) Knuth I think. But note the "premature" bit - around here people sometimes give the impression that it goes "optimisation is the root of all evil". -- http://mail.python.org/mailman/li

Re: Fastest way to store ints and floats on disk

2008-08-08 Thread Laszlo Nagy
Hmm... I wrote an browser based analysis tool and used the working name pyvot... Is this for the public domain? I found Numeric to provide the best balance of memory footprint and speed. I also segregated data prep into a separate process to avoid excessive memory use at run time. Turns o

Re: Fastest way to store ints and floats on disk

2008-08-07 Thread Emile van Sebille
Laszlo Nagy wrote: Hi, I'm working on a pivot table. Hmm... I wrote an browser based analysis tool and used the working name pyvot... I found Numeric to provide the best balance of memory footprint and speed. I also segregated data prep into a separate process to avoid excessive memor

Re: Fastest way to store ints and floats on disk

2008-08-07 Thread castironpi
On Aug 7, 1:41 pm, Laszlo Nagy <[EMAIL PROTECTED]> wrote: >   Hi, > > I'm working on a pivot table. I would like to write it in Python. I > know, I should be doing that in C, but I would like to create a cross > platform version which can deal with smaller databases (not more than a > million facts

Re: Fastest way to store ints and floats on disk

2008-08-07 Thread M.-A. Lemburg
On 2008-08-07 20:41, Laszlo Nagy wrote: Hi, I'm working on a pivot table. I would like to write it in Python. I know, I should be doing that in C, but I would like to create a cross platform version which can deal with smaller databases (not more than a million facts). The data is first i

Fastest way to store ints and floats on disk

2008-08-07 Thread Laszlo Nagy
Hi, I'm working on a pivot table. I would like to write it in Python. I know, I should be doing that in C, but I would like to create a cross platform version which can deal with smaller databases (not more than a million facts). The data is first imported from a csv file: the user selects