Re: python+sqlite3 in memory database

2010-08-30 Thread Benjamin Peterson
John Nagle animats.com> writes: > "sqlite" has reasonably good SELECT performance on simple indices, > but anything beyond that isn't all that great. Multiple processes > updating the same sqlite database will have terrible performance, > because the locking mechanism not only locks the entir

Re: python+sqlite3 in memory database

2010-08-30 Thread John Nagle
On 8/30/2010 12:00 PM, Benjamin Peterson wrote: Denis Gomes gmail.com> writes: Eventually my goal is to dynamically load and unload sections of a file based database (could be tables or rows) in and out of memory for effeciency purposes. Have you actually found this to be an useful optimiz

Re: python+sqlite3 in memory database

2010-08-30 Thread Denis Gomes
Yep, I see what you are saying. I am going to do a bit more research to see how sqlite3 works internally, ie. cache size, page size, etc, and then decide if I will need to mess with in-memory databases. Thanks for your insight, appreciate it. Denis On Mon, Aug 30, 2010 at 3:51 PM, Benjamin Pete

Re: python+sqlite3 in memory database

2010-08-30 Thread Benjamin Peterson
Denis Gomes gmail.com> writes: > > > Hey Benjamin, >   >  Take a look at this website I found about cached and in-memory databases.  I think the gist of the article is that caching is good if you are doing SELECTs on data that is frequently used whereas in-memory speeds up writes, (inserts and

Re: python+sqlite3 in memory database

2010-08-30 Thread Denis Gomes
Hey Benjamin, Take a look at this website I found about cached and in-memory databases. I think the gist of the article is that caching is good if you are doing SELECTs on data that is frequently used whereas in-memory speeds up writes, (inserts and updates) to the db as well as querying. Maybe I

Re: python+sqlite3 in memory database

2010-08-30 Thread Benjamin Peterson
Denis Gomes gmail.com> writes: > > Eventually my goal is to dynamically load and unload sections of a file based database (could be tables or rows) in and out of memory for effeciency purposes. Have you actually found this to be an useful optimization? SQLite already internally caches database

python+sqlite3 in memory database

2010-08-30 Thread Denis Gomes
Hi Everyone, I am using sqlite3 with python2.5 and the pysqlite wrapper. I am trying to copy tables from one database (in memory) to another database (file) using ATTACH. I looked on the internet and found a couple of sites that show how to do this but the table schema is not copied. def Save

Re: (in memory) database

2008-09-02 Thread Paul Boddie
On 2 Sep, 17:38, Zentrader <[EMAIL PROTECTED]> wrote: > > I don't understand why Cameron has a different version of Python which > > doesn't seem to have sqlite support enabled. > > Agreed, but won't the package manager tell him if python-sqlite is > installed? It shouldn't need to be installed: t

Re: (in memory) database

2008-09-02 Thread Peter Otten
Zentrader wrote: >> I don't understand why Cameron has a different version of Python which >> doesn't seem to have sqlite support enabled. > > Agreed, but won't the package manager tell him if python-sqlite is > installed? That would be the next step since it appears that SQLite > intself is alr

Re: (in memory) database

2008-09-02 Thread Zentrader
> I don't understand why Cameron has a different version of Python which > doesn't seem to have sqlite support enabled. Agreed, but won't the package manager tell him if python-sqlite is installed? That would be the next step since it appears that SQLite intself is already installed. Since Ubunt

Re: (in memory) database

2008-09-02 Thread Paul Boddie
On 2 Sep, 04:46, Zentrader <[EMAIL PROTECTED]> wrote: > On Ubuntu you want to install something like python-sqlite (a search > for "python" should turn up everything).  There are 2 parts to this, > SQLite and the python bindings to SQLite.  So you seem to have SQLite > installed but not the Python

Re: (in memory) database

2008-09-01 Thread Zentrader
On Ubuntu you want to install something like python-sqlite (a search for "python" should turn up everything). There are 2 parts to this, SQLite and the python bindings to SQLite. So you seem to have SQLite installed but not the Python bindings. Also, on some systems you have to have python-sqlit

Re: (in memory) database

2008-09-01 Thread M.-A. Lemburg
On 2008-08-31 15:15, mark wrote: > Hi there, > > I need to extract data from text files (~4 GB) on this data some > operations are performed like avg, max, min, group etc. The result is > formated and written in some other text files (some KB). > > I currently think about database tools might be

Re: (in memory) database

2008-09-01 Thread Bruno Desthuilliers
mark a écrit : Hi there, I need to extract data from text files (~4 GB) on this data some operations are performed like avg, max, min, group etc. The result is formated and written in some other text files (some KB). I currently think about database tools might be suitable for this. I would jus

Re: (in memory) database

2008-09-01 Thread Roel Schroeven
Roel Schroeven schreef: Cameron Laird schreef: I now suspect that my 2.5 packaging has something to do with 64-bit builds; all my 32-bit Ubuntu servers have Python 2.5.2, while the 64-bit ones are at Python 2.5. Strange: my 64-bit Ubuntu 8.04 has Python 2.5.2, with working sqlite: (mine is

Re: (in memory) database

2008-08-31 Thread Cousin Stanley
> > Yes and no. My own experience with Debian packages > is that with a standard > apt-get install python2.5 > an attempt to > import sqlite3 > results in > ImportError: No module named _sqlite3 > From Kubuntu 8.04 $ uname -a Linux em1 2.6.24-19-generic #1 SMP Wed Aug 20

Re: (in memory) database

2008-08-31 Thread Cousin Stanley
> . > Yes and no. My own experience with Debian packages > is that with a standard > > apt-get install python2.5 > > an attempt to > import sqlite3 > > results in > ImportError: No module named _sqlite3 > No problems here with Debian Lenny All package

Re: (in memory) database

2008-08-31 Thread Roel Schroeven
Cameron Laird schreef: I now suspect that my 2.5 packaging has something to do with 64-bit builds; all my 32-bit Ubuntu servers have Python 2.5.2, while the 64-bit ones are at Python 2.5. Strange: my 64-bit Ubuntu 8.04 has Python 2.5.2, with working sqlite: [EMAIL PROTECTED] $ uname -a Linux

Re: (in memory) database

2008-08-31 Thread Paul Boddie
On 31 Aug, 21:29, [EMAIL PROTECTED] (Cameron Laird) wrote: > [Lots of output suggesting correct package configuration] > I'm certainly perplexed, and welcome suggestions. Maybe... which python I think Jean-Paul might be on to something with his response. Are we referring to the system-packag

Re: (in memory) database

2008-08-31 Thread Cameron Laird
In article <[EMAIL PROTECTED]>, Jean-Paul Calderone <[EMAIL PROTECTED]> wrote: >On Sun, 31 Aug 2008 18:05:08 +, Cameron Laird <[EMAIL PROTECTED]> wrote: >>In article <[EMAIL PROTECTED]>, >>Paul Boddie <[EMAIL PROTECTED]> wrote: >> [snip] >> >>Thanks for pursuing this, Paul. You have me curio

Re: (in memory) database

2008-08-31 Thread Cameron Laird
In article <[EMAIL PROTECTED]>, Paul Boddie <[EMAIL PROTECTED]> wrote: >On 31 Aug, 20:05, [EMAIL PROTECTED] (Cameron Laird) wrote: >> >> Let's take a definite example: I have a convenient >> Ubuntu 8.04.1 >> The content of /etc/apt/sources.list is >> debhttp://us.archive.ubuntu.com/ubuntu

Re: (in memory) database

2008-08-31 Thread Jean-Paul Calderone
On Sun, 31 Aug 2008 18:05:08 +, Cameron Laird <[EMAIL PROTECTED]> wrote: In article <[EMAIL PROTECTED]>, Paul Boddie <[EMAIL PROTECTED]> wrote: [snip] Thanks for pursuing this, Paul. You have me curious now. Let's take a definite example: I have a convenient Ubuntu 8.04.1 The content

Re: (in memory) database

2008-08-31 Thread Paul Boddie
On 31 Aug, 20:05, [EMAIL PROTECTED] (Cameron Laird) wrote: > > Let's take a definite example: I have a convenient > Ubuntu 8.04.1 > The content of /etc/apt/sources.list is > debhttp://us.archive.ubuntu.com/ubuntuhardy main restricted > debhttp://us.archive.ubuntu.com/ubuntuhardy-update

Re: (in memory) database

2008-08-31 Thread Cameron Laird
In article <[EMAIL PROTECTED]>, Paul Boddie <[EMAIL PROTECTED]> wrote: >On 31 Aug, 16:45, [EMAIL PROTECTED] (Cameron Laird) wrote: >> Yes and no. My own experience with Debian packages is that with a >> standard >> apt-get install python2.5 >> an attempt to >> import sqlite3 >> results in >>

Re: (in memory) database

2008-08-31 Thread Paul Boddie
On 31 Aug, 16:45, [EMAIL PROTECTED] (Cameron Laird) wrote: > Yes and no. My own experience with Debian packages is that with a > standard > apt-get install python2.5 > an attempt to > import sqlite3 > results in > ImportError: No module named _sqlite3 That's strange from the perspective of

Re: (in memory) database

2008-08-31 Thread Cameron Laird
In article <[EMAIL PROTECTED]>, Fredrik Lundh <[EMAIL PROTECTED]> wrote: >mark wrote: . . . >> Unfortunately I have only some knowledge of SQLite which is not an >> option here. > >why is sqlite not an option? it's is bundled

Re: (in memory) database

2008-08-31 Thread Fredrik Lundh
mark wrote: > I need to extract data from text files (~4 GB) on this data some > operations are performed like avg, max, min, group etc. The result is > formated and written in some other text files (some KB). you could probably do all that with data stream processing, but if you haven't worked

(in memory) database

2008-08-31 Thread mark
Hi there, I need to extract data from text files (~4 GB) on this data some operations are performed like avg, max, min, group etc. The result is formated and written in some other text files (some KB). I currently think about database tools might be suitable for this. I would just write the impor