Re: Advice on using model to bulk/batch load database

2006-03-23 Thread DavidA
My motivation for using the Django ORM layer was simply to reuse the logic that parses text values to native values (html2python) and any column name mapping that goes on, both of which are managed by the *Field members. I know that's not a huge win, but since I'm at an early point in the project

Re: Advice on using model to bulk/batch load database

2006-03-23 Thread Adrian Holovaty
On 3/23/06, DavidA <[EMAIL PROTECTED]> wrote: > Occasionally I will modify my schema, drop the old tables and reload > all the data from the "cached" files. Over time that could easily be > millions of rows and some optimization of my technique will be in > order. But for now, the simple approach

Re: Advice on using model to bulk/batch load database

2006-03-23 Thread DavidA
Eric, The typical mode of operation will be to incrementally load new data each night - about 10 files each containing dozens to hundreds of rows. Maybe only a couple will be in the thousands category. But I've also created my scripts to work in two steps: 1) download the data from various source

Re: Advice on using model to bulk/batch load database

2006-03-23 Thread Eric Walstad
DavidA wrote: > I am prepopulating my database with data from a number of flat files. > I've written a small script to do this using my model class. While this > works, there are a couple of kludges I made to get it to work and I was > hoping someone could advise my on a better way to do this. H

Re: Advice on using model to bulk/batch load database

2006-03-23 Thread DavidA
> BTW, if you do this repeatedly and format of those flat files is not a > requirement it is possible to have all initial data in a form of SQL > script. If you place it in /sql/.sql then Django will > include it in 'sqlall' command right after DB creation. While I'm not in control of the file fo

Re: Advice on using model to bulk/batch load database

2006-03-23 Thread DavidA
> You can set the environment variable at the top of your scripts by > importing the Python os module, just like you can import sys and set > sys.path. Thanks, I added this and it works fine: os.environ['DJANGO_SETTINGS_MODULE'] = 'data.settings' > The Python datetime and time modules are huge

Re: Advice on using model to bulk/batch load database

2006-03-22 Thread Ivan Sagalaev
DavidA wrote: >I am prepopulating my database with data from a number of flat files. > > BTW, if you do this repeatedly and format of those flat files is not a requirement it is possible to have all initial data in a form of SQL script. If you place it in /sql/.sql then Django will include i

Re: Advice on using model to bulk/batch load database

2006-03-22 Thread Max Battcher
DavidA wrote: > First, to use the model from a script I had to set the > DJANGO_SETTINGS_MODULE environment variable and add the base directory > to sys.path. I've seen this before so I guess this is just the way it > is but it would be nice not to have dependencies on environment > variables. Is