FYI: release version 0.5.0 of DSE yesterday. Source is available at https://bitbucket.org/weholt/dse and pypi.
Regards, Thomas Weholt On Tue, Feb 15, 2011 at 10:10 AM, Sithembewena Lloyd Dube <zebr...@gmail.com> wrote: > Hi Everyone, > > Coming back weeks later, to say thank you for your contributions. I got > quite sidetracked from this task, but I should report back on this soon. > > Thank you! > > On Wed, Jan 19, 2011 at 10:05 AM, Thomas Weholt <thomas.weh...@gmail.com> > wrote: >> >> Bulk inserts are the way the go if you can. When inserting a bunch of >> data, avoid using the django orm. Do it in plain SQL. The overhead of >> creating django orm model instances is way too expensive. Alltough it >> may not be bulk insert the sense Nick mentioned above I wrote DSE [ >> http://pypi.python.org/pypi/dse/0.3.1 ] for that exact purpose, to >> insert or update a bunch of data using plain SQL, but it hasn`t been >> tested much so don`t use it in production. The thing it solves, can be >> done in simple SQL; >> >> Use cursor.executemany("<prepared insert statement>", <list of tuples >> with params>). >> >> DSE takes care of creating SQL-statements for insert and updates based >> on your models, handles any default values you might have defined in >> your model, caches lists of params and executes cursor.executemany >> when the list of cached items reaches a specified limit. My experience >> is that the performance gain of this solution or a similar one is >> huge. Using cursor.executemany might be what Nick meant by bulk >> insert, but I think different DB backends handles it differently. I >> don`t know. Anyway, I've inserted many thousands of records using DSE >> and it takes a fraction of the time when compared to doing it with the >> orm. >> >> >> NB! DSE is a proof-of-concept project more than anything else. It >> needs a good re-write, extensive testing and docs, but it might be >> helpful. >> >> Thomas >> >> >> >> >> >> On Wed, Jan 19, 2011 at 2:35 AM, Nick Arnett <nick.arn...@gmail.com> >> wrote: >> > >> > >> > On Tue, Jan 18, 2011 at 12:04 PM, Sithembewena Lloyd Dube >> > <zebr...@gmail.com> wrote: >> >> >> >> Hi all, >> >> >> >> I am building a search app. that will query an API. The app. will also >> >> store search terms in a very simple table structure. >> >> >> >> Big question: if the app. eventually hit 10 million searches and I was >> >> storing every single search term, would the table hold or would I run >> >> into >> >> issues? >> > >> > As someone else said, 10 million records is no big deal for MySQL, in >> > principle. >> > However, you probably would do better to avoid all the overhead of a >> > database transaction for storing each of these. I'm going to assume >> > that >> > there will be duplicates, especially if you normalize the queries. It >> > would >> > make a lot more sense to log the queries into a text file, which has >> > extremely low overhead. Then you'd periodically process the log files, >> > normalizing and eliminating duplicates, producing a bulk insert to load >> > into >> > the database. Bulk inserts will be FAR more efficient than using >> > Django. >> > Nick >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "Django users" group. >> > To post to this group, send email to django-users@googlegroups.com. >> > To unsubscribe from this group, send email to >> > django-users+unsubscr...@googlegroups.com. >> > For more options, visit this group at >> > http://groups.google.com/group/django-users?hl=en. >> > >> >> >> >> -- >> Mvh/Best regards, >> Thomas Weholt >> http://www.weholt.org >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django users" group. >> To post to this group, send email to django-users@googlegroups.com. >> To unsubscribe from this group, send email to >> django-users+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/django-users?hl=en. >> > > > > -- > Regards, > Sithembewena Lloyd Dube > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > -- Mvh/Best regards, Thomas Weholt http://www.weholt.org -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
