Thanks Thomas :) On Tue, Feb 15, 2011 at 11:16 AM, Thomas Weholt <thomas.weh...@gmail.com>wrote:
> FYI: release version 0.5.0 of DSE yesterday. Source is available at > https://bitbucket.org/weholt/dse and pypi. > > Regards, > Thomas Weholt > > On Tue, Feb 15, 2011 at 10:10 AM, Sithembewena Lloyd Dube > <zebr...@gmail.com> wrote: > > Hi Everyone, > > > > Coming back weeks later, to say thank you for your contributions. I got > > quite sidetracked from this task, but I should report back on this soon. > > > > Thank you! > > > > On Wed, Jan 19, 2011 at 10:05 AM, Thomas Weholt <thomas.weh...@gmail.com > > > > wrote: > >> > >> Bulk inserts are the way the go if you can. When inserting a bunch of > >> data, avoid using the django orm. Do it in plain SQL. The overhead of > >> creating django orm model instances is way too expensive. Alltough it > >> may not be bulk insert the sense Nick mentioned above I wrote DSE [ > >> http://pypi.python.org/pypi/dse/0.3.1 ] for that exact purpose, to > >> insert or update a bunch of data using plain SQL, but it hasn`t been > >> tested much so don`t use it in production. The thing it solves, can be > >> done in simple SQL; > >> > >> Use cursor.executemany("<prepared insert statement>", <list of tuples > >> with params>). > >> > >> DSE takes care of creating SQL-statements for insert and updates based > >> on your models, handles any default values you might have defined in > >> your model, caches lists of params and executes cursor.executemany > >> when the list of cached items reaches a specified limit. My experience > >> is that the performance gain of this solution or a similar one is > >> huge. Using cursor.executemany might be what Nick meant by bulk > >> insert, but I think different DB backends handles it differently. I > >> don`t know. Anyway, I've inserted many thousands of records using DSE > >> and it takes a fraction of the time when compared to doing it with the > >> orm. > >> > >> > >> NB! DSE is a proof-of-concept project more than anything else. It > >> needs a good re-write, extensive testing and docs, but it might be > >> helpful. > >> > >> Thomas > >> > >> > >> > >> > >> > >> On Wed, Jan 19, 2011 at 2:35 AM, Nick Arnett <nick.arn...@gmail.com> > >> wrote: > >> > > >> > > >> > On Tue, Jan 18, 2011 at 12:04 PM, Sithembewena Lloyd Dube > >> > <zebr...@gmail.com> wrote: > >> >> > >> >> Hi all, > >> >> > >> >> I am building a search app. that will query an API. The app. will > also > >> >> store search terms in a very simple table structure. > >> >> > >> >> Big question: if the app. eventually hit 10 million searches and I > was > >> >> storing every single search term, would the table hold or would I run > >> >> into > >> >> issues? > >> > > >> > As someone else said, 10 million records is no big deal for MySQL, in > >> > principle. > >> > However, you probably would do better to avoid all the overhead of a > >> > database transaction for storing each of these. I'm going to assume > >> > that > >> > there will be duplicates, especially if you normalize the queries. It > >> > would > >> > make a lot more sense to log the queries into a text file, which has > >> > extremely low overhead. Then you'd periodically process the log > files, > >> > normalizing and eliminating duplicates, producing a bulk insert to > load > >> > into > >> > the database. Bulk inserts will be FAR more efficient than using > >> > Django. > >> > Nick > >> > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups > >> > "Django users" group. > >> > To post to this group, send email to django-users@googlegroups.com. > >> > To unsubscribe from this group, send email to > >> > django-users+unsubscr...@googlegroups.com. > >> > For more options, visit this group at > >> > http://groups.google.com/group/django-users?hl=en. > >> > > >> > >> > >> > >> -- > >> Mvh/Best regards, > >> Thomas Weholt > >> http://www.weholt.org > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Django users" group. > >> To post to this group, send email to django-users@googlegroups.com. > >> To unsubscribe from this group, send email to > >> django-users+unsubscr...@googlegroups.com. > >> For more options, visit this group at > >> http://groups.google.com/group/django-users?hl=en. > >> > > > > > > > > -- > > Regards, > > Sithembewena Lloyd Dube > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Django users" group. > > To post to this group, send email to django-users@googlegroups.com. > > To unsubscribe from this group, send email to > > django-users+unsubscr...@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/django-users?hl=en. > > > > > > -- > Mvh/Best regards, > Thomas Weholt > http://www.weholt.org > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > > -- Regards, Sithembewena Lloyd Dube -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
