Thank you very much for this. I'll catch up with those threads and read more about DSE, which looks really interesting work.
regards George On Wednesday, 2 January 2013 11:50:54 UTC, Cal Leeming [Simplicity Media Ltd] wrote: > > Hi George, > > This is one area I spent quite a lot of time in personally, see; > > https://groups.google.com/forum/?fromgroups=#!msg/django-users/iRhWD0FtW8k/0KAMF3ub-ZYJ > https://groups.google.com/forum/#!topic/django-users/hgLrwMoFLII > > Bulk operations using the ORM isn't always the right thing to do - and it > entirely depends on what you consider bulk and acceptable performance. > > You might want to look at the source code for this, to see how they handle > bulk operations (they implemented the same bulk_update approach mentioned > in the above threads) > http://pypi.python.org/pypi/dse > > Although bypassing the ORM might feel wrong at first, sometimes it is > completely acceptable - you just need to make sure you don't abuse/misuse > it unnecessarily. > > Cal > > On Wed, Jan 2, 2013 at 11:29 AM, George Lund <gl...@mintel.com<javascript:> > > wrote: > >> I'm trying to bulk-delete several million rows from my database. >> >> The docs for Django >> 1.3<https://docs.djangoproject.com/en/1.3/topics/db/queries/#deleting-objects>say >> "this will, whenever possible, be executed purely in SQL". A pure-SQL >> delete is what I want in this case, so that's fine. >> >> However, the code is never getting as far as running any SQL. >> >> Interrupting the script shows that the delete method on the QuerySet is >> trying to use a "Collector" to construct model instances for each row I'm >> trying to delete. This is going to take too long (and may in fact consume >> all the memory available) -- I don't think it's practical to wait in this >> case. (I've tried waiting over half an hour!) >> >> (I'm looking at django.db.models.query.QuerySet.delete and >> django.db.models.deletion.Collector.collect >> / Collector.add.) >> >> What's the point in doing the delete "purely in SQL" if all of the >> objects are getting constructed anyway? Why do they need to be "collected" >> before the SQL DELETE is run? The model instance in this case has no >> child rows, to which the delete might need to be cascaded. >> >> Meanwhile I can construct the SQL by hand easily enough, but I feel this >> isn't doing things the right way. >> >> thanks for any help >> >> George >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django users" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/django-users/-/W4LqKzcnlaYJ. >> To post to this group, send email to django...@googlegroups.com<javascript:> >> . >> To unsubscribe from this group, send email to >> django-users...@googlegroups.com <javascript:>. >> For more options, visit this group at >> http://groups.google.com/group/django-users?hl=en. >> > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/q1ZxexzIdX8J. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
