#26400: QuerySet bulk_create method to handle generators to prevent loading all
objects in memory at once
-------------------------------------+-------------------------------------
Reporter: likeon | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: bulk_create | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by charettes):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
I'm not sure this is worth including into Django, is there a reason you
can't split your `bulk_create` calls into batches that fit into memory?
{{{#!python
from itertools import islice, chain
# Taken from https://code.activestate.com/recipes/303279-getting-items-in-
batches/
def split(iterable, size):
sourceiter = iter(iterable)
while True:
batchiter = islice(sourceiter, size)
yield chain([batchiter.next()], batchiter)
for batch in split(large_generator, 10000): # Adjust size to fit in
memory
MyModel.objects.bulk_create(batch)
}}}
You could even make this a manager method if required.
--
Ticket URL: <https://code.djangoproject.com/ticket/26400#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/064.7b4916e6d63ab270b1455b8cfdcd1fd3%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
