#28231: bulk_create: avoid iterating `objs` more than necessary when
`batch_size`
is provided
-------------------------------------+-------------------------------------
Reporter: Nir Izraeli | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Description changed by Nir Izraeli:
Old description:
> When `bulk_size` provided in `bulk_create`, a user might assume (As I
> myself have) that `objs` iterable will not be iterated more than
> `bulk_size` times at once, and that no more than roughly `bulk_size`
> Model objects reside in memory at any given time.
>
> When using `bulk_create` for relatively big sets of objects provided by a
> generator object, it would be prefered to avoid iterating over the entire
> generator object. Moreover, if not iterating over the generator object is
> deemed unnecessary or out-of-scope for django it would be prefered to
> make a comment on said behavior in documentation.
>
> I suggest two possible solutions:
> 1. Document this behavior (`bulk_create` converts passed `objs` iterator
> to a list), or
> 2. Avoid doing so when `bulk_size` is given (or default is other than
> `None`. i.e. sqlite).
>
> I did not research the possibility of avoiding the list conversion, but
> if that solution is accepted by the community I volunteer to investigate
> further and claim this ticket.
New description:
When `batch_size` provided in `bulk_create`, a user might assume (As I
myself have) that `objs` iterable will not be iterated more than
`batch_size` times at once, and that no more than roughly `batch_size`
Model objects reside in memory at any given time.
When using `bulk_create` for relatively big sets of objects provided by a
generator object, it would be prefered to avoid iterating over the entire
generator object. Moreover, if not iterating over the generator object is
deemed unnecessary or out-of-scope for django it would be prefered to make
a comment on said behavior in documentation.
I suggest two possible solutions:
1. Document this behavior (`bulk_create` converts passed `objs` iterator
to a list), or
2. Avoid doing so when `batch_size` is given (or default is other than
`None`. i.e. sqlite).
I did not research the possibility of avoiding the list conversion, but if
that solution is accepted by the community I volunteer to investigate
further and claim this ticket.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/28231#comment:2>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/064.b863b9087d3b3bd1d598ce4f8d567d09%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
