#31202: Bulk update suffers from poor performance with large numbers of models
and
columns
-------------------------------------+-------------------------------------
Reporter: Tom Forbes | Owner: Tom
Type: | Forbes
Cleanup/optimization | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Tom Forbes):
* owner: nobody => Tom Forbes
* status: new => assigned
Old description:
> A user has reported seeing extremely slow update times when using
> bulk_update. With the django-bulk-update package, which does not use the
> expressions API and constructs raw SQL directly, an update with 100,000
> objects and and 10 fields takes 24 seconds. With the built in bulk_update
> it takes 2 minutes and 24 seconds - 6x as slow.
>
> The user has provided a reproduction case here: https://github.com/mikicz
> /bulk-update-tests/blob/master/apps/something/models.py and
> https://github.com/mikicz/bulk-update-
> tests/blob/master/apps/something/test_bulk_update.py
>
> From an initial look at the profiling that has been provided
> (https://github.com/aykut/django-bulk-
> update/files/4060369/inbuilt_pyinstrument.txt) it seems a lot of overhead
> comes from building the SQL query rather than executing it - I think if
> we can improve the performance it could speed up other workloads.
>
> See: https://github.com/aykut/django-bulk-
> update/issues/75#issuecomment-576886385
New description:
A user has reported seeing extremely slow update times when using
bulk_update. With the django-bulk-update package, which does not use the
expressions API and constructs raw SQL directly (https://github.com/aykut
/django-bulk-update/blob/master/django_bulk_update/helper.py#L202), an
update with 100,000 objects and and 10 fields takes 24 seconds. With the
built in bulk_update it takes 2 minutes and 24 seconds - 6x as slow.
The user has provided a reproduction case here: https://github.com/mikicz
/bulk-update-tests/blob/master/apps/something/models.py and
https://github.com/mikicz/bulk-update-
tests/blob/master/apps/something/test_bulk_update.py
From an initial look at the profiling that has been provided
(https://github.com/aykut/django-bulk-
update/files/4060369/inbuilt_pyinstrument.txt) it seems a lot of overhead
comes from building the SQL query rather than executing it - I think if we
can improve the performance it could speed up other workloads.
See: https://github.com/aykut/django-bulk-
update/issues/75#issuecomment-576886385
--
--
Ticket URL: <https://code.djangoproject.com/ticket/31202#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/061.8e250b61bafbdfc0df0faf9e7a25b075%40djangoproject.com.
