#30563: django.forms.widgets.Media.__add__ Performance Issue
-------------------------------------+-------------------------------------
Reporter: didorothy | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Forms | Version: master
Severity: Normal | Keywords: performance,
Triage Stage: | unintended consequences
Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
While working with another project that make extensive use of
`django.forms.widgets.Media` I discovered that the fix for ticket #30153
has unintended consequences on the performance of `Media.__add__`. If the
number of Media objects added grows beyond a certain point (not sure it
may be machine specific) then the performance of all subsequent
`Media.__add__` calls becomes terrible.
This was causing page load times of several minutes on my development
machine. I agree that you need to delay as long as possible as #30153
intends but it seems that there probably should be an upper bound on this
so that performance does not suddenly decrease.
Here is some sample code that can reproduce the issue:
{{{
from django.forms import Media
import datetime
def create_media(MediaClass):
'''Creates a simple Media object with only one or two items.'''
return MediaClass(css={'all': ['main.css']}, js=['main.js'])
start = datetime.datetime.now()
media = create_media(Media)
for i in range(100000):
media = media + create_media(Media)
print('100000 additions took: %s' % (datetime.datetime.now() - start))
}}}
On my machine several runs of this code result in times between 1:35 -
1:44 (eg. 1 minute 35 seconds to 1 minute 44 seconds). However, taking
away one zero from the number of media objects runs in under a second.
Near as I can tell this has to do with the memory used to store these
arrays and when it gets past a certain point the performance is awful.
Since I am not sure if it is machine specific, for reference my machine is
a i7-8700 with 64 GB RAM.
Here is a sample that has a modified Media class that does not have theses
issues:
{{{
from django.forms import Media
import datetime
def create_media(MediaClass):
'''Creates a simple Media object with only one or two items.'''
return MediaClass(css={'all': ['main.css']}, js=['main.js'])
class CustomMedia(Media):
def __add__(self, other):
combined = CustomMedia()
if len(self._css_lists) + len(other._css_lists) > 1000:
combined._css_lists = [self._css, other._css]
else:
combined._css_lists = self._css_lists + other._css_lists
if len(self._js_lists) + len(other._js_lists) > 1000:
combined._js_lists = [self._js, other._js]
else:
combined._js_lists = self._js_lists + other._js_lists
return combined
start = datetime.datetime.now()
media = create_media(CustomMedia)
for i in range(100000):
media = media + create_media(CustomMedia)
print('100000 additions took: %s' % (datetime.datetime.now() - start))
}}}
With this change it again runs in under a second. If I increase the number
of loops the performance seems to change at a much more expected level.
I set an upper limit on the length allowed before a merge occurs. I'm not
sure if the number of additions allowed should be a setting that can be
adjusted to meet specific needs or if something that is just "reasonably"
high is sufficient. It does appear that limiting the total number of items
in the list to about 1000 works best on my machine. I'm also not sure that
this is the best solution.
Thanks for your consideration.
--
Ticket URL: <https://code.djangoproject.com/ticket/30563>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/052.412a1c59baea95a7aa46e980197a2c1b%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
