Preston - I'll send the template to you directly, I'm not sure how useful
it will be so I don't want to spend time checking if it's fine for public
consumption unnecessarily.
The test template we were using to test the performance was a simple:
{% for item in item_list %}{% include "item.html" %}{% endfor %}
Where "item.html" is the complex one (for a real world example, it's the
"cards" you can see in the chart on https://www.mixcloud.com/tag/house/)
and item_list is a list of 200 items (with all of the database queries done
in the view). Averaging runs after the first run (which would include the
compile) it was taking close to 1s per render. With the non-breaking
changes (i.e. not commenting out necessary engine or template code) I
managed to get it down to just under 0.9s. In both cases there was an
overhead of around 0.2s (render time if I ran with the entire template
commented out). Running the test with pypy (after warming it up) the render
time was 0.5s. Sorry I don't have more exact numbers - this is just the
headline notes I took away from the task.
As I removed chunks of template or engine I just found that anything I
removed made a couple of percent of difference. The first thing I tried to
optimise was the include tag and it only made a small difference - I even
tried merging the "for", "include" and "with" tags into a new tag to avoid
the 2 unnecessary contexts created by the two inner tags - my rough memory
was that made a 3-4% difference in this particular case and we decided not
to use it due to the added complexity of our code.
Most of our page render times on www.mixcloud.com are proportional to the
number of cards we have on the page.
Sam
On Thu, 12 Mar 2015 at 18:07 Preston Timmons <[email protected]>
wrote:
> I've done a couple of days of investigation into template performance
>> recently trying to speed up our site and my main takeaway was that there
>> was no silver bullet - no particular node taking up all of the time. I was
>> mostly trying to optimise a particularly complicated template we render a
>> lot in a loop so I was modifying the template, Django code and making
>> custom tags to see what would make a difference rather than trying to
>> decipher profiles. We use the cached template loader so compile time wasn't
>> really considered.
>>
>
> Thanks, Sam. That's helpful information.
>
> I'd be interested to know the template you used, or at least one
> representative of the template you used. Templates that highlight
> real-world pain points would help to benchmark the things that matter.
>
> Did you apply all the internal Django changes you mentioned cumulatively?
> If so, what was your overall rendering time improvement? How did the
> speedups from changing nodes compare to the speedups from modifying the
> engine? It sounds like the engine changes were a factor because of multiple
> calls to the include tag.
>
> While the changes you mentioned don't sound like ones that could go into
> Django, they do shed light on what can potentially be done or not for
> optimization.
>
> Preston
>
>
>
>
>> We tried the following things and none of them made more than a couple of
>> percent of difference each:
>>
>> - we made a cut down {% url %} tag that just does what we need - the
>> built in tag can handle a lot more at a performance cost
>> - I grouped {% with %} statements together - ideally grouping into {%
>> include ... a=b c=d %} - to avoid extra layers of context
>> - I ditched TextNodes that just contained whitespace (both by removing
>> the whitespace and by automatically removing the Nodes on compile) - it's
>> easy for these to build up when you have code like the following and every
>> extra node slows things down a bit (whitespace is sometimes meaningful so
>> we would have only implemented this for particular bits of code)
>>
>> {% if whatever %}
>> {{ my_var }}
>> {% endif %}
>>
>> - I also tried commenting out (or replacing with "return '' ") chunks of
>> the template engine code and our template and it just seemed that the more
>> I commented out, the faster it ran - no particular jumps in speed, just a
>> gradual change as more was removed. Escaping was one of the first things I
>> commented out and it made a surprisingly small amount of difference.
>>
>> I'd be very happy to be proven wrong but thought it was worth sharing my
>> findings since - particularly in the context of Preston's suggestion that
>> we might find a bottleneck - I don't think there is one particular
>> bottleneck.
>>
>> Our "solution" for now has been to speed up the processors in our servers
>> and investigate switching to pypy - we'll probably be looking at Jinja2
>> once we upgrade to 1.8 as well.
>>
>> Sam
>>
>> On Thu, 12 Mar 2015 at 15:57 Preston Timmons <[email protected]>
>> wrote:
>>
> After a while I believe layers and layers of caution have accrued, and
>>>> nobody is sure any more where these have overlapped excessively.
>>>>
>>>
>>> Do you have examples of which layers these are?
>>>
>>> Escaping seems to happen in Variable, VariableNode, FilterExpression,
>>> and render_value_in_context. I don't see a lot of work being done twice
>>> there.
>>>
>>> If you think the escape implementation is slow, though, it wouldn't be
>>> hard to simply remove that and benchmark with no escape code running. That
>>> would at least reveal the theoretical limit to which the escape code could
>>> be improved.
>>>
>>> On another note, in my benchmark I see the difference to render a
>>> somewhat complex template between Django and Jinja2 at 8-9 times.
>>>
>>> From one run, just grabbing the minimum time to render a template:
>>>
>>> Django: 9.08e-05
>>> Jinja2: 1.38e-05
>>>
>>> The big difference here is because Django uses a recursive node-based
>>> renderer, whereas Jinja2 just translates the template into Python. That
>>> means a lot less overhead when rendering happens.
>>>
>>> Optimizing Django rendering means either:
>>>
>>> 1) Identifying the nodes which are a bottleneck and reducing the work
>>> they do
>>> 2) Replacing node rendering with something that's faster.
>>>
>>> Option 2 probably has the biggest opportunity for gain, but it would
>>> require some real creativity to maintain backwards-compatibility with
>>> existing tags.
>>>
>>> Preston
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>>
>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to [email protected].
>>> To post to this group, send email to [email protected].
>>
>>
>>> Visit this group at http://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/django-developers/957b9840-b537-4861-9390-
>>> e7b6f44dd72c%40googlegroups.com
>>> <https://groups.google.com/d/msgid/django-developers/957b9840-b537-4861-9390-e7b6f44dd72c%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/1b2d01a2-7660-42e5-a860-5781ad13b264%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/1b2d01a2-7660-42e5-a860-5781ad13b264%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CAMyx3A3yOi%3Dv_tNZt%3DaNh7SHeOhRs-mW7PA0RdMihMvAtRCArg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
