:: bugzilla and performance, 2014
http://globau.wordpress.com/2014/07/04/bugzilla-and-performance-2014/
bugzilla is a large legacy perl application initially developed at the
birth of the world wide web. it has been refactored significantly in the
15 years since its release; however, some design decisions made at the
start of the project carry significant technical debt in today's modern web.
while there have been a large number of micro-optimisations over the
course of bugzilla's life (generally as a result of performance
profiling), there are limits to the benefits these sorts of
optimisations can provide.
this year has seen a lot of focus on improving bugzilla's performance
within mozilla, centred around the time it takes to display a bug to
authenticated users.
tl;dr bugzilla is faster
:: memcached
the design of a modern large web application is generally centred around
caches, between the business logic layer and the data sources, as well
as between the users and the user interface. while bugzilla has been
refactored to use objects, database abstraction, templates, etc, it had
zero caching capabilities. this coupled with completely stateless
processing of each request meant that every byte of html returned to the
user was regenerated from scratch, starting with a new connection to the
database.
towards the end of 2013 i worked on implementing a memcached framework
into bugzilla [bug 237498].
retrofitting a caching mechanism into a large extendible framework
proved to be a significant challenge. bugzilla provides the ability for
developers to extend bugzilla's functionality via extensions, including
but not limited to adding new fields, user interface, or process. the
extensions system conflicts with caching as it's possible for an
extension to conditionally alter an object in ways that would render it
impossible to cache (eg. add a new field only if the current user is a
member of a particular group).
some compromises had to be made. instead of caching fully constructed
objects, the cache sits between the object's constructor and the
database. we avoid a trip to the database, but still have to construct
objects from that data (which allows extensions to modify the object
during construction).
code which updated the database directly instead of using bugzilla's
objects had to be found and rewritten to use the objects or updated to
manually clear the cache entries. extra care had to be taken as
returning stale data could silently result in data loss. to placate
concerns that these errors would be impossible to detect, the caller of
an object's constructor must pass in a parameter to "opt-in" to caching.
in 2014 i built upon the memcached framework to support most of
bugzilla's objects [bug 956233], with the "bugs" object being the only
notable exception. memcached also caches bugzilla's configuration
parameters (classifications, products, components, groups, flags, ...)
[bug 987032]. although caching the "bugs" object would be beneficial
given its central role in all things bugzilla, it is highly likely that
enabling this by default would break bugzilla extensions and
customisations as a high proportion of them update the database directly
instead of using bugzilla's object layer. this would manifest as data
which is silently stale, making undetectable dataloss a high probability.
memcached support will be released with bugzilla 5.0, but has been live
on bugzilla.mozilla.org (bmo) since february 2014.
:: instrumentation
while profilling tools such as nytprof have often been pointed at
bugzilla, bmo's high number of concurrent requests and usage patterns
time and time again resulted in performance optimisations performing
worse than expected once deployed to production.
we took the decision to deploy instrumentation code into bugzilla
itself, reporting on each http request, database query, and template
execution. as bugzilla is written in perl, support was absent for
off-the-shelf instrumentation tools such as new relic, so we had to roll
our own data collection and reporting system [bug 956230].
the collector wraps specific Bugzilla::DB, Bugzilla::Memcached and
Bugzilla::Template calls via subclassing, then reports data to an
elasticsearch cluster. currently all reporting solutions are ad-hoc and
involve running of scripts which identify the most costly database
queries and templates.
this data identified areas of bugzilla which require optimisation or
caching. examples include the optimisation of a single query which
shaved 200ms (~15%) off page load times for most users [bug 993894],
caching of all queries which read an entire table [part of bug 987032],
and caching of user groups [bugs 993939] and settings [bug 993926].
we continue to revisit the instrumentation reports in order to guide
further improvements.
:: stylesheet concatenation and minification
improving the performance of a web applications isn't limited to
focusing on server-side execution speed.
due to bugzilla's extensions we ended up in a situation where bugzilla
was serving multiple small css files - on bmo we loaded 17 stylesheets
as part of show_bug in comparison with 5 for an installation of bugzilla
without any extensions installed.
similar to the issue encountered with memcached, extensions have
complete control with regards to optionally loading stylesheets, which
means any css concatenation and minification solution needed to be
implemented at run-time.
[bug 977969] does exactly that - the template passes an array of
stylesheets to load to bugzilla's global header, where a hash of the
array is used to find a unified stylesheet. simple minification is
performed which dropped the stylesheet size from 54kb to 43kb on
show_bug on bmo.
stylesheet concatenation and minification support will be released with
bugzilla 5.0, and has been live on bugzilla.mozilla.org since may 2014.
:: database
in order to address performance issues caused by bugzilla's use of the
myisam table type, in march our DBAs upgraded our database cluster to
mysql version 5.6. this was the result of analysis by the DBAs into
replication and stability issues around our myisam table.
as mysql 5.6 adds support for fulltext indexing to its innodb table
type, bugzilla was able to switch away from myisam. this immediately
fixed the database replication issues, and provided a noticeable
performance boost to searches involving comments.
:: looking forward
the next large project is to update bugzilla so the bug object can use
memcached on an unmodified installation without any non-default
extensions enabled. for reasons previously covered it's unlikely we'll
ship a version of bugzilla with this enabled by default, however this
will allow sites to audit their own code (if any) and enable caching of
the bug object if required.
we periodically enable instrumentation and use it to identify the next
set of targets for optimisation. this will continue for the foreseeable
future.
we also plan on to build on the css concatenation and minification work
to provide the same benefits to javascript files.
--
byron jones - :glob - bugzilla.mozilla.org team -
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
