Welcome to "This year in web-platform-tests"; possibly the lowest
frequency regular project update at Mozilla. In a break from tradition
the MoCo All Hands moving to January means that this is the first
edition to come from a normal office and not a hotel corridor somewhere
in North America.
This year has seen big improvements in our infrastructure, the addition
of some new features to improve parity with Gecko-specific test
harnesses, and a lot of improvements to our sync. We have also started
to see the overall project focus shift a little away from ensuring that
we have great infrastructure for writing cross-browser tests, to
ensuring that engineers are seeing interop problems and acting on them.
I expect to see more work along those lines in the forthcoming year.
== Gecko CI ==
* Added wpt support for GeckoView and enabled it as T-1 across all CI
platforms
* Added fission support and enabled it on Linux64-qr and Win10-qr
builds
* Added support for multiple test statuses so that we can mark
unstable tests directly rather than disabling them (special shoutout
to Outreachy intern Nikki Sharpley who did this work, as well as
adding support for annotating test expectations with multiple statuses
on import).
* Fixed the metadata update to produce more minimal if conditions and to
remove stale metadata.
* Started the work to remove duplicate test coverage across test
types, and so reduce CI load.
== Documentation ==
* Launched a revamped https://web-platform-tests.org with improved
documentation and full text search.
== Testability ==
* Added support for `TestRendered` event in reftests; analogous to
`MozReftestInvalidate` in Gecko reftests
* Implemented a crash test type, after Emilio found that running crash
tests cross-browser produced a large number of previously unknown
bugs.
* Turned on the http/2 support developed last year, now that all the
CI systems are using a new enough Python. Added a `.h2` flag for
test names to indicate that they are using http/2.
* Added support for fuzzy annotations in reftests, so that we are able
to ignore differences up to a specified tolerance in terms of pixel
values.
* Added a `PRECONDITION_FAILED​` status for tests and associated
`assert_precondition` function for cases where the tests depend on a
specific feature being implemented to work as intended.
* Added a `promise_setup` function to `testharness.js` to allow
sequencing asynchronous test setup steps before tests defined with
`promise_test`.
* Changed the reftest logic so that in any test all mismatch
conditions must be satisfied and at least one match condition must
be satisfied. This corresponds to the behaviour that many existing
tests were expecting.
* Required an opt-in for "single page" tests to prevent tests that
error early being incorrectly documented as single-page.
* Required Ahem to be supplied as a webfont rather than as a system
font, avoiding various problems with system fonts.
== Results Notification ==
* Created a repository
(https://github.com/web-platform-tests/wpt-metadata) holding links
between test results and browser bugs, and the ability to query it
* Implemented https://jgraham.github.io/wptdash/ a dashboard that by
default shows Gecko-only failures organised by bug component, and
allows filtering according to whether the result is linked to a
Gecko bug.
* Initial implementation of auto-filing triagable bugs for test changes
that show various kinds of more serious problems like Gecko-only
failures or regressions (e.g. changing from PASS to FAIL). We will be
rolling this out across components in the new year; please get in
touch if you'd like to be an early adopter.
* Added support for storing screenshots for failing reftests on
wpt.fyi and a reftest-analyzer-like UI for viewing them.
* Lots of new search operators in wpt.fi including the ability to
query by result status
* Launched https://wpt.live as a replacement for https://w3c-test.org
== Sync ==
* About 700 changes made by Gecko engineers were upstreamed from
mozilla-central to GitHub, and over 2100 changes were upstreamed
from Chromium, together making up a little over half of the total
~5000 PRs merged this year.
* Over half of Chromium CLs merged in Q3 that contained test changes
modified web-platform-tests (we don't yet have this data for Gecko).
* Lots of work to make the Gecko sync more reliable and performant;
sync latency is now tracked at https://arewewptyet.com/sync.html
Since the middle of the year this has remained under 1 week apart
from a period in October/November when both taskcluster and autoland
migrations caused disruption.
* Changed the integration branch for the sync from mozilla-inbound to
autoland
* Started work on Phabricator integration for earlier notification
when Gecko test changes are going to break upstream CI.
== GitHub Infrastructure ==
* Added daily runs of webkitgtk_minibrowser and Servo.
* Removed EdgeHTML-based Edge runs and added Chromium-based Edge.
* Added support for local GeckoView runs using the Android emulator​;
CI (and hence results on wpt.fyi) will be added once the relevant
machine types are available.
* Migrated the Taskcluster infra to the community deployment.
* Implemented a decision task to improve scheduling and avoid spinning
up unnecessary tasks in Taskcluster.
* Significant performance improvements for manifest generation.
== Meta ==
* Adopted an RFC process for proposing changes and collating feedback,
using https://github.com/web-platform-tests/rfcs
* Moved to an unmodified 3-clause BSD license from the W3C-variant
license that was previously used.
As usual all these improvements are the result of hard work not just
from Interop Engineering here at Mozilla, but also the Ecosystem Infra
team at Google and many others in the wider web-platform-tests
community. Huge thanks to everyone involved, and apologies for any
important items I forgot to include.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
