Congratulations on getting to this stage!
I would like to help dogfood this, and want to know if I can trust the
data parity of Treeherder and TBPL, or if that is something that you
would like testing on? IOW, should I keep them both open when watching
a tree / try push?
Thanks!
Ehsan
On 2014-07-22, 9:01 PM, Jonathan Eads wrote:
Hello,
TBPL (https://tbpl.mozilla.org) is Mozilla’s primary tool for visualizing and
analyzing automated test data. It was a huge step forward when we transitioned
from Tinder Box to TBPL, and it has allowed us to push forward with new
products and platforms. Many thanks to all the Mozillians who made great
contributions to it!
TBPL was not designed to manage the quantity or breadth of data we are working
with now. The Automation and Tools team reached a point where it often takes
longer to put data into TBPL than it does to set up new automation on a test
device, and it should be the other way around. There are many limitations we
struggle with regularly. We’ve bolted a veritable cornocopia of ad-hoc features
on to TBPL to get product out the door and solve problems fast. That’s been
good for supporting many projects and releases, but over time, the technical
debt has backed us up against the wall.
We’ve put serious mileage on TBPL, and it has earned a long and blissful
retirement!
We need a data reporting and analysis system that’s more sustainable, that can
scale with the diverse set of automation and testing requirements of Mozilla’s
broad product base, and that can keep up with our constant fast pace.
The replacement we’ve been working on is called Treeherder
(https://treeherder.mozilla.org). Our initial goal was to reach full feature
parity with TBPL, but with a scalable and extensible data model, RESTful web
service, and UI. We are planning on transitioning to it within Q3.
We’ve got lots of plans for useful bells and whistles in future releases, but
the first step is reaching full feature parity with TBPL. We need to make sure
sheriffs and developers can carry out business as usual.
So here’s some of the fun stuff that’s in this first release. Some of it may
not be immediately applicable to you, but it sets the stage for lots of future
goodness:
* Publicly available RESTful api that supports loading data from any build
system using OAuth credentials. We decoupled the buildbot-isms so we can better
support jenkins, taskcluster, and whatever else comes up in the future. Among
other things, this will allow us to display results for on-device tests for
B2G, something that's impossible with TBPL.
* The data model binds the push log to all of the build and test data
permanently. This allows us to provide test data in sync with the push log to
downstream consumers. It also allows us to address questions regarding
build/test platform combination trends over long timelines. Expect a number of
new dashboards in the near future. There’s no longer real-time dependency in
the UI on the https://hg.mozilla.org/(insert favorite repo/branch)/json-pushes
web service.
* Integrated Talos performance data. This is not quite done yet but it will
land within Q3. We can visualize and annotate Talos data inline with other test
results. We hope this will allow sheriffs and developers to make better use of
performance data in general.
* More comprehensive platform/test filtering all throughout the application.
* More UI/UX scalability. We intend to add different top-level tabs to support
a variety of different dashboards and views, in addition to the standard TBPL
view of the build/test universe.
* More descriptive semantics to classify failures and manage annotating
build/test results.
The Sheriffs have given it a thorough and greatly appreciated beating and we
hope that developers and anyone else using TBPL will join in and help identify
as many bugs as possible over the coming weeks. We still have some bugs to work
through, but we’re getting very close. We will be giving a demo and update in
the Monday morning project meeting on August 4th.
Jeads
Background Stuff
----------------
General background information and all of the treeherder bugs (thanks to
edmorley!):
https://wiki.mozilla.org/Auto-tools/Projects/Treeherder
You can kick the tires here:
https://treeherder.mozilla.org
General documentation:
https://treeherder-service.readthedocs.org
Please file bugs here:
https://bugzilla.mozilla.org/enter_bug.cgi?product=Tree+Management&component=Treeherder
If you have questions or concerns drop us a line in #treeherder on IRC.
If you have any interest in this stuff, patches are most welcome! Take a look
at the installation docs to get started
https://treeherder-service.readthedocs.org/en/latest/installation.html.
There are three repositories associated with Treeherder.
The code for the data ingestion etl, database, and web service can be found
here:
https://github.com/mozilla/treeherder-service
The code for the user interface is here:
https://github.com/mozilla/treeherder-ui
The code for the python client that helps you get up and running to submit data
to treeherder is here:
https://github.com/mozilla/treeherder-client
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
