Hello, TBPL (https://tbpl.mozilla.org) is Mozilla’s primary tool for visualizing and analyzing automated test data. It was a huge step forward when we transitioned from Tinder Box to TBPL, and it has allowed us to push forward with new products and platforms. Many thanks to all the Mozillians who made great contributions to it!
TBPL was not designed to manage the quantity or breadth of data we are working with now. The Automation and Tools team reached a point where it often takes longer to put data into TBPL than it does to set up new automation on a test device, and it should be the other way around. There are many limitations we struggle with regularly. We’ve bolted a veritable cornocopia of ad-hoc features on to TBPL to get product out the door and solve problems fast. That’s been good for supporting many projects and releases, but over time, the technical debt has backed us up against the wall. We’ve put serious mileage on TBPL, and it has earned a long and blissful retirement! We need a data reporting and analysis system that’s more sustainable, that can scale with the diverse set of automation and testing requirements of Mozilla’s broad product base, and that can keep up with our constant fast pace. The replacement we’ve been working on is called Treeherder (https://treeherder.mozilla.org). Our initial goal was to reach full feature parity with TBPL, but with a scalable and extensible data model, RESTful web service, and UI. We are planning on transitioning to it within Q3. We’ve got lots of plans for useful bells and whistles in future releases, but the first step is reaching full feature parity with TBPL. We need to make sure sheriffs and developers can carry out business as usual. So here’s some of the fun stuff that’s in this first release. Some of it may not be immediately applicable to you, but it sets the stage for lots of future goodness: * Publicly available RESTful api that supports loading data from any build system using OAuth credentials. We decoupled the buildbot-isms so we can better support jenkins, taskcluster, and whatever else comes up in the future. Among other things, this will allow us to display results for on-device tests for B2G, something that's impossible with TBPL. * The data model binds the push log to all of the build and test data permanently. This allows us to provide test data in sync with the push log to downstream consumers. It also allows us to address questions regarding build/test platform combination trends over long timelines. Expect a number of new dashboards in the near future. There’s no longer real-time dependency in the UI on the https://hg.mozilla.org/(insert favorite repo/branch)/json-pushes web service. * Integrated Talos performance data. This is not quite done yet but it will land within Q3. We can visualize and annotate Talos data inline with other test results. We hope this will allow sheriffs and developers to make better use of performance data in general. * More comprehensive platform/test filtering all throughout the application. * More UI/UX scalability. We intend to add different top-level tabs to support a variety of different dashboards and views, in addition to the standard TBPL view of the build/test universe. * More descriptive semantics to classify failures and manage annotating build/test results. The Sheriffs have given it a thorough and greatly appreciated beating and we hope that developers and anyone else using TBPL will join in and help identify as many bugs as possible over the coming weeks. We still have some bugs to work through, but we’re getting very close. We will be giving a demo and update in the Monday morning project meeting on August 4th. Jeads Background Stuff ---------------- General background information and all of the treeherder bugs (thanks to edmorley!): https://wiki.mozilla.org/Auto-tools/Projects/Treeherder You can kick the tires here: https://treeherder.mozilla.org General documentation: https://treeherder-service.readthedocs.org Please file bugs here: https://bugzilla.mozilla.org/enter_bug.cgi?product=Tree+Management&component=Treeherder If you have questions or concerns drop us a line in #treeherder on IRC. If you have any interest in this stuff, patches are most welcome! Take a look at the installation docs to get started https://treeherder-service.readthedocs.org/en/latest/installation.html. There are three repositories associated with Treeherder. The code for the data ingestion etl, database, and web service can be found here: https://github.com/mozilla/treeherder-service The code for the user interface is here: https://github.com/mozilla/treeherder-ui The code for the python client that helps you get up and running to submit data to treeherder is here: https://github.com/mozilla/treeherder-client _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
