Hello everyone,
My name is Jeb, and I want to share information on the MariaDB Foundation’s
Test Automation Framework, TAF.
Additionally, I'm asking for contributors who want to help shape the next
stages of this framework.
TAF is the unified, open source, and reproducible system for performance
and performance change testing across database systems.
The purpose is simple. Provide a consistent, transparent, and repeatable
way to measure how database performance behaves and how it changes across
versions, branches, and individual commits.
TAF already supports multiple engines.
*Today TAF supports:*
- MariaDB
- MySQL
This allows direct comparison of behavior, performance movement, and change
patterns across both engines using the same workloads, the same hardware,
and the same methodology.
There are active plans to extend TAF to include other database systems like
PostgreSQL.
The long term goal is a unified, open source framework for performance and
performance change testing across many database engines.
TAF supports a full spectrum of real world testing, with more test suites
planned.
*Currently supported test suites include:*
- *Sysbench lua *
- Focused OLTP micro benchmarks and low level behavior. Ideal for
isolating specific performance changes and validating small
scale movement.
- *HammerDB TPROC C *
- A full OLTP transactional workload that exercises concurrency,
contention, and end to end transactional behavior.
- *HammerDB TPROC H *
- An analytical and decision support workload that stresses large
scans, joins, and complex query behavior.
This combination gives both micro level visibility and full end to end
system behavior. It allows clear identification of performance changes,
validation of improvements, and a consistent view of how the server evolves
over time.
TAF is fully operational today and actively expanding. It automates
environment setup, executes workloads, captures results, and creates
comparison reports from plugins.
Current work in progress includes sysbench patches from contributors and a
backend results database with results parser under construction.
*Recent contributions*
TAF has already started receiving community contributions. In the last week
we merged new sysbench tests, updated configuration handling, and fixed
issues in the oltp scan path. Additional fixes and improvements are pending
review.
This early activity is encouraging and shows that TAF is already useful to
contributors who want to improve performance testing across engines.
*TAF was explained in detail in a recent interview.*
-
https://www.youtube.com/watch?v=3_ZgUtCjSgc&list=PL8DHenLlKmXdaXEh5oGG6HWrmEUsmx_Js
*TAF home and source code*
- https://github.com/MariaDB/TAF
*Important note on contributions*
TAF is fully open source, and every contribution is carried forward in the
commit history. Nothing is hidden or proprietary. Contributor credit
remains attached to the work and moves forward with the project.
*Call for contributors*
We are now looking for contributors in two specific areas.
- *PostgreSQL TAF Database Plugin*
TAF needs a PostgreSQL DB plugin. This is a natural next step for the
framework and will allow cross engine performance change testing across
MariaDB, MySQL, and PostgreSQL using the same workloads and the same
methodology.
Anyone with PostgreSQL experience, especially around setup, configuration,
or workload automation, would be very welcome.
- *TAF Results Frontend*
TAF will be needing a web frontend for the TAF results backend that is
currently being created and tested.
The backend will store run data, metrics, and performance change
information. A frontend will make this visible, explorable, and useful for
developers, contributors, and anyone tracking performance movement across
commits or releases.
If you are interested in contributing to either of these areas, or if you
have ideas for additional test suites, workload coverage, or reporting
improvements, please let us know.
TAF is open source and community contributions will directly strengthen the
performance story for MariaDB and the broader database ecosystem.
The more real world scenarios we capture, the stronger and more transparent
the ecosystem becomes.
Best,
/Jeb
_______________________________________________
discuss mailing list -- [email protected]
To unsubscribe send an email to [email protected]
