I don't expect everyone I've CC'd to give thorough review (or any
review), I've mostly CC'd people who I think would be interested in this
work, or who's work flow I might be altered by it.
Piglit has struggled to cope with the growing number of tests that it
contains, especially with startup time. Piglit has always calculated
tests at runtime, which was not a problem when there were only a few
hundred or even thousand tests. Piglit now has roughly 55,000
OpenGL/OpenGL ES tests, which is a lot to calculate at start up. It also
means that piglit needs to keep a python object for each of those tests
in memory, which has sent the resident memory usage soaring. We've also
moved to automatic test discovery for glslparser, asmparser, and shader
tests, which is very convenient and reduces typing, but further
increases the amount of time spent starting up. This has even made
features which decrease runtime, like fast skipping, hurt startup
performance, making it a less than desirable tradeoff in some cases.
Even on a relatively fast machine with an nvme disk 15-20 seconds is not
an unheard of startup time. That might be okay to run 55,000 tests, but
not if you only need a dozen, such as when bisecting.
This series is my proposal to fix that, mainly by moving much of that
cost to build time. This series creates the infrastructure build XML
base profiles at build time, which are installed with piglit instead of
the python profiles. These profiles are lazily iterated over to ease
memory usage, test objects are created as they are run, and python can
garbage collect them as soon as they are done running. Along with that
any filters applied to profiles (like removing 80% of the vs_in shader
tests in quick) are done before the profile is serialized, and all fast
skipping information is collected at build time as well, and encoded in
the XML. All this means that start times are vastly reduced.
For example:
XML profiles
quick: 0.5
shader: 0.5
master
quick: 11.6
shader: 7.3
This series also implements some optimizations for running without
filters or test-lists, if you add a filter quick would take 2.5
seconds, because that is necessary to calculate the total number of
tests before starting.
To keep classic profiles like all, quick, quick_cl, gpu, cpu, and
llvmpipe working this series adds meta profiles, small XML snippets that
list other profiles. These can contain other meta profiles, xml
profiles, or python profiles. This means that for most uses cases your
existing command line will still work, `./piglit run quick out -c` will
still do exactly the same thing as before, just faster.
The XML generated is dumb, there is no encoding of options or logic. An
early version of this series did contain logic and options, but the
result was pretty terrible. It was very hard to read, and the code to
handle it was very complicated. I've chosen not to go down that path.
There are drawbacks, some things that relied on run time generation have
cannot be handled the same way, among them the "multi shader" concept,
where shader_runner consumes a directory of shader_tests at a time. This
was previously handled via a --process-isolation=false flag, now its
encoded into profiles, "shader_multi" and "quick_shader_multi"; there
was also an option to use glslparsertest with ES shaders and
ARB_ES_compatibility, that is now "glslparser_arb_compat". I haven't
added metaprofiles for these cases, although we certainly could (or you
can write your own, the schema is dead simple), so `./piglit run quick
out --process-isolation=false` is now `./piglit run quick_gl glslparser
quick_shader_multi out`.
I've run this through our CI extensively, and gotten green results out
of it across the board.
I know this is a big series, but piglit makes a lot of assumptions about the
test profiles being created at runtime, and we've had to changes those
assumptions.
Dylan Baker (35):
update git ignore for this series
test/piglit_test: add ROOT_DIR variable
framework/profile: Allow a group manager class to be overwritten
framework/test: Use getter for altering PiglitBaseTest Command
framework/test: expose required and excluded platforms
framework/profile: Add a __len__ method to TestProfile
framework: Use custom class for ASM parser tests
framework/test: add a test class for built-in constants
tests: use BuiltInConstantsClass
framework: use a class method for building test via parsing
framework: do the same for shader test
framework/test: Split multishader too
framework/test/piglit_test: make cl_concurrency always a boolean
framework/test: Add class for cl-program-tester
framework/test: Make shader paths relative
framework/test: use relative paths for GLSLParser tests
tests/all: Make asmparser tests path relative
framework/test: make BuiltInConstantTest files relative
framework/test: make CLProgramTester take relative paths
profile: Add support for loading xml based profiles
profile: allow forcing python or xml loading
framework/profile: Add support for meta profiles
py_modules: Add support for out of tree builds
tests/quick: fix filtering of vs_in shader tests
tests: use meta profiles
shader_tests: correctly generate xml during out of tree builds
tests/glsl_parser_test.py: fix is_skip for serialized profiles
fix glslparser test for out of tree builds
fix asmparser test serialization for out of tree builds
tests/cl.py: fix out of tree serialization
opengl.py: Remove exported gl extensions
Add script to serialize profiles to XML
tests: Add script to find all hand written test files
Generate xml for builtin profiles
profile: use gz to compress profiles
.gitignore | 4 +-
CMakeLists.txt | 10 +-
framework/profile.py | 203 +-
framework/test/glsl_parser_test.py | 60 +-
framework/test/piglit_test.py | 68 +-
framework/test/shader_test.py | 131 +-
tests/CMakeLists.no_api.txt | 89 +-
tests/CMakeLists.txt | 2 +-
tests/all.meta.xml | 6 +-
tests/all.py | 4999 +--------------
tests/cl.py | 71 +-
tests/cpu.meta.xml | 4 +-
tests/cpu.py | 31 +-
tests/find_static_tests.py | 74 +-
tests/glslparser.py | 63 +-
tests/gpu.meta.xml | 5 +-
tests/gpu.py | 18 +-
tests/llvmpipe.meta.xml | 5 +-
tests/llvmpipe.py | 33 +-
tests/llvmpipe_gl.py | 33 +-
tests/no_error.py | 17 +-
tests/opencl_foreign.py | 34 +-
tests/opengl.py | 4837 ++++++++++++++-
tests/py_modules/constants.py | 8 +-
tests/quick.meta.xml | 6 +-
tests/quick.py | 97 +-
tests/quick_cl.meta.xml | 5 +-
tests/quick_cl.py | 37 +-
tests/quick_gl.py | 78 +-
tests/quick_shader.py | 53 +-
tests/serializer.py | 151 +-
tests/shader.py | 71 +-
unittests/framework/test/test_glsl_parser_test.py | 61 +-
unittests/framework/test/test_shader_test.py | 37 +-
unittests/framework/test_profile.py | 5 +-
35 files changed, 6009 insertions(+), 5397 deletions(-)
create mode 100644 tests/CMakeLists.no_api.txt
create mode 100644 tests/all.meta.xml
delete mode 100644 tests/all.py
create mode 100644 tests/cpu.meta.xml
delete mode 100644 tests/cpu.py
create mode 100644 tests/find_static_tests.py
create mode 100644 tests/gpu.meta.xml
delete mode 100644 tests/gpu.py
create mode 100644 tests/llvmpipe.meta.xml
delete mode 100644 tests/llvmpipe.py
create mode 100644 tests/llvmpipe_gl.py
create mode 100644 tests/opencl_foreign.py
create mode 100644 tests/opengl.py
create mode 100644 tests/quick.meta.xml
delete mode 100644 tests/quick.py
create mode 100644 tests/quick_cl.meta.xml
delete mode 100644 tests/quick_cl.py
create mode 100644 tests/quick_gl.py
create mode 100644 tests/quick_shader.py
create mode 100644 tests/serializer.py
base-commit: 385fbbe2e73bf4f65f425101e8fa890ae5b52649
--
git-series 0.9.1
_______________________________________________
Piglit mailing list
Piglit@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/piglit
