Quoting Dylan Baker (2018-04-04 15:26:48) > I don't expect everyone I've CC'd to give thorough review (or any > review), I've mostly CC'd people who I think would be interested in this > work, or who's work flow I might be altered by it. > > Piglit has struggled to cope with the growing number of tests that it > contains, especially with startup time. Piglit has always calculated > tests at runtime, which was not a problem when there were only a few > hundred or even thousand tests. Piglit now has roughly 55,000 > OpenGL/OpenGL ES tests, which is a lot to calculate at start up. It also > means that piglit needs to keep a python object for each of those tests > in memory, which has sent the resident memory usage soaring. We've also > moved to automatic test discovery for glslparser, asmparser, and shader > tests, which is very convenient and reduces typing, but further > increases the amount of time spent starting up. This has even made > features which decrease runtime, like fast skipping, hurt startup > performance, making it a less than desirable tradeoff in some cases. > Even on a relatively fast machine with an nvme disk 15-20 seconds is not > an unheard of startup time. That might be okay to run 55,000 tests, but > not if you only need a dozen, such as when bisecting. > > This series is my proposal to fix that, mainly by moving much of that > cost to build time. This series creates the infrastructure build XML > base profiles at build time, which are installed with piglit instead of > the python profiles. These profiles are lazily iterated over to ease > memory usage, test objects are created as they are run, and python can > garbage collect them as soon as they are done running. Along with that > any filters applied to profiles (like removing 80% of the vs_in shader > tests in quick) are done before the profile is serialized, and all fast > skipping information is collected at build time as well, and encoded in > the XML. All this means that start times are vastly reduced. > > For example: > XML profiles > quick: 0.5 > shader: 0.5 > > master > quick: 11.6 > shader: 7.3 > > This series also implements some optimizations for running without > filters or test-lists, if you add a filter quick would take 2.5 > seconds, because that is necessary to calculate the total number of > tests before starting. > > To keep classic profiles like all, quick, quick_cl, gpu, cpu, and > llvmpipe working this series adds meta profiles, small XML snippets that > list other profiles. These can contain other meta profiles, xml > profiles, or python profiles. This means that for most uses cases your > existing command line will still work, `./piglit run quick out -c` will > still do exactly the same thing as before, just faster. > > The XML generated is dumb, there is no encoding of options or logic. An > early version of this series did contain logic and options, but the > result was pretty terrible. It was very hard to read, and the code to > handle it was very complicated. I've chosen not to go down that path. > There are drawbacks, some things that relied on run time generation have > cannot be handled the same way, among them the "multi shader" concept, > where shader_runner consumes a directory of shader_tests at a time. This > was previously handled via a --process-isolation=false flag, now its > encoded into profiles, "shader_multi" and "quick_shader_multi"; there > was also an option to use glslparsertest with ES shaders and > ARB_ES_compatibility, that is now "glslparser_arb_compat". I haven't > added metaprofiles for these cases, although we certainly could (or you > can write your own, the schema is dead simple), so `./piglit run quick > out --process-isolation=false` is now `./piglit run quick_gl glslparser > quick_shader_multi out`. > > I've run this through our CI extensively, and gotten green results out > of it across the board. > > I know this is a big series, but piglit makes a lot of assumptions about the > test profiles being created at runtime, and we've had to changes those > assumptions. > > > Dylan Baker (35): > update git ignore for this series > test/piglit_test: add ROOT_DIR variable > framework/profile: Allow a group manager class to be overwritten > framework/test: Use getter for altering PiglitBaseTest Command > framework/test: expose required and excluded platforms > framework/profile: Add a __len__ method to TestProfile > framework: Use custom class for ASM parser tests > framework/test: add a test class for built-in constants > tests: use BuiltInConstantsClass > framework: use a class method for building test via parsing > framework: do the same for shader test > framework/test: Split multishader too > framework/test/piglit_test: make cl_concurrency always a boolean > framework/test: Add class for cl-program-tester > framework/test: Make shader paths relative > framework/test: use relative paths for GLSLParser tests > tests/all: Make asmparser tests path relative > framework/test: make BuiltInConstantTest files relative > framework/test: make CLProgramTester take relative paths > profile: Add support for loading xml based profiles > profile: allow forcing python or xml loading > framework/profile: Add support for meta profiles > py_modules: Add support for out of tree builds > tests/quick: fix filtering of vs_in shader tests > tests: use meta profiles > shader_tests: correctly generate xml during out of tree builds > tests/glsl_parser_test.py: fix is_skip for serialized profiles > fix glslparser test for out of tree builds > fix asmparser test serialization for out of tree builds > tests/cl.py: fix out of tree serialization > opengl.py: Remove exported gl extensions > Add script to serialize profiles to XML > tests: Add script to find all hand written test files > Generate xml for builtin profiles > profile: use gz to compress profiles > > .gitignore | 4 +- > CMakeLists.txt | 10 +- > framework/profile.py | 203 +- > framework/test/glsl_parser_test.py | 60 +- > framework/test/piglit_test.py | 68 +- > framework/test/shader_test.py | 131 +- > tests/CMakeLists.no_api.txt | 89 +- > tests/CMakeLists.txt | 2 +- > tests/all.meta.xml | 6 +- > tests/all.py | 4999 +-------------- > tests/cl.py | 71 +- > tests/cpu.meta.xml | 4 +- > tests/cpu.py | 31 +- > tests/find_static_tests.py | 74 +- > tests/glslparser.py | 63 +- > tests/gpu.meta.xml | 5 +- > tests/gpu.py | 18 +- > tests/llvmpipe.meta.xml | 5 +- > tests/llvmpipe.py | 33 +- > tests/llvmpipe_gl.py | 33 +- > tests/no_error.py | 17 +- > tests/opencl_foreign.py | 34 +- > tests/opengl.py | 4837 ++++++++++++++- > tests/py_modules/constants.py | 8 +- > tests/quick.meta.xml | 6 +- > tests/quick.py | 97 +- > tests/quick_cl.meta.xml | 5 +- > tests/quick_cl.py | 37 +- > tests/quick_gl.py | 78 +- > tests/quick_shader.py | 53 +- > tests/serializer.py | 151 +- > tests/shader.py | 71 +- > unittests/framework/test/test_glsl_parser_test.py | 61 +- > unittests/framework/test/test_shader_test.py | 37 +- > unittests/framework/test_profile.py | 5 +- > 35 files changed, 6009 insertions(+), 5397 deletions(-) > create mode 100644 tests/CMakeLists.no_api.txt > create mode 100644 tests/all.meta.xml > delete mode 100644 tests/all.py > create mode 100644 tests/cpu.meta.xml > delete mode 100644 tests/cpu.py > create mode 100644 tests/find_static_tests.py > create mode 100644 tests/gpu.meta.xml > delete mode 100644 tests/gpu.py > create mode 100644 tests/llvmpipe.meta.xml > delete mode 100644 tests/llvmpipe.py > create mode 100644 tests/llvmpipe_gl.py > create mode 100644 tests/opencl_foreign.py > create mode 100644 tests/opengl.py > create mode 100644 tests/quick.meta.xml > delete mode 100644 tests/quick.py > create mode 100644 tests/quick_cl.meta.xml > delete mode 100644 tests/quick_cl.py > create mode 100644 tests/quick_gl.py > create mode 100644 tests/quick_shader.py > create mode 100644 tests/serializer.py > > base-commit: 385fbbe2e73bf4f65f425101e8fa890ae5b52649 > -- > git-series 0.9.1 > _______________________________________________ > Piglit mailing list > Piglit@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/piglit
Since I'm sure that some of these will get held up in moderation, or if you don't want to git-apply 35 patches :), a branch is here: github.com/dcbaker/piglit xml-profile Dylan
signature.asc
Description: signature
_______________________________________________ Piglit mailing list Piglit@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/piglit
