On Mon, Apr 23, 2018 at 10:47:08AM -0700, Dylan Baker wrote: > I'm planning to just push this Wednesday if no one expresses any more > concerns, > or signals that they want time to test or review this.
I've been running some tests this morning (regular runs, using filters, testing print-cmd and summary), and it all worked just fine for me. It's not an in deep testing, but at least I can say it works for my common use cases. So if you want, feel free to add Tested-by: Rafael Antognolli <rafael.antogno...@intel.com> > Dylan > > Quoting Dylan Baker (2018-04-04 15:26:48) > > I don't expect everyone I've CC'd to give thorough review (or any > > review), I've mostly CC'd people who I think would be interested in this > > work, or who's work flow I might be altered by it. > > > > Piglit has struggled to cope with the growing number of tests that it > > contains, especially with startup time. Piglit has always calculated > > tests at runtime, which was not a problem when there were only a few > > hundred or even thousand tests. Piglit now has roughly 55,000 > > OpenGL/OpenGL ES tests, which is a lot to calculate at start up. It also > > means that piglit needs to keep a python object for each of those tests > > in memory, which has sent the resident memory usage soaring. We've also > > moved to automatic test discovery for glslparser, asmparser, and shader > > tests, which is very convenient and reduces typing, but further > > increases the amount of time spent starting up. This has even made > > features which decrease runtime, like fast skipping, hurt startup > > performance, making it a less than desirable tradeoff in some cases. > > Even on a relatively fast machine with an nvme disk 15-20 seconds is not > > an unheard of startup time. That might be okay to run 55,000 tests, but > > not if you only need a dozen, such as when bisecting. > > > > This series is my proposal to fix that, mainly by moving much of that > > cost to build time. This series creates the infrastructure build XML > > base profiles at build time, which are installed with piglit instead of > > the python profiles. These profiles are lazily iterated over to ease > > memory usage, test objects are created as they are run, and python can > > garbage collect them as soon as they are done running. Along with that > > any filters applied to profiles (like removing 80% of the vs_in shader > > tests in quick) are done before the profile is serialized, and all fast > > skipping information is collected at build time as well, and encoded in > > the XML. All this means that start times are vastly reduced. > > > > For example: > > XML profiles > > quick: 0.5 > > shader: 0.5 > > > > master > > quick: 11.6 > > shader: 7.3 > > > > This series also implements some optimizations for running without > > filters or test-lists, if you add a filter quick would take 2.5 > > seconds, because that is necessary to calculate the total number of > > tests before starting. > > > > To keep classic profiles like all, quick, quick_cl, gpu, cpu, and > > llvmpipe working this series adds meta profiles, small XML snippets that > > list other profiles. These can contain other meta profiles, xml > > profiles, or python profiles. This means that for most uses cases your > > existing command line will still work, `./piglit run quick out -c` will > > still do exactly the same thing as before, just faster. > > > > The XML generated is dumb, there is no encoding of options or logic. An > > early version of this series did contain logic and options, but the > > result was pretty terrible. It was very hard to read, and the code to > > handle it was very complicated. I've chosen not to go down that path. > > There are drawbacks, some things that relied on run time generation have > > cannot be handled the same way, among them the "multi shader" concept, > > where shader_runner consumes a directory of shader_tests at a time. This > > was previously handled via a --process-isolation=false flag, now its > > encoded into profiles, "shader_multi" and "quick_shader_multi"; there > > was also an option to use glslparsertest with ES shaders and > > ARB_ES_compatibility, that is now "glslparser_arb_compat". I haven't > > added metaprofiles for these cases, although we certainly could (or you > > can write your own, the schema is dead simple), so `./piglit run quick > > out --process-isolation=false` is now `./piglit run quick_gl glslparser > > quick_shader_multi out`. > > > > I've run this through our CI extensively, and gotten green results out > > of it across the board. > > > > I know this is a big series, but piglit makes a lot of assumptions about the > > test profiles being created at runtime, and we've had to changes those > > assumptions. > > > > > > Dylan Baker (35): > > update git ignore for this series > > test/piglit_test: add ROOT_DIR variable > > framework/profile: Allow a group manager class to be overwritten > > framework/test: Use getter for altering PiglitBaseTest Command > > framework/test: expose required and excluded platforms > > framework/profile: Add a __len__ method to TestProfile > > framework: Use custom class for ASM parser tests > > framework/test: add a test class for built-in constants > > tests: use BuiltInConstantsClass > > framework: use a class method for building test via parsing > > framework: do the same for shader test > > framework/test: Split multishader too > > framework/test/piglit_test: make cl_concurrency always a boolean > > framework/test: Add class for cl-program-tester > > framework/test: Make shader paths relative > > framework/test: use relative paths for GLSLParser tests > > tests/all: Make asmparser tests path relative > > framework/test: make BuiltInConstantTest files relative > > framework/test: make CLProgramTester take relative paths > > profile: Add support for loading xml based profiles > > profile: allow forcing python or xml loading > > framework/profile: Add support for meta profiles > > py_modules: Add support for out of tree builds > > tests/quick: fix filtering of vs_in shader tests > > tests: use meta profiles > > shader_tests: correctly generate xml during out of tree builds > > tests/glsl_parser_test.py: fix is_skip for serialized profiles > > fix glslparser test for out of tree builds > > fix asmparser test serialization for out of tree builds > > tests/cl.py: fix out of tree serialization > > opengl.py: Remove exported gl extensions > > Add script to serialize profiles to XML > > tests: Add script to find all hand written test files > > Generate xml for builtin profiles > > profile: use gz to compress profiles > > > > .gitignore | 4 +- > > CMakeLists.txt | 10 +- > > framework/profile.py | 203 +- > > framework/test/glsl_parser_test.py | 60 +- > > framework/test/piglit_test.py | 68 +- > > framework/test/shader_test.py | 131 +- > > tests/CMakeLists.no_api.txt | 89 +- > > tests/CMakeLists.txt | 2 +- > > tests/all.meta.xml | 6 +- > > tests/all.py | 4999 +-------------- > > tests/cl.py | 71 +- > > tests/cpu.meta.xml | 4 +- > > tests/cpu.py | 31 +- > > tests/find_static_tests.py | 74 +- > > tests/glslparser.py | 63 +- > > tests/gpu.meta.xml | 5 +- > > tests/gpu.py | 18 +- > > tests/llvmpipe.meta.xml | 5 +- > > tests/llvmpipe.py | 33 +- > > tests/llvmpipe_gl.py | 33 +- > > tests/no_error.py | 17 +- > > tests/opencl_foreign.py | 34 +- > > tests/opengl.py | 4837 ++++++++++++++- > > tests/py_modules/constants.py | 8 +- > > tests/quick.meta.xml | 6 +- > > tests/quick.py | 97 +- > > tests/quick_cl.meta.xml | 5 +- > > tests/quick_cl.py | 37 +- > > tests/quick_gl.py | 78 +- > > tests/quick_shader.py | 53 +- > > tests/serializer.py | 151 +- > > tests/shader.py | 71 +- > > unittests/framework/test/test_glsl_parser_test.py | 61 +- > > unittests/framework/test/test_shader_test.py | 37 +- > > unittests/framework/test_profile.py | 5 +- > > 35 files changed, 6009 insertions(+), 5397 deletions(-) > > create mode 100644 tests/CMakeLists.no_api.txt > > create mode 100644 tests/all.meta.xml > > delete mode 100644 tests/all.py > > create mode 100644 tests/cpu.meta.xml > > delete mode 100644 tests/cpu.py > > create mode 100644 tests/find_static_tests.py > > create mode 100644 tests/gpu.meta.xml > > delete mode 100644 tests/gpu.py > > create mode 100644 tests/llvmpipe.meta.xml > > delete mode 100644 tests/llvmpipe.py > > create mode 100644 tests/llvmpipe_gl.py > > create mode 100644 tests/opencl_foreign.py > > create mode 100644 tests/opengl.py > > create mode 100644 tests/quick.meta.xml > > delete mode 100644 tests/quick.py > > create mode 100644 tests/quick_cl.meta.xml > > delete mode 100644 tests/quick_cl.py > > create mode 100644 tests/quick_gl.py > > create mode 100644 tests/quick_shader.py > > create mode 100644 tests/serializer.py > > > > base-commit: 385fbbe2e73bf4f65f425101e8fa890ae5b52649 > > -- > > git-series 0.9.1 > > _______________________________________________ > > Piglit mailing list > > Piglit@lists.freedesktop.org > > https://lists.freedesktop.org/mailman/listinfo/piglit > _______________________________________________ > Piglit mailing list > Piglit@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/piglit _______________________________________________ Piglit mailing list Piglit@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/piglit
