Marek Olšák <[email protected]> writes: > On Fri, May 27, 2016 at 3:18 AM, Mark Janes <[email protected]> wrote: >> Marek Olšák <[email protected]> writes: >> >>> On Mon, Apr 18, 2016 at 6:45 PM, Dylan Baker <[email protected]> >>> wrote: >>>> Quoting Marek Olšák (2016-04-16 15:16:34) >>>>> Hi, >>>>> >>>>> This makes shader_runner very fast. The expected result is 40% >>>>> decrease in quick.py running time, or a 12x faster piglit run if you >>>>> run shader tests alone. >>>>> >>>>> Branch: >>>>> https://cgit.freedesktop.org/~mareko/piglit/log/?h=shader-runner >>>>> >>>>> Changes: >>>>> >>>>> 1) Any number of test files can be specified as command-line >>>>> parameters. Those command lines can be insanely long. >>>>> >>>>> 2) shader_runner can re-create the window & GL context if test >>>>> requirements demand different settings when going from one test to >>>>> another. >>>>> >>>>> 3) all.py generates one shader_runner instance per group of tests >>>>> (usually one or two directories - tests and generated_tests). >>>>> Individual tests are reported as subtests. >>>>> >>>>> The shader_runner part is done. The python part needs more work. >>>>> >>>>> >>>>> What's missing: >>>>> >>>>> Handling of crashes. If shader_runner crashes: >>>>> - The crash is not shown in piglit results (other tests with subtests >>>>> already have the same behavior) >>>>> - The remaining tests will not be run. >>>>> >>>>> The ShaderTest python class has the list of all files and should be >>>>> able to catch a crash, check how many test results have been written, >>>>> and restart shader_runner with the remaining tests. >>>>> >>>>> shader_runner prints TEST %i: and then the subtest result. %i is the >>>>> i-th file in the list. Python can parse that and re-run shader_runner >>>>> with the first %i tests removed. (0..%i-1 -> parse subtest results; %i >>>>> -> crash; %i+1.. -> run again) >>>>> >>>>> I'm by no means a python expert, so here's an alternative solution (for >>>>> me): >>>>> - Catch crash signals in shader_runner. >>>>> - In the single handler, re-run shader_runner with the remaining tests. >>>>> >>>>> Opinions welcome, >> >> Per-test process isolation is a key feature of Piglit that the Intel CI >> relies upon. If non-crash errors bleed into separate tests, results >> will be unusable. >> >> In fact, we wrap all other test suites in piglit primarily to provide >> them with per-test process isolation. >> >> For limiting test run-time, we shard tests into groups and run them on >> parallel systems. Currently this is only supported by dEQP features, >> but it can make test time arbitrarily low if you have adequate hardware. >> >> For test suites that don't support sharding, I think it would be useful >> to generate suites from start/end times that can run the maximal set of >> tests in the targeted duration. >> >> I would be worried by complex handling of crashes. It would be >> preferable if separate suites were available to run with/without shader >> runner process isolation. >> >> Users desiring faster execution can spend the saved time figuring out >> which test crashed. > > I would say that the majority of upstream users care more about piglit > running time and less about process isolation. > > Process isolation can be an optional piglit flag.
WFM. >> >>>>> Marek >>>>> _______________________________________________ >>>>> Piglit mailing list >>>>> [email protected] >>>>> https://lists.freedesktop.org/mailman/listinfo/piglit >>>> >>>> Thanks for working on this Marek, >>>> >>>> This has been discussed here several times amongst the intel group, and >>>> the recurring problem to solve is crashing. I don't have a strong >>>> opinion on python vs catching a fail in the signal handler, except that >>>> handling in the python might be more robust, but I'm not really familiar >>>> with what a C signal handler can recover from, so it may not. >>> >>> I can catch signals like exceptions and report 'crash'. Then I can >>> open a new process from the handler to run the remaining tests, wait >>> and exit. >> >> Will an intermittent crash be run again until it passes? > > No. > > Marek _______________________________________________ Piglit mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/piglit
