Hi everybody, recently I have sent to the list the first series with patches for testing ARB_gl_spirv [1]. On the cover letter I mention "barebone tests" and "borrowed tests", and the need fordiscussion, so here we are. And sorry if the email is somewhat long and messy, but it is intended as a brainstorm. FWIW, this was already introduced in my presentation at FOSDEM, as some people (specifically Mark Janes) asked about that.
* Barebone tests Basically smoke testing for all the general shader features supported (uniforms, UBO, SSBO, etc), plus tests for some specific ARB_gl_spirv features (like specialization constants). Why "just barebone"? The issue with this extension is that the functionality it covers is basically all current GLSL functionality, with some exceptions, so it is complex to get a full testing coverage from scratch. So the idea would be to touch a little of everything, so any implementation of the extension should pass, but would not provide a full coverage of the extension for practical reasons. We would leavethat for the "borrowed tests" (more about that below) An additional advantage of this approach is that those GLSL tests are compatible withthe ARB_gl_spirv/GL_KHR_vulkan_glsl and would not need an automatic conversion like the borrowed tests from other specs (more about that below). The series I sent has25 tests. On our development branch we have ~70 individual tests, and we foresee that the final version would have a little more than100 tests. We also have some hundred generated builtin tests. Those generate SPIR-Vdirectly. This allows testingthem without the need torely on the borrowed tests, so it avoidsthe dependency on glslang to test them. It also reduces the number of builtin tests needed for a SPIR-V run, as we don't bother generating tests that doesn't make sense for SPIR-V, such as operators with mixed types. Thesebarebone tests only add a dependency to spirv-tools, as the SPIR-V is included as assembly. And in any case, it is executed as anexternal binary, as we assumed that a runtime dependency would be easier to be accepted thana build dependency. But it would be easy to replace this with a dependency onspirv-tools, and call the library directly. As far as we know, adding a dependency to spirv-tools would be considered as "fine". Some people would argue that most of that is already tested on other specs, so it would be redundant, and we should only test the ARB_gl_spirv specifics, and let the full general coverage on the borrowed tests. But there are some extra things to discuss about that. * Borrowed tests Those are the shader_tests that we re-use from other specifications. In this case, the conversion to SPIR-V is a two pass process, done by a custom script. In thefirst pass the script checks the shader and fixesit in order to be compatible with glslang shader expectations, and then the script calls glslang to convert it to SPIR-V. Fix in this case means things like change gl_Vertex/gl_FragColor to generic in/outs. In addition to fixingthe shaders, the script hassome features like markingthe tests to be skipped on the following piglit run, and allows storingthe SPIR-V assembly in anexternal file, or include it in the shader_testitself. The former is mostly useful for borrowed tests, the latter for barebone tests. Nicolai started that script, that we have been using and extended for a while. Right now we have the current conversion rate: * piglit/tests directory: [2028] skip: 537, excluded: 0, success: 1363, fail: 128 * piglit/generated_tests directory: [32354] skip: 1110, excluded: 0, success: 31244, fail: 0 That is a (IMHO) good conversion rate right now. Issues: * Dependency on glslang to create the binaries: right now the GLSL->SPIR-V pass is done using glslang. As with spirv-tools, as a runtime binary, but again we could use the library. The problem with thisapproachis that after talking with some people, they don't consider glslang as a "fine" dependency of piglit. The library API is somewhat unstable (although not sure if that changed lately), and there are also some issues with the SPIR-V generated now and then. So there is the possiblity of this dependency to be vetoed. This is also the reason barebone tests include the assembly, and only depend on spirv-tools to re-assemble it. The idea is that for the barebone tests it should not matter how SPIR-Vwas generated, you could do it writing SPIR-V assembly directly without any tool, for example. FWIW, as part of our work we created issues on glslang and provided some patches, and we got feedback/fixes, so the library is active and receptive to suggestions. * Dependency on glslang to fix the shaders: this is something new that didn't have Nicolai'soriginal script. Initially we added support for fixing uniforms lacking explicit locationson the script directly. But glslang has already some limited support for this, and other automatic fixes, so we moved to a mixed approach of using glslang features plus some minor fixes. The main advantage of this is that reduces how much the script modifies GLSL, letting glslang to do that. So if finally glslang is vetoed, we would need to reimplement those features. * Not other real alternative right now: if glslang is vetoed, we would need to face that right now there is no real alternative. There is one coming, as Ian is working on a Mesa backend for SPIR-V [1], first suggested on a previous thread [2]. Additionally, Ian's work adds an interesting alternative. If Mesa itself is able to do the GLSL->SPIR-V pass, then it would be feasible to add a special debug mode configured by an envvar, so internally it would do the GLSL->SPIR-V pass, and then the linking would use the ARB_gl_spirv path. In that way we would not need a custom script on piglit, but to enablethis envvar. But then this backend would need to also support shader fixing (more advanced thanglslang), and this would not work for non-Mesa drivers using piglit (although Iam not sure if this is a use case with high priority for piglit). In any case, as I said, this alternative is still not available. * Slow conversion: assuming that glslang dependency is okish, ideally we would like the GLSL->SPIR-V conversion to be done while building piglit, as with the generated tests. But the problem is that it is a slow process. Tests directory need 2min, and generated_tests need 50 min on my i5. Those times are with one process, but although we added support for forking N processes, generated_tests conversion would be already too slow, 25 minutes on my i5. * Slow piglit run: with ARB_gl_spirv we would be duplicating the tests to run on the shaders.py profile. And in practice, the time needed would be more than the double Due to all this issues, one alternative would betonot include/run by default the SPIR-V borrowed tests, and only include the script and instructions to usein piglit, so that any driver developer interested in the feature can useit. But as we know, usually tools that are not used tend to rot. Taking into account the above considerations, our suggestion is the followingcompromise: * Go on with the barebone tests. Add a spirv-tools runtime dependency (those tests would just fail without spirv-tools) * Accept the use of glslang, as there is no real alternative right now. * Continueexecuting itas a binary, sothat it remainsa runtime dependency. This way, platforms not implementing the ARB_gl_spirvextensionwon't have adependency onglslang, and those tests willsimplyfailor be skippped . * Only convert to SPIR-V the shaders of the test directory and not the generated_tests * Not only because of the time to convert the generated_tests. * Also because as mentioned, we are adding SPIR-V generated tests for some of the existing cases. * Revisit if generated_tests got slimmed * Get the shader.py profile to do a new run but passing a --spirv option to the test directory (or perhaps a new generated_spirv_tests directory) * Revisit everything when the Mesa GLSL->SPIR-V backend got in good shape. Opinions? PS: if you are curious, here the comparison of a full piglit shader.py run using GLSL vs using SPIR-V with our really-dirty development branch for i965 on Skylake: name: 20180418-glsl-code-p 20180418-spirv-code- ---- -------------------- -------------------- pass: 30475 28324 fail: 15 287 crash: 5 41 skip: 3887 5730 timeout: 0 0 warn: 0 0 incomplete: 0 0 dmesg-warn: 0 0 dmesg-fail: 0 0 changes: 0 2176 fixes: 0 9 regressions: 0 324 total: 34382 34382 [1] https://lists.freedesktop.org/archives/piglit/2018-April/024145.html [2] https://lists.freedesktop.org/archives/mesa-dev/2017-November/176407.html [3] https://lists.freedesktop.org/archives/mesa-dev/2017-May/156592.html [4] https://fosdem.org/2018/schedule/event/spirv/
_______________________________________________ Piglit mailing list Piglit@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/piglit
