> -----Original Message----- > From: cfe-dev <cfe-dev-boun...@lists.llvm.org> On Behalf Of David Greene > via cfe-dev > Sent: Tuesday, October 08, 2019 12:50 PM > To: llvm-...@lists.llvm.org; cfe-...@lists.llvm.org; openmp- > d...@lists.llvm.org; lldb-dev@lists.llvm.org > Subject: [cfe-dev] RFC: End-to-end testing > > [ I am initially copying only a few lists since they seem like > the most impacted projects and I didn't want to spam all the mailing > lists. Please let me know if other lists should be included. ] > > I submitted D68230 for review but this is not about that patch per se. > The patch allows update_cc_test_checks.py to process tests that should > check target asm rather than LLVM IR. We use this facility downstream > for our end-to-end tests. It strikes me that it might be useful for > upstream to do similar end-to-end testing. > > Now that the monorepo is about to become the canonical source of truth, > we have an opportunity for convenient end-to-end testing that we didn't > easily have before with svn (yes, it could be done but in an ugly way). > AFAIK the only upstream end-to-end testing we have is in test-suite and > many of those codes are very large and/or unfocused tests. > > With the monorepo we have a place to put lit-style tests that exercise > multiple subprojects, for example tests that ensure the entire clang > compilation pipeline executes correctly. We could, for example, create > a top-level "test" directory and put end-to-end tests there. Some of > the things that could be tested include: > > - Pipeline execution (debug-pass=Executions) > - Optimization warnings/messages > - Specific asm code sequences out of clang (e.g. ensure certain loops > are vectorized) > - Pragma effects (e.g. ensure loop optimizations are honored) > - Complete end-to-end PGO (generate a profile and re-compile) > - GPU/accelerator offloading > - Debuggability of clang-generated code > > Each of these things is tested to some degree within their own > subprojects, but AFAIK there are currently no dedicated tests ensuring > such things work through the entire clang pipeline flow and with other > tools that make use of the results (debuggers, etc.). It is relatively > easy to break the pipeline while the individual subproject tests > continue to pass.
I agree with all your points. End-to-end testing is a major hole in the project infrastructure; it has been largely left up to the individual vendors/packagers/distributors. The Clang tests verify that Clang will produce some sort of not-unreasonable IR for given situations; the LLVM tests verify that some (other) set of input IR will produce something that looks not-unreasonable on the target side. Very little connects the two. There is more than nothing: - test-suite has some quantity of code that is compiled end-to-end for some targets. - lldb can be set up to use the just-built Clang to compile its tests, but those are focused on debug info and are nothing like comprehensive. - libcxx likely also can use the just-built Clang, although I've never tried it so I don't know for sure. It obviously exercises just the runtime side of things. - compiler-rt likewise. The sanitizer tests in particular I'd expect to be using the just-built Clang. - debuginfo-tests also uses the just-built Clang but is a pathetically small set, and again focused on debug info. I'm not saying the LLVM Project should invest in a commercial suite (although I'd expect vendors to do so; Sony does). But a place to *put* end-to-end tests seems entirely reasonable and useful. Although I would resist calling it simply "tests" (we have too many directories with that name already). > > I realize that some folks prefer to work on only a portion of the > monorepo (for example, they just hack on LLVM). I am not sure how to > address those developers WRT end-to-end testing. On the one hand, > requiring them to run end-to-end testing means they will have to at > least check out and build the monorepo. On the other hand, it seems > less than ideal to have people developing core infrastructure and not > running tests. People should obviously be running the tests for the project(s) they're modifying. People aren't expected to run everything. That's why... Bots. "Don't argue with the bots." I don't checkout and build and test everything, and I've broken LLDB, compiler-rt, and probably others from time to time. Probably everybody has broken other projects unexpectedly. That's what bots are for, to run the combinations and the projects that I don't have the infrastructure or resources to do myself. It's not up to me to run everything possible before committing; it IS up to me to respond promptly to bot failures for my changes. I don't see a new end-to-end test project being any different in that respect. > > I don't yet have a formal proposal but wanted to put this out to spur > discussion and gather feedback and ideas. Thank you for your interest > and participation! Thanks for bringing it up! It's been a pebble in my shoe for a long time. --paulr > > -David > _______________________________________________ > cfe-dev mailing list > cfe-...@lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev _______________________________________________ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
