martong added a comment. In D79432#2022207 <https://reviews.llvm.org/D79432#2022207>, @xazax.hun wrote:
> I think testing summaries this way can be really hard to manage in the future. > I see two possible ways forward to make this easier: > a) Make something like https://reviews.llvm.org/D78118 that will also dump > the actual summary in a textual form, not only the fact that a summary was > loaded and check for that textual form. > b) Make a small helper library for testing summaries. E.g., most of the > buffer handling functions have similar summaries, so we could have only a > couple of functions for testing buffer related summaries and we could just > pass in function pointers (as templates or params) instead of duplicating > code. Similarly, testing output ranges could be generalized. > > I think b) might be a bit better solution. > Although I do see why some people like to keep tests simple without > additional layers of abstractions, so I do not insist :) My idea about the testing is the following: We test each individual argument constraint by itself with a test function. E.g. in case of buffer sizes we have a test function specifically for it (and it requires a debug checker to be enabled) addToFunctionSummaryMap( "__buf_size_arg_constraint", Summary(ArgTypes{ConstVoidPtrTy, SizeTy}, RetType{IntTy}, EvalCallAsPure) .ArgConstraint( BufferSize(/*Buffer=*/ArgNo(0), /*BufSize=*/ArgNo(1)))); And then in the test file we have the exact expectations: int __buf_size_arg_constraint(const void *, size_t); void test_buf_size_concrete() { char buf[3]; // bugpath-note{{'buf' initialized here}} __buf_size_arg_constraint(buf, 4); // \ // report-warning{{Function argument constraint is not satisfied}} \ // bugpath-warning{{Function argument constraint is not satisfied}} \ // bugpath-note{{Function argument constraint is not satisfied}} } ... This way we have a test coverage for each argument constraint. The summaries are composed from these argument constraints. There may be issues with the composition, so we do have some additional test cases for that, e.g we have a test to check the expectations when one function has more than one constraints: int __two_constrained_args(int, int); void test_constraints_on_multiple_args(int x, int y) { // State split should not happen here. I.e. x == 1 should not be evaluated // FALSE. __two_constrained_args(x, y); clang_analyzer_eval(x == 1); // \ // report-warning{{TRUE}} \ // bugpath-warning{{TRUE}} \ // bugpath-note{{TRUE}} clang_analyzer_eval(y == 1); // \ // report-warning{{TRUE}} \ // bugpath-warning{{TRUE}} \ // bugpath-note{{TRUE}} } So, we have tests for each individual building blocks and for their conjunctions, which renders the testing of all concrete summaries superfluous and bloated. It would be as if a compiler was tested for all possible inputs. On top of all this, errors (assertions) with concrete summaries should be discovered by running the analysis on real world projects like tmux, bitcoin, etc. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D79432/new/ https://reviews.llvm.org/D79432 _______________________________________________ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
