One framework and justice for all (or to rule them all) :) > One framework :)
I think most of us already agree that we should have: 1 unit test framework and 1 integration test framework and have all the tests run as part of our build / verification stage. How to get to that point is where I think people disagree (i.e. do we rip out all old / out-of-date / broken tests and store them on separate branches / PRs and later port them one-by-one, how do we keep updating the code+tests while tests are still to be ported, do we run with the old and new frameworks simultaneously for some time/forever etc.?) AuTest seems our integration test framework of choice. Do we want to stick with the “make check/TestBox”? Based on what I have seen so far on this thread looks like people generally like GTest. Any more ideas or suggestions? Do we drop the GTest idea? May be plugins are not so special case? :) I think we can consider plugins in more generic terms and develop them with a better testability in mind so most of their functionality can be tested w/o having to bring the whole ATS code-base up in order to test a simple feature/function/line. When thinking in terms of testing the code on “a local level" people tend to write better / more modular code, also sometimes getting a good test/branch coverage through an integration test framework feels like "fixing the car engine through the exhaust pipe”. For bigger plugins (i.e. thousands of lines) unit tests would make more sense. Although lots of things can be achieved by using AuTest, a Unit Test Framework can be much more efficient and easy to use when one thinks in terms of code / branch test coverage. In this particular case the s3_auth_v4 signing itself can be tested outside ATS code which makes iterating while developing much faster, easier to test / hit corner cases / refactor and experiment with the code / measure specific code performance / check for memory problems etc. Holy wars: vim vs emacs Choosing a unit test is not easy, there are lots of alternatives: https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C.2B.2B The main thing is people to like the framework and use it. By choosing an easy to use and less obscure framework and by being reasonably flexible we might be able to get more people to contribute tests (an area where we are somehow behind). > Do these run as part of regressions now? Cachekey plugin TSQA tests are not part of the regression since by the time they were created/finished our TSQA regression tests were broken and not being run and there were already talks/plans to replace it (with AuTest). Have not had chance but will work on porting / moving it, to port / integrate there should be much less effort than to create the test benches. The s3_auth_v4 GTest unit tests were not meant to run with the build since we have never agreed on using GTest and of coarse the plumbing was not ready, just used it to deliver the new code (this is to address Alan’s comment on the PR). Cheers, —Gancho > On May 22, 2017, at 10:21 PM, James Peach <jpe...@apache.org> wrote: > > On 20 May 2017, at 20:12, gan...@apache.org <mailto:gan...@apache.org> wrote: > >> Hi trafficserver-dev, >> >> >> (1) I was wondering if we have any preference for a C/C++ unit test >> framework in the ATS community. >> >> Background and thoughts: >> >> I needed a unit test framework quickly to do TDD in order to deliver the AWS >> auth v4 (PR #1946). >> Google Test seemed easy to bootstrap, popular, feature rich and easy to use, >> also its license is "BSD 3-clauses”. >> The unit tests I wrote (PR# 1953) got the work done, they don't do anything >> fancy or Google Test specific and can be converted to use any other >> framework of our choice. >> >> >> (2) I was wondering what to do with the unit tests until we decide on a unit >> test framework that we all going to use. >> >> Background and thoughts: >> >> People have concerns that if every developer starts checking into master >> unit tests using frameworks of their choice (even if not run) we will end up >> with a mess/chaos. >> >> I understand the benefits of having one or two agreed upon frameworks and >> run all our tests with every build. > > One framework :) > >> I have a similar case with the cachekey plugin tests where I used the “old” >> TSQA framework (at the time) which I still run to test for regressions and >> enhance when I make changes. > > Do these run as part of regressions now? > >> If we plan to rip out the “old” TSQA framework should I then remove those >> tests from the source tree until I convert them to “autest"? > > We should have one true integration test framework. AFAIK autest is it, but I > still think it needs a bit better documentation and getting started > affordances. > >> My personal take is that in both examples (s3_auth_v3 and cachekey) the >> tests have 100% code and very high branch coverage (the wonders of TDD) so >> it would be a pity to rip them out and keep them some else until converted. > > See ci/coverage for test coverage instrumentation example. It would be pretty > great to have persistent coverage data from the automated testing. > >> If they are checked into the source tree we could still use them “offline” >> to test for regressions in meanwhile, also they will be available to anyone >> to work on them and would be a pretty good starting point for bringing the >> tests up-to-date to the agreed upon framework when we have a chance. > > We have to converge on 1 unit test framework and 1 integration test > framework. I really think there are only 2 realistic unit test choices: > > 1. We have an internal unit test harness with existing tests. It’s primitive, > but basically works for unit testing of traffic_server and standalone test > drivers. If we don’t run with this, we should figure out how to replace it. > > 2. gtest is good and widely used. The google benchmark package is nice and it > would be helpful to have micro benchmarks for hot path code. If we go down > this path, we should bundle gtest and not use the system version at all. > > James
