Even though I wrote TestBox I think we need to migrate away from it. It's big flaw is it is built to support regression tests which means using it involves bringing up the traffic_server process. That's much more heavy weight than a good unit test framework should be. Real unit testing should load as little additional code as possible.
My view is all of what we call "regression tests", e.g. those tests run via "traffic_server -R 1" should be migrated elsewhere, either up to AUtest if full up testing is needed, or down to unit tests. On Wednesday, May 24, 2017, 12:56:53 PM CDT, Masakazu Kitajo <mas...@apache.org> wrote:> Do we want to stick with the “make check/TestBox”? > Based on what I have seen so far on this thread looks like people generally like GTest. Any more ideas or suggestions? Do we drop the GTest idea? I wrote some simple unit tests for HTTP/2 related stuff with TestBox and actually I have some for QUIC stuff too, I think I could use GTest for them, and I'd happy to migrate them to use GTest if it was chosen as the official. However, I'm ok with TestBox so far (I'm not saying "TestBox is good enough" nor "GTest is too much"). A reason I used TestBox is that I thought it's the official one. That's it. Personally, I'd like to use general test frameworks because I don't want to maintain test frameworks without big reasons. Gancho, can you share the reasons that you didn't use TestBox? It'd be key points. > feels like "fixing the car engine through the exhaust pipe”. I agree, and I like the expression :) I think features themselves should be tested independently, and we should have another test for integration. This makes it clear which part (ATS core, the plugin, or sub-modules) is wrong. Thanks, Masakazu On Tue, May 23, 2017 at 6:09 PM, Gancho Tenev <gan...@apache.org> wrote: > One framework and justice for all (or to rule them all) :) > > > One framework :) > > I think most of us already agree that we should have: 1 unit test > framework and 1 integration test framework and have all the tests run as > part of our build / verification stage. > > How to get to that point is where I think people disagree (i.e. do we rip > out all old / out-of-date / broken tests and store them on separate > branches / PRs and later port them one-by-one, how do we keep updating the > code+tests while tests are still to be ported, do we run with the old and > new frameworks simultaneously for some time/forever etc.?) > > AuTest seems our integration test framework of choice. > > Do we want to stick with the “make check/TestBox”? > > Based on what I have seen so far on this thread looks like people > generally like GTest. Any more ideas or suggestions? Do we drop the GTest > idea? > > > > May be plugins are not so special case? :) > > I think we can consider plugins in more generic terms and develop them > with a better testability in mind so most of their functionality can be > tested w/o having to bring the whole ATS code-base up in order to test a > simple feature/function/line. > When thinking in terms of testing the code on “a local level" people tend > to write better / more modular code, also sometimes getting a good > test/branch coverage through an integration test framework feels like > "fixing the car engine through the exhaust pipe”. > > For bigger plugins (i.e. thousands of lines) unit tests would make more > sense. > > Although lots of things can be achieved by using AuTest, a Unit Test > Framework can be much more efficient and easy to use when one thinks in > terms of code / branch test coverage. > > In this particular case the s3_auth_v4 signing itself can be tested > outside ATS code which makes iterating while developing much faster, easier > to test / hit corner cases / refactor and experiment with the code / > measure specific code performance / check for memory problems etc. > > > > Holy wars: vim vs emacs > > Choosing a unit test is not easy, there are lots of alternatives: > https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C.2B.2B > > The main thing is people to like the framework and use it. By choosing an > easy to use and less obscure framework and by being reasonably flexible we > might be able to get more people to contribute tests (an area where we are > somehow behind). > > > > > Do these run as part of regressions now? > > Cachekey plugin TSQA tests are not part of the regression since by the > time they were created/finished our TSQA regression tests were broken and > not being run and there were already talks/plans to replace it (with > AuTest). > Have not had chance but will work on porting / moving it, to port / > integrate there should be much less effort than to create the test benches. > > The s3_auth_v4 GTest unit tests were not meant to run with the build since > we have never agreed on using GTest and of coarse the plumbing was not > ready, just used it to deliver the new code (this is to address Alan’s > comment on the PR). > > > Cheers, > —Gancho > > > > > > On May 22, 2017, at 10:21 PM, James Peach <jpe...@apache.org> wrote: > > > > On 20 May 2017, at 20:12, gan...@apache.org <mailto:gan...@apache.org> > wrote: > > > >> Hi trafficserver-dev, > >> > >> > >> (1) I was wondering if we have any preference for a C/C++ unit test > framework in the ATS community. > >> > >> Background and thoughts: > >> > >> I needed a unit test framework quickly to do TDD in order to deliver > the AWS auth v4 (PR #1946). > >> Google Test seemed easy to bootstrap, popular, feature rich and easy to > use, also its license is "BSD 3-clauses”. > >> The unit tests I wrote (PR# 1953) got the work done, they don't do > anything fancy or Google Test specific and can be converted to use any > other framework of our choice. > >> > >> > >> (2) I was wondering what to do with the unit tests until we decide on a > unit test framework that we all going to use. > >> > >> Background and thoughts: > >> > >> People have concerns that if every developer starts checking into > master unit tests using frameworks of their choice (even if not run) we > will end up with a mess/chaos. > >> > >> I understand the benefits of having one or two agreed upon frameworks > and run all our tests with every build. > > > > One framework :) > > > >> I have a similar case with the cachekey plugin tests where I used the > “old” TSQA framework (at the time) which I still run to test for > regressions and enhance when I make changes. > > > > Do these run as part of regressions now? > > > >> If we plan to rip out the “old” TSQA framework should I then remove > those tests from the source tree until I convert them to “autest"? > > > > We should have one true integration test framework. AFAIK autest is it, > but I still think it needs a bit better documentation and getting started > affordances. > > > >> My personal take is that in both examples (s3_auth_v3 and cachekey) the > tests have 100% code and very high branch coverage (the wonders of TDD) so > it would be a pity to rip them out and keep them some else until converted. > > > > See ci/coverage for test coverage instrumentation example. It would be > pretty great to have persistent coverage data from the automated testing. > > > >> If they are checked into the source tree we could still use them > “offline” to test for regressions in meanwhile, also they will be available > to anyone to work on them and would be a pretty good starting point for > bringing the tests up-to-date to the agreed upon framework when we have a > chance. > > > > We have to converge on 1 unit test framework and 1 integration test > framework. I really think there are only 2 realistic unit test choices: > > > > 1. We have an internal unit test harness with existing tests. It’s > primitive, but basically works for unit testing of traffic_server and > standalone test drivers. If we don’t run with this, we should figure out > how to replace it. > > > > 2. gtest is good and widely used. The google benchmark package is nice > and it would be helpful to have micro benchmarks for hot path code. If we > go down this path, we should bundle gtest and not use the system version at > all. > > > > James > >
