> On May 24, 2017, at 11:28 AM, Alan Carroll > <solidwallofc...@yahoo-inc.com.INVALID> wrote: > > Even though I wrote TestBox I think we need to migrate away from it. It's big > flaw is it is built to support regression tests which means using it involves > bringing up the traffic_server process. That's much more heavy weight than a > good unit test framework should be. Real unit testing should load as little > additional code as possible.
There are a number of examples of using TestBox and REGRESSION_TEST for unit tests that don't depend on traffic_server. Depending on what you are testing, the big-ball-of-mud dependencies might bite you but there's no dependency from RegressionTest to traffic_server itself. > > My view is all of what we call "regression tests", e.g. those tests run via > "traffic_server -R 1" should be migrated elsewhere, either up to AUtest if > full up testing is needed, or down to unit tests. > > On Wednesday, May 24, 2017, 12:56:53 PM CDT, Masakazu Kitajo > <mas...@apache.org> wrote:> Do we want to stick with the “make check/TestBox”? >> Based on what I have seen so far on this thread looks like people > generally like GTest. Any more ideas or suggestions? Do we drop the GTest > idea? > > I wrote some simple unit tests for HTTP/2 related stuff with TestBox and > actually I have some for QUIC stuff too, I think I could use GTest for > them, and I'd happy to migrate them to use GTest if it was chosen as the > official. However, I'm ok with TestBox so far (I'm not saying "TestBox is > good enough" nor "GTest is too much"). A reason I used TestBox is that I > thought it's the official one. That's it. Personally, I'd like to use > general test frameworks because I don't want to maintain test frameworks > without big reasons. > > Gancho, can you share the reasons that you didn't use TestBox? It'd be > key points. > > >> feels like "fixing the car engine through the exhaust pipe”. > > I agree, and I like the expression :) > > I think features themselves should be tested independently, and we should > have another test for integration. This makes it clear which part (ATS > core, the plugin, or sub-modules) is wrong. > > > Thanks, > Masakazu > > > On Tue, May 23, 2017 at 6:09 PM, Gancho Tenev <gan...@apache.org> wrote: > >> One framework and justice for all (or to rule them all) :) >> >>> One framework :) >> >> I think most of us already agree that we should have: 1 unit test >> framework and 1 integration test framework and have all the tests run as >> part of our build / verification stage. >> >> How to get to that point is where I think people disagree (i.e. do we rip >> out all old / out-of-date / broken tests and store them on separate >> branches / PRs and later port them one-by-one, how do we keep updating the >> code+tests while tests are still to be ported, do we run with the old and >> new frameworks simultaneously for some time/forever etc.?) >> >> AuTest seems our integration test framework of choice. >> >> Do we want to stick with the “make check/TestBox”? >> >> Based on what I have seen so far on this thread looks like people >> generally like GTest. Any more ideas or suggestions? Do we drop the GTest >> idea? >> >> >> >> May be plugins are not so special case? :) >> >> I think we can consider plugins in more generic terms and develop them >> with a better testability in mind so most of their functionality can be >> tested w/o having to bring the whole ATS code-base up in order to test a >> simple feature/function/line. >> When thinking in terms of testing the code on “a local level" people tend >> to write better / more modular code, also sometimes getting a good >> test/branch coverage through an integration test framework feels like >> "fixing the car engine through the exhaust pipe”. >> >> For bigger plugins (i.e. thousands of lines) unit tests would make more >> sense. >> >> Although lots of things can be achieved by using AuTest, a Unit Test >> Framework can be much more efficient and easy to use when one thinks in >> terms of code / branch test coverage. >> >> In this particular case the s3_auth_v4 signing itself can be tested >> outside ATS code which makes iterating while developing much faster, easier >> to test / hit corner cases / refactor and experiment with the code / >> measure specific code performance / check for memory problems etc. >> >> >> >> Holy wars: vim vs emacs >> >> Choosing a unit test is not easy, there are lots of alternatives: >> https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C.2B.2B >> >> The main thing is people to like the framework and use it. By choosing an >> easy to use and less obscure framework and by being reasonably flexible we >> might be able to get more people to contribute tests (an area where we are >> somehow behind). >> >> >> >>> Do these run as part of regressions now? >> >> Cachekey plugin TSQA tests are not part of the regression since by the >> time they were created/finished our TSQA regression tests were broken and >> not being run and there were already talks/plans to replace it (with >> AuTest). >> Have not had chance but will work on porting / moving it, to port / >> integrate there should be much less effort than to create the test benches. >> >> The s3_auth_v4 GTest unit tests were not meant to run with the build since >> we have never agreed on using GTest and of coarse the plumbing was not >> ready, just used it to deliver the new code (this is to address Alan’s >> comment on the PR). >> >> >> Cheers, >> —Gancho >> >> >> >> >>> On May 22, 2017, at 10:21 PM, James Peach <jpe...@apache.org> wrote: >>> >>> On 20 May 2017, at 20:12, gan...@apache.org <mailto:gan...@apache.org> >> wrote: >>> >>>> Hi trafficserver-dev, >>>> >>>> >>>> (1) I was wondering if we have any preference for a C/C++ unit test >> framework in the ATS community. >>>> >>>> Background and thoughts: >>>> >>>> I needed a unit test framework quickly to do TDD in order to deliver >> the AWS auth v4 (PR #1946). >>>> Google Test seemed easy to bootstrap, popular, feature rich and easy to >> use, also its license is "BSD 3-clauses”. >>>> The unit tests I wrote (PR# 1953) got the work done, they don't do >> anything fancy or Google Test specific and can be converted to use any >> other framework of our choice. >>>> >>>> >>>> (2) I was wondering what to do with the unit tests until we decide on a >> unit test framework that we all going to use. >>>> >>>> Background and thoughts: >>>> >>>> People have concerns that if every developer starts checking into >> master unit tests using frameworks of their choice (even if not run) we >> will end up with a mess/chaos. >>>> >>>> I understand the benefits of having one or two agreed upon frameworks >> and run all our tests with every build. >>> >>> One framework :) >>> >>>> I have a similar case with the cachekey plugin tests where I used the >> “old” TSQA framework (at the time) which I still run to test for >> regressions and enhance when I make changes. >>> >>> Do these run as part of regressions now? >>> >>>> If we plan to rip out the “old” TSQA framework should I then remove >> those tests from the source tree until I convert them to “autest"? >>> >>> We should have one true integration test framework. AFAIK autest is it, >> but I still think it needs a bit better documentation and getting started >> affordances. >>> >>>> My personal take is that in both examples (s3_auth_v3 and cachekey) the >> tests have 100% code and very high branch coverage (the wonders of TDD) so >> it would be a pity to rip them out and keep them some else until converted. >>> >>> See ci/coverage for test coverage instrumentation example. It would be >> pretty great to have persistent coverage data from the automated testing. >>> >>>> If they are checked into the source tree we could still use them >> “offline” to test for regressions in meanwhile, also they will be available >> to anyone to work on them and would be a pretty good starting point for >> bringing the tests up-to-date to the agreed upon framework when we have a >> chance. >>> >>> We have to converge on 1 unit test framework and 1 integration test >> framework. I really think there are only 2 realistic unit test choices: >>> >>> 1. We have an internal unit test harness with existing tests. It’s >> primitive, but basically works for unit testing of traffic_server and >> standalone test drivers. If we don’t run with this, we should figure out >> how to replace it. >>> >>> 2. gtest is good and widely used. The google benchmark package is nice >> and it would be helpful to have micro benchmarks for hot path code. If we >> go down this path, we should bundle gtest and not use the system version at >> all. >>> >>> James >>
