Hi Paul, On 2015-03-21 14:23, Paul Tan wrote:
> Thanks for the review, though I would like to work on the proposal now > before the deadline passes :) That makes sense. > On Thu, Mar 19, 2015 at 1:52 AM, Matthieu Moy > <matthieu....@grenoble-inp.fr> wrote: >> Paul Tan <pyoka...@gmail.com> writes: >> >>> Ideally, I think the solution is to >>> improve the test suite and make it as comprehensive as possible, but >>> writing a comprehensive test suite may be too time consuming. >> >> time-consuming, but also very beneficial since the code would end up >> being better tested. For sure, a rewrite is a good way to break stuff, >> but anything untested can also be broken by mistake rather easily at any >> time. >> >> I'd suggest doing a bit of manual mutation testing: take your C code, >> comment-out a few lines of code, see if the tests still pass, and if >> they do, add a failing test that passes again once you uncomment the >> code. > > Maybe code coverage tools could help here so we only need to focus on > the code paths that are untested by the test suite. At the minimum, > all of the non-trivial code paths in both the shell script and the > converted builtin must be covered by tests. This should help to > eliminate most sources of breakages. Anything further than that would > require an experienced understanding of all the possible important > inputs to be tested, which I personally feel would make the project > quite tedious. > > I see git already has gcov support. For shell scripts, maybe kcov[1] > could be used. With some slight code changes, I managed to generate a > report for the git-pull tests[2] which should at least provide a good > starting point for how the tests can be improved. While it is often a tempting idea to make test suites as thorough as possible, there lies a true danger herein. True war story: in one of the projects I was involved in, the test suite grew to a size that one complete run lasted two weeks. Yes, that is fourteen days. Needless to say: this test suite was run rarely. How useful is a test suite that is run rarely? More useful than a non-existent one, to be sure, but it is still more of a burden than a boon. Now, on Windows the test suite takes almost three hours to run. This really, really slows down development. So while we are not yet at the "too large to be useful state", I would caution against trying to get there. Instead, I would really like to focus on the *usage*. Calling `git grep "git pull" t/` should give you an idea what usage of `git pull` is already tested. It should be pretty easy to come up with a list of *common* use cases, and if any of them are not covered, adding tests for them is simple and straight-forward, too. Ciao, Johannes -- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
