On Thu, 27 Feb 2014, brian avants wrote: > i havent followed the whole discussion but agree that it would be nice to > separate our testing data entirely from the source, as yarik suggests, at > least for ANTs. > i have been struggling a bit with testing lately, specifically, testing > combinations of parameters on combinations of data. � is there any > new/better approach to this?
do you mean like a good/fancy testDriver which would be given a routine, data, expected output to compare? indeed it would be nice to have some easy/standardized way. I am only familiar somewhat with Torsten's cmtk way http://anonscm.debian.org/gitweb/?p=pkg-exppsy/cmtk.git;a=blob;hb=HEAD;f=testing/apps/appsTestDriver.sh.in where the driver is the actual collection of tests to be ran. In Python world there is more or less now standardized way to run parametric tests (the same test used on a set of input parameters), and in PyMVPA we still use our home-brewed @sweepargs decorator [1] which allows to explore the combinations of the parameters for the test [1] http://github.com/PyMVPA/PyMVPA/blob/HEAD/mvpa2/testing/sweep.py Also, although may be a bit early to relate, Michael Hanke initiated testkraut https://testkraut.readthedocs.org/en/latest/ validation platform, which although targeting fingerprinting (state/versions of libraries etc)/validation of complete analysis pipelines, could be brought in for parametric testing of specific tools. Its runtime fingerprinting feature could actually become very beneficial for bisecting sporadic failures across different environments and/or upgrades. > � if i am going to put effort into this, then > would like to improve the testing situation, in general, if possible. I would also be interested to hear about a good approach for testing, especially in neuroimaging projects relying on cmake. We are (very) slowly working on introducing some public testing for AFNI so we could QA it at build-time. -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik -- To UNSUBSCRIBE, email to [email protected] with a subject of "unsubscribe". Trouble? Contact [email protected] Archive: https://lists.debian.org/[email protected]
