On Tue, 2019-01-22 at 17:42 +0800, Yeoh Ee Peng wrote: > OEQA outputs test results into json files and these files were > archived by Autobuilder during QA releases. Example: each oe-selftest > run by Autobuilder for different host distro generate a > testresults.json file. > > These scripts were developed as a test result tools to manage > these testresults.json file. > > Using the "store" operation, user can store multiple testresults.json > files as well as the pre-configured directories used to hold those > files. > > Using the "merge" operation, user can merge multiple testresults.json > files to a target file. > > Using the "report" operation, user can view the test result summary > for all available testresults.json files inside a ordinary directory > or a git repository. > > Using the "regression" operation, user can perform regression > analysis > on testresults.json files specified.
Thanks Ee Peng, this version is much improved! As an experiment I had a local test results file and I was able to run: $ resultstool regression /tmp/repo/testresults.json /tmp/repo/testresults.json -b sdk_core-image-sato_x86_64_qemumips_20181219111311 -t sdk_core-image-sato_x86_64_qemumips_20181219200052 Successfully loaded base test results from: /tmp/repo/testresults.json Successfully loaded target test results from: /tmp/repo/testresults.json Getting base test result with result_id=sdk_core-image-sato_x86_64_qemumips_20181219111311 Getting target test result with result_id=sdk_core-image-sato_x86_64_qemumips_20181219200052 ============================Start Regression============================ Only print regression if base status not equal target <test case> : <base status> -> <target status> ======================================================================== assimp.BuildAssimp.test_assimp : ERROR -> PASSED ==============================End Regression============================== I was able to clearly see that my failing test case went from ERROR to PASSED which is good. The interface and the way the information is being presented and stored are now the things we need to work on. What is odd about the current tool/behaviour is that it sometimes expects files that are stored in a repository and sometimes expects standalone files. It doesn't feel like the work flow and the way the user would interact with this is quite correct. You can see from my test case above that the results file I wanted to compare were both in the same repo file which I'd already merged/stored. The tool can do it by passing the file twice and specifying the IDs but its awkward and not obvious. I'm still a little unsure how we expect to use the tool and whether the layout of the files in the git repository is how we're going to ultimately want to do this. For example, combining all results into a single json file means we can't really show useful comparison information directly from "git diff". Would we be better with a file per results ID for example with a layout allowing "git diff" to be useful? A directory structure like: selftest/<HOST_DISTRO>/ runtime/<DISTRO>/<MACHINE> sdk/<MACHINE>/<SDKMACHINE> esdk/<MACHINE>/<SDKMACHINE> build-perf/<HOST_DISTRO>/ for example? I don't have the "right" answer here as I've not experimented with this but its something we need to think about. As another example, the regression command can't compare the file from two different git commits easily, you have to copy one file out the git repo and then checkout the other version to compare them. This isn't easy for the user. In summary, I think the base underlying tools here are heading the right way but the layout of the git repo and the way people are expected to use and interact with it needs a little bit more work... Cheers, Richard -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
