Hi,
On 7/24/2014 19:40, Eric Yu wrote: > Hello, > My name is Eric Yu, and I am an intern at National Instruments for > this summer. The project I am currently > working on involves integrating ptest with our current automated testing > framework to help test our open- > embedded distributions. On the Yocto Project wiki, it states that one major > point of ptest is to consolidate the > output of different package tests in a common “<result>: <testname>” format. > I was hoping that this common > format would allow ptest results to be easily machine parsed and integrated > with our current testing framework. > However, after enabling and running the ptests, I discovered that the > formatting of the results was not as > friendly to automation as I had hoped. > > It appears that each separate package prints out its own errors, warnings, > and other output along with these > test results, burying the common output in lots of other text. Indeed (one of) the purpose of ptest is to facilitate automation by using the common <result>: <testname> format. However, ptest should not filter the output of a package test (so that the automation could be done easier). That output might be useful in later investigation about why a test failed and further improving the package (testing). Is should be easier to implement this filter in your automation scripts. > > Also, one package (gdk-pixbuff) used “FAILED” when reporting results rather > than the expected “FAIL”. This sounds like a bug. You can help us sending a patch for this. :) > > In the bash ptests, several tests print warnings saying to ignore failures > where the output differs only by > whitespace. This seems to be bad practice for test writing and is not > friendly to automated analysis of test > results. This is a good example to support my statement above: While automation scripts may easily add some PASS, FAIL, SKIP, XFAIL, XPASS and ERROR entries into a database, those (additional) warning are helpful for people interested in improving bash (an bash testing), hence the output should be kept along with the results. > > At the conclusion of each ptest, some packages give a summary of how many > tests were passed, skipped, and > failed, while others do not. I find that having these summaries gives a > useful overview of how the test went > overall and is a good reference in case some tests fail to produce the common > “<result>: <testname>” output. > > I understand that much of this is due to the fact that separate developers > write the tests for different packages, > but it would be beneficial if ptest was friendlier to automated parsing and > analysis of test results. Currently, I > have addressed some of these obstacles by writing a simple script that parses > the output of each ptest and only > outputs only the “<result>: <testname>” results while accounting for both > “FAIL” and “FAILED”. The script keeps > a running count of how many tests were reported as failed, skipped or passed, > and at the conclusion of each > ptest, the script prints a summary including the number of tests passed, > skipped, and failed along with a total > number of tests run. While this works with the current set of ptests, as more > and more packages add ptest > functionality, this script may not scale well if more inconsistencies in > formatting are introduced. Therefore, I > believe it would be a good idea to enforce a more consistent formatting of > ptest results to assist in the use of > ptest for automated testing. Are there any plans to further consolidate the > ptest result format such that it is > more accessible for automated testing? I hope I have answered at least partially to your questions above. Kind regards, Tudor.
-- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
