On 06/07/2018 09:32, Jan Beulich wrote: >> Jenkins uses weather icons to attempt to convey if this test is >> trending worse or better or successful or broken. If it fails but not >> every time and the amount of failures is increasing over time then its >> got storm clouds. If the amount of failures is decreasing there's a >> little bit of sun peaking out. >> >> Just some kind of dashboard which would tell me what would provide the >> most value to drill into would likely go a long way. But again, this is >> just an assumption and could be a time waste. > I think every test failure warrants looking into. It is just the case that > after having seen a certain "uninteresting" case a number of times, I > for instance make further implications from that on later flight reports. > Maybe I shouldn't, but I also can't afford spending endless hours on > looking all the details of all the flights.
The results of testing should be a single bit. Yes or No. No means that someone needs to investigate and get it back to saying Yes. I've said this before, but categories like "fail never pass" and "fail not blocking" only muddy the water and train people to get complacent at the results. (Also, fail never pass is a 100% waste of time running in the first place, and this isn't the first time I've suggested that it is the lowest hanging of the low hanging fruit...) ~Andrew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
