> -----Original Message----- > From: Lespiau, Damien > Sent: Thursday, November 13, 2014 8:28 PM > To: He, Shuang > Cc: intel-gfx@lists.freedesktop.org; daniel.vet...@ffwll.ch > Subject: Re: [Intel-gfx] [PATCH] drm/i915: Delete outdated comment in > > Hi, > > This is a great example that shows that either some tests or the testing > methodology are not quite right and need a few more iterations. [He, Shuang] Thanks for your great feedback. Actually I'm also curious why no one is asking how the result is formatted (I thought engineers are concentrating on codes only:)). It definitely need improvement.
> > This patch just removes a comment, so the ideal case is no delta in the > test results. Two things that may help improving the situation. > > 1/ Some tests may be unreliable. It's be great to have a way to mark > them as such and display that state in the results so we don't worry > too much if an unreliable test fails. I think the test itself is a > great place to store that information. [He, Shuang] We have already tried our best to remove as many unreliable tests as possible. Actually my query will only select pass many times in the recent testing, and no fail result in recent testing. But still no luck, we still have kind of noise there. Yi and one of our GB has spent great effort recently to have known unstable tests blacklisted, while combine those works together there is still unreliable test case show randomly. And actually these unreliable tests greatly impact bisection efficiency. That's one of major reason that PRTS is now using that blacklist as nightly does (PRTS used to run all test cases). > > 2/ The reference against which this delta is taken seems not > completely up to date, otherwise we would have fewer failing cases? [He, Shuang] The only thing I can guarantee is baseline results are based on the same kernel commit, while it might run on a different hardware. In our design, our script will re-run diff results (patched result compared with baseline result) on same test machine (the machine your patch is tested). > > Some other remarks: > > - I don't actually understand some of the delta shown: > > IVB: Intel_gpu_tools, igt_gem_bad_reloc_negative-reloc, NSPT(7, > M21M34M4)PASS(12, M34M21M4) -> NSPT(1, M34)PASS(3, M34) > > going from NPST to NSPT, which is that line shown? IVB shows > pass/total=545/546->545/546 and yet we have two IVB lines in the > details [He, Shuang] 545/546->546/546 simply means the pass rate doesn't change, while in some cases that result changed while keep the passrate unchanged. For example, 1 case change from PASS to NSPT, while another case changed from NSPT to PASS, you will see the passrate doesn't change. But the truth is there is some change happened, and we have to let you know about it. And for NSPT->NSPT as you noticed, is actually have sometime passed and sometime NSPT with baseline kernel, and it also sometime get NSPT and sometime get PASS with patched kernel. Before we have one solid way to tell if one test case is really unreliable in an automatically, telling you the truth we have is the best we can do right now. > > - What does 'count' means in the results below? [He, Shuang] The count means how many times the test case get that result > > - The results are somewhat hard to read and would benefit from a bit > more formatting, even if just alignment of columns. [He, Shuang] Yes, it absolutely need more refinement. I would be very happen to apply them if you have any suggestion > > - You're missing the In-reply-to header in your answers which breaks > mail threading [He, Shuang] It is in the header. You could check archieve of intel-gfx mailing list http://lists.freedesktop.org/archives/intel-gfx/2014-November/thread.html. It's correctly threaded by intel-gfx mailing > > - It'd be great to have public links with logs to inspect the failure > cases. [He, Shuang] that is one exactly same concern I had also, since we can't send internal link to external mailing list. While we need one way to hold logs Thanks --Shuang > > Thanks! > > -- > Damien > > On Wed, Nov 12, 2014 at 11:21:35PM -0800, shuang...@intel.com wrote: > > Tested-By: PRC QA PRTS (Patch Regression Test System Contact: > shuang...@intel.com) > > -------------------------------------Summary------------------------------------- > > Platform: baseline_drm_intel_nightly_pass_rate->patch_applied_pass_rate > > BYT: pass/total=291/291->290/291 > > PNV: pass/total=356/356->356/356 > > ILK: pass/total=372/372->365/372 > > IVB: pass/total=545/546->545/546 > > SNB: pass/total=380/380->379/380 > > HSW: pass/total=579/579->579/579 > > BDW: pass/total=434/435->434/435 > > -------------------------------------Detailed------------------------------------- > > test_platform: test_suite, test_case, result_with_drm_intel_nightly(count, > machine_id...)...->result_with_patch_applied(count, machine_id)... > > BYT: Intel_gpu_tools, > igt_gem_userptr_blits_forked-sync-swapping-multifd-mempressure-interruptib > le, PASS(4, M31M38) -> NO_RESULT(1, M38)PASS(3, M38) > > ILK: Intel_gpu_tools, igt_kms_flip_bcs-wf_vblank-vs-dpms-interruptible, > DMESG_WARN(3, M26)PASS(4, M26) -> DMESG_WARN(1, M26)PASS(3, M26) > > ILK: Intel_gpu_tools, igt_kms_flip_flip-vs-dpms-off-vs-modeset, > DMESG_WARN(2, M26)PASS(2, M26) -> DMESG_WARN(2, M26)PASS(2, M26) > > ILK: Intel_gpu_tools, igt_kms_flip_flip-vs-modeset-vs-hang, NSPT(1, > M26)DMESG_WARN(1, M26)PASS(2, M26) -> DMESG_WARN(1, M26)PASS(3, > M26) > > ILK: Intel_gpu_tools, igt_kms_flip_flip-vs-wf_vblank-interruptible, > DMESG_WARN(1, M26)PASS(6, M26) -> DMESG_WARN(1, M26)PASS(3, M26) > > ILK: Intel_gpu_tools, igt_kms_flip_nonexisting-fb-interruptible, > DMESG_WARN(1, M26)PASS(3, M26) -> DMESG_WARN(2, M26)PASS(2, M26) > > ILK: Intel_gpu_tools, igt_kms_flip_plain-flip-ts-check-interruptible, > DMESG_WARN(1, M26)PASS(3, M26) -> DMESG_WARN(2, M26)PASS(2, M26) > > ILK: Intel_gpu_tools, igt_kms_setmode_invalid-clone-single-crtc, FAIL(3, > M26)DMESG_FAIL(1, M26)TIMEOUT(17, M37M6M26)PASS(1, M26) -> > TIMEOUT(1, M26)PASS(3, M26) > > IVB: Intel_gpu_tools, igt_gem_bad_reloc_negative-reloc, NSPT(7, > M21M34M4)PASS(12, M34M21M4) -> NSPT(1, M34)PASS(3, M34) > > IVB: Intel_gpu_tools, igt_kms_setmode_invalid-clone-single-crtc, > TIMEOUT(18, M34M21M4)PASS(1, M21) -> TIMEOUT(1, M34)PASS(3, M34) > > SNB: Intel_gpu_tools, igt_kms_setmode_invalid-clone-single-crtc, > TIMEOUT(18, M35M22)PASS(1, M35) -> TIMEOUT(1, M22)PASS(3, M22) > > BDW: Intel_gpu_tools, igt_gem_reset_stats_ban-bsd, DMESG_WARN(1, > M28)PASS(18, M42M30M28) -> PASS(4, M28) > > BDW: Intel_gpu_tools, igt_kms_setmode_invalid-clone-single-crtc, > TIMEOUT(18, M42M30M28)PASS(1, M28) -> TIMEOUT(1, M28)PASS(3, M28) > > _______________________________________________ > > Intel-gfx mailing list > > Intel-gfx@lists.freedesktop.org > > http://lists.freedesktop.org/mailman/listinfo/intel-gfx _______________________________________________ Intel-gfx mailing list Intel-gfx@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/intel-gfx
