I submitted this patch to include "ERROR" lines in buildbot step results. http://reviews.llvm.org/rL255145
Error results will be displayed in step result like this after the patch, "ERROR: 9 (SIGKILL) test_buildbot_catches_exceptional_exit_dwarf" Thanks, Ying On Wed, Dec 9, 2015 at 10:45 AM, Todd Fiala <todd.fi...@gmail.com> wrote: > Great, thanks Tamas! > > I left the default turned on, and just essentially removed the issues by > parking them as .py.parked files. That way we can flip them on in the > future if we want to verify a testbot's detection of these. > > I will be going back to the xUnit Results formatter and making sure it > maps timeouts and exceptional errors to the xUnit error type with details. > > On Wed, Dec 9, 2015 at 10:30 AM, Tamas Berghammer <tbergham...@google.com> > wrote: > >> Thank you for making the experiment. It looks reasonable. For the ERROR >> the buildbot detected it and it will fail the build but it isn't listed in >> the list of failing tests what should be fixed. After this experiment I >> think it is fine to change the default output formatter from our side. >> >> Tamas >> >> On Wed, Dec 9, 2015 at 6:26 PM Todd Fiala <todd.fi...@gmail.com> wrote: >> >>> The reports look good at the test level: >>> >>> >>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/9294 >>> >>> I'd say the buildbot reflection script missed the ERROR, so that is >>> something maybe Ying can look at (the summary line in the build run), but >>> that is unrelated AFAICT. >>> >>> I'm going to move aside the failures. >>> >>> On Wed, Dec 9, 2015 at 10:13 AM, Todd Fiala <todd.fi...@gmail.com> >>> wrote: >>> >>>> I am going to stop the current build on that builder. There was one >>>> change in it, and it will be another 20 minutes before it completes. I >>>> don't want the repo in a known broken state that long. >>>> >>>> On Wed, Dec 9, 2015 at 10:07 AM, Todd Fiala <todd.fi...@gmail.com> >>>> wrote: >>>> >>>>> I forced a build on the ubuntu 14.04 cmake builder. The build _after_ >>>>> 9292 will contain the two changes (and we will expect failures on it). >>>>> >>>>> On Wed, Dec 9, 2015 at 10:05 AM, Todd Fiala <todd.fi...@gmail.com> >>>>> wrote: >>>>> >>>>>> These went in as: >>>>>> >>>>>> r255130 - turn it on by default >>>>>> r255131 - create known issues. This one is to be reverted if all 3 >>>>>> types show up properly. >>>>>> >>>>>> On Wed, Dec 9, 2015 at 9:41 AM, Todd Fiala <todd.fi...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> It is a small change. >>>>>>> >>>>>>> I almost have all the trial tests ready, so I'll just commit both >>>>>>> changes at the same time (the flip on, and the trial balloon issues). >>>>>>> >>>>>>> If all goes well and the three types of issue show up, then the last >>>>>>> of the two will get reverted (the one with the failures). >>>>>>> >>>>>>> If none (or only some) of the issues show up, they'll both get >>>>>>> reverted. >>>>>>> >>>>>>> -Todd >>>>>>> >>>>>>> On Wed, Dec 9, 2015 at 9:39 AM, Pavel Labath <lab...@google.com> >>>>>>> wrote: >>>>>>> >>>>>>>> If it's not too much work, I think the extra bit of noise will not >>>>>>>> be >>>>>>>> a problem. But I don't think it is really necessary either. >>>>>>>> >>>>>>>> I assume the actual flip will be a small change that we can back out >>>>>>>> easily if we notice troubles... After a sufficient grace period we >>>>>>>> can >>>>>>>> remove the old formatter altogether and hopefully simplify the code >>>>>>>> somewhat. >>>>>>>> >>>>>>>> pl >>>>>>>> >>>>>>>> On 9 December 2015 at 17:08, Todd Fiala <todd.fi...@gmail.com> >>>>>>>> wrote: >>>>>>>> > Here's what I can do. >>>>>>>> > >>>>>>>> > Put in the change (setting the default to use the new format). >>>>>>>> > >>>>>>>> > Separately, put in a trial balloon commit with one failing test, >>>>>>>> one >>>>>>>> > exceptional exit test, and one timeout test, and watch the ubuntu >>>>>>>> 14.04 >>>>>>>> > buildbot catch it and fail. Then reverse this out. That should >>>>>>>> show beyond >>>>>>>> > a reasonable doubt whether the buildbot catches new failures and >>>>>>>> errors. (I >>>>>>>> > think this is a noisy way to accomplish this, but it certainly >>>>>>>> would >>>>>>>> > validate if its working). >>>>>>>> > >>>>>>>> > -Todd >>>>>>>> > >>>>>>>> > On Wed, Dec 9, 2015 at 8:06 AM, Todd Fiala <todd.fi...@gmail.com> >>>>>>>> wrote: >>>>>>>> >> >>>>>>>> >> Specifically, the markers for issue details are: >>>>>>>> >> >>>>>>>> >> FAIL >>>>>>>> >> ERROR >>>>>>>> >> UNEXPECTED SUCCESS >>>>>>>> >> TIMEOUT >>>>>>>> >> >>>>>>>> >> (These are the fourth field in the array entries (lines 275 - >>>>>>>> 290) of >>>>>>>> >> packages/Python/lldbsuite/test/basic_results_formatter.py). >>>>>>>> >> >>>>>>>> >> -Todd >>>>>>>> >> >>>>>>>> >> On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala <todd.fi...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>> >>>>>>>> >>> That's a good point, Tamas. >>>>>>>> >>> >>>>>>>> >>> I use (so I claim) the same all upper-case markers for the test >>>>>>>> result >>>>>>>> >>> details. Including, not using XPASS but rather UNEXPECTED >>>>>>>> SUCCESS for >>>>>>>> >>> unexpected successes. (The former would trigger the lit script >>>>>>>> IIRC to >>>>>>>> >>> parse that as a failing-style result). >>>>>>>> >>> >>>>>>>> >>> The intent is this is a no-op on the test runner. >>>>>>>> >>> >>>>>>>> >>> On Wed, Dec 9, 2015 at 8:02 AM, Tamas Berghammer < >>>>>>>> tbergham...@google.com> >>>>>>>> >>> wrote: >>>>>>>> >>>> >>>>>>>> >>>> +Ying Chen >>>>>>>> >>>> >>>>>>>> >>>> Ying, what do we have to do on the build bot side to support a >>>>>>>> change in >>>>>>>> >>>> the default test result summary formatter? >>>>>>>> >>>> >>>>>>>> >>>> On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev >>>>>>>> >>>> <lldb-dev@lists.llvm.org> wrote: >>>>>>>> >>>>> >>>>>>>> >>>>> Hi all, >>>>>>>> >>>>> >>>>>>>> >>>>> Per a previous thread on this, I've made all the changes I >>>>>>>> intended to >>>>>>>> >>>>> make last night to get the intended replacement of test run >>>>>>>> results meet or >>>>>>>> >>>>> exceed current requirements. >>>>>>>> >>>>> >>>>>>>> >>>>> I'd like to switch over to that by default. I'm depending on >>>>>>>> the test >>>>>>>> >>>>> event system to be able to handle test method reruns in test >>>>>>>> results >>>>>>>> >>>>> accounting. >>>>>>>> >>>>> >>>>>>>> >>>>> The primary thing missing before was that timeouts were not >>>>>>>> routed >>>>>>>> >>>>> through the test events system, nor were exception process >>>>>>>> exits (i.e. test >>>>>>>> >>>>> inferiors exiting with a signal on POSIX systems). Those >>>>>>>> were added last >>>>>>>> >>>>> night so that test events are generated for those, and the >>>>>>>> >>>>> BasicResultsFormatter presents that information properly. >>>>>>>> >>>>> >>>>>>>> >>>>> I will switch it over to being the default output in a bit >>>>>>>> here. >>>>>>>> >>>>> Please let me know if you have any concerns once I flip it on >>>>>>>> by default. >>>>>>>> >>>>> >>>>>>>> >>>>> Thanks! >>>>>>>> >>>>> -- >>>>>>>> >>>>> -Todd >>>>>>>> >>>>> _______________________________________________ >>>>>>>> >>>>> lldb-dev mailing list >>>>>>>> >>>>> lldb-dev@lists.llvm.org >>>>>>>> >>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> -- >>>>>>>> >>> -Todd >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> -- >>>>>>>> >> -Todd >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > -- >>>>>>>> > -Todd >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> -Todd >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> -Todd >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> -Todd >>>>> >>>> >>>> >>>> >>>> -- >>>> -Todd >>>> >>> >>> >>> >>> -- >>> -Todd >>> >> > > > -- > -Todd >
_______________________________________________ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
