On 6/9/11, Janis Johnson <jani...@codesourcery.com> wrote:
> On 06/09/2011 01:30 PM, Lawrence Crowl wrote:
>> On 6/9/11, Janis Johnson <jani...@codesourcery.com> wrote:
>>> On 06/08/2011 01:54 PM, Lawrence Crowl wrote:
>>>> On 6/6/11, Janis Johnson <jani...@codesourcery.com> wrote:
>>>>> On 06/03/2011 11:14 AM, Lawrence Crowl wrote:
>>>>>> The PPH project has tests that compile two different ways, and
>>>>>> then compare the assembly. If either of the compiles fails, the
>>>>>> comparison will fail. We'd like to simply not run the comparison.
>>>>>>
>>>>>> We currently have:
>>>>>>
>>>>>> set have_errs [llength [grep $test "{\[ \t\]\+dg-error\[\t\]\+.*\[
>>>>>> \t\]\+}"]]
>>>>>> # Compile the file the first time for a base case.
>>>>>> dg-test -keep-output $test "$options -I." ""
>>>>>>
>>>>>> if { $have_errs } {
>>>>>> verbose -log "regular compilation failed"
>>>>>> fail "$nshort $options, regular compilation failed"
>>>>>> return
>>>>>> }
>>>>>>
>>>>>> But that only stops subsequent actions when the test is known
>>>>>> a priori to have errors. How do we detect compilation errors,
>>>>>> so as to skip the remainder of the actions?
>>>>>
>>>>> Complicated GCC tests do this by using local procs instead of
>>>>> dg-runtest
>>>>> and dg-test. See, for example, gcc.dg/lto/lto.exp,
>>>>> gcc.dg/compat/compat.exp and gcc.dg/tree-prof/tree-prof.exp, which use
>>>>> lto.exp, compat.exp and profopt.exp from GCC's testsuite/lib. Those
>>>>> have scenarios in which further testing is skipped after a compile or
>>>>> link fails.
>>>>
>>>> So, I ended up changing the definition of fail from "reports test
>>>> failing" to "does not produce an assembly file". We really need
>>>> the latter for comparison, so it is the true measure. Once I made
>>>> that change in orientation, I was able to achieve what I wanted.
>>>>
>>>> The simple part is the regular compile.
>>>>
>>>> # Compile the file the first time for a base case.
>>>> set dg-do-what-default compile
>>>> dg-test -keep-output $test "$options -I." ""
>>>>
>>>> # Quit if it did not compile successfully.
>>>> if { ![file_on_host exists "$bname.s"] } {
>>>> # All regular compiles should pass.
>>>> fail "$nshort $options (regular assembly missing)"
>>>> return
>>>> }
>>>
>>> Don't use dg-test, use a new variant of it as is done with the lto,
>>> compat, and profopt tests. You'll have much more control that way
>>> and can better check the success of individual steps to decide what
>>> to do next.
>>
>> I am having trouble identifying the variant. Does it have a name,
>> or is it inline code?
>
> I meant that you should write one. For example, gcc.dg/lto/lto.exp
> uses lto_execute which is defined in lib/lto.exp, and
> gcc.dg/compat/compat.exp uses compat-execute defined in lib/compat.exp.
>
>>>> The complicated part is the compile we are comparing against, which
>>>> required knowing whether or not a compile failure is expected. For
>>>> that we grep for dg-xfail-if and the appropriate option.
>>>>
>>>> # Compile a second time using the pph files.
>>>> dg-test -keep-output $test "$options $mapflag -I." ""
>>>>
>>>> # Quit if it did not compile successfully.
>>>> if { ![file_on_host exists "$bname.s"] } {
>>>> # Expect assembly to be missing when the compile is an expected
>>>> fail.
>>>> if { ![llength [grep $test "dg-xfail-if.*-fpph-map"]] } {
>>>> fail "$nshort $options (pph assembly missing)"
>>>> }
>>>> return
>>>> }
>>>
>>> Relying on the existence of dg-xfail-if won't work when an expected
>>> compilation failure starts working, or an unexpected one arises.
>>
>> If the compilation starts working, I get an assembly file, and
>> continue to assembly comparisons. If the compilation fails,
>> but with a different error, then the other (typically dg-bogus)
>> directives should report the unexpected failure. In either case,
>> I think, I get proper notice. Am I missing something?
>
> I don't fully understand the purpose of grepping for dg-xfail-if.
> You shouldn't care about that within the test, just whether the
> compile succeeded or not. If there are messages that prevent the
> compile from succeeding, then they should be handled with
> appropriate test directives.
Each .cc file is compiled twice. First without PPH and then
with PPH. We compare the assembly generated as a proxy for "did
we compile correctly". The PPH code is in development, and so
several tests fail when compiled with PPH. We use the dg-xfail-if
to indicate when we expect the PPH compilation to fail. The specific
failure is indicated by other directives, usually dg-bogus.
The goal is to ensure that our tests are clean, either all pass
or xfail.
> Apart from that, whether or not a dg-xfail-if directives takes
> effect depends on the target and options used for the test.
Yes, we are counting on that. All the non-PPH compilations pass,
but only most of the PPH compilations pass. The others get the
dg-xfail-if.
> Just in case you're not aware of it (many people aren't), the
> test directives are documented in
> http://gcc.gnu.org/onlinedocs/gccint/Directives.html#Directive.
I had found that. One facility that seems to be missing is a
dg-bogus-if directive that enables bogus only when certain flags are
present. As a result, we get xpass messages on the non-PPH compiles.
I am working around that with a filter on the make output.
--
Lawrence Crowl
