-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 07/01/2011 02:24 AM, Paul Larson wrote:
> On Thu, Jun 30, 2011 at 11:24 PM, Daniel Lezcano
> <daniel.lezc...@linaro.org>wrote:
>
>> When all tests will be finished I wish to switch the new way test suite
>> execution in lava, if it is possible.
>>
> Given that the old tests are broken at the moment and disabled, any reason
> we shouldn't switchover now?
Ok, let's switch. Expect the messages output I have to change, is there
something I should change to integrate with lava (eg. for the 'make
check' invocation) ?
>> Will the following format be ok ?
>>
>> test_01/cpu0 : checking scaling_available_frequencies file ... PASS
>> ...
>
> That would probably translate internally to something like:
> {test_id="test_01_cpu0", message="checking scaling_available_frequencies
> file", result="PASS"}
> Is that ok? Seems like something we should have no trouble making sense out
> of later I think. Also, the exact output is saved as an attachment as well.
That sounds good.
>> The result for a test case is PASS or FAIL.
>>
> We support "unknown" as a result as well, if that helps you at all.
Ok.
> Sometimes results can be indeterminate, or unimportant (odd as that may
> sound, sometimes the message is really what you're after, as illustrated by
> some of the previous pm qa tests)
>
> But under some circumstances, we need to do some extra work where a
>> failure does not mean the test case failed but the pre-requisite for the
>> test case is not met.
>>
> ...and we also support "skip" as a result. That seems like a correct use
> for it.
Ok.
> You don't have to report it literally as "skip", you can call it "oink" for
> all we care, and just provide a translation table that converts whatever
> your result strings are to {pass, fail, skip, unknown}. For example, see
> the test definition for ltp.
>
>
>> deviation 0 % for 2333000 is ... VERY
>>>> GOOD
>>>
>>> Same comments as above about having an easier to interpret format, but
>> the
>>> result here: "VERY GOOD" - what does that mean? What are the other
>> possible
>>> values? Is this simply another way of saying "PASS"? Or should it
>> actually
>>> be a measurement reported here?
>>
>> Yep, I agree it is an informational message and should go to a logging
>> file. I will stick to a simple result 'PASS' or 'FAIL' and I will let
>> the user to read the documentation of the test in the wiki page to
>> understand the meaning of these messages (GOOD, VERY GOOD...).
>>
> keeping to the template you mentioned earlier, I wonder if we could do
> something like this:
> deviation_0_for_2333000: VERY GOOD ... PASS
> (are the numbers there consistent and useful as a test case name? I'm
> assuming so here)
> That would allow you to capture "VERY GOOD" as details in the message (one
> of the fields we keep). Also, your test could be smart enough to know that
> good, or verygood = pass, while bad, verybad = fail. Possibly I'm making a
> lot of assumptions here, but I think you see what I mean.
Yes, I think you are right. It should be better presented like that.
> Keeping to a consistent results format in your output is a good practice,
> and makes this *much* easier for capturing the data important to you. Of
> course, anything is possible. We have some tests with rather elaborate
> results and for those, the test definition just inherits from a base class
> and defines it's own parser. If you're not feeling very pythonic, you could
> provide your own parser as part of the test download written in shell, c,
> ruby, go, whatever, that just acts as a filter, then have it just read it
> all in directly. There are lots of options, but I'm of the opinion that a
> consistent format makes it easier on humans looking at it as much as machine
> parsers. And since you have control over that, easiest to do it now. :)
Agree :)
I will rework the tests and resend.
Thanks Paul
-- Daniel
- --
<http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs
Follow Linaro: <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJODYCOAAoJEAKBbMCpUGYAcdkIAMitOMR1b8mY/hwmiTNTJ1/5
JrZdIW+FFFY8tGuFF7Jq7KTiRURNYo4sXP+fsK3pOMxtH8RgTpRytM2Y3fXMMaaC
8CHnkYFwAAIxmIKBKiIhGLOgqT3VUqU9nxkAJBEzFX09FaysarmN92PNaO5hSXXI
XOSpbflWm4JsT0rUVLVzxAHYMDqCqpLZkTLfBXi0nqowAp4iOUqTyB2/WWzjMn+m
5PfWVOezjiduUECZ4ssiGZGm7usIzw0AX4RWqR5DDPaJIUdCfqp+0nvIas+/UBcF
fF4H5EDMdharAZEg6VNRWr/487d59AOlWuITq5GHYBRP4zMaYgYEFXApnEECgzQ=
=J9F8
-----END PGP SIGNATURE-----
_______________________________________________
linaro-dev mailing list
linaro-dev@lists.linaro.org
http://lists.linaro.org/mailman/listinfo/linaro-dev
