Hi David,
On Fri, 28 Mar 2025 at 09:07, David Gow <david...@google.com> wrote:
> Thanks for sending this out: I think this raises some good questions
> about exactly how to handle long running tests (particularly on
> older/slower hardware).
>
> I've put a few notes below, but, tl;dr: I think these are all good
> changes, even if there's more we can do to better scale to slower
> hardware.
>
> On Fri, 28 Mar 2025 at 00:07, Geert Uytterhoeven <ge...@linux-m68k.org> wrote:
> > 2. Increase timeout by ten; ideally this should only be done for very
> > slow tests, but I couldn't find how to access kunit_case.attr.case
> > from kunit_try_catch_run(),
>
>
> My feeling for tests generally is:
> - Normal: effectively instant on modern hardware, O(seconds) on
> ancient hardware.
> - Slow: takes O(seconds) to run on modern hardware, O(minutes)..O(10s
> of minutes) on ancient hardware.
> - Very slow: O(minutes) or higher on modern hardware, infeasible on
> ancient hardware.
>
> Obviously the definition of "modern" and "ancient" hardware here is
> pretty arbitrary: I'm using "modern, high-end x86" ~4GHz as my
> "modern" example, and "66MHz 486" as my "ancient" one, but things like
> emulation or embedded systems fit in-between.
>
> Ultimately, I think the timeout probably needs to be configurable on a
> per-machine basis more than a per-test one, but having a 10x
> multiplier (or even a 100x multiplier) for very slow tests would also
> work for me.
Yes, adapting automatically to the speed of the target maachine
would be nice, but non-trivial.
> I quickly tried hacking together something to pass through the
> attribute and implement this. Diff (probably mangled by gmail) below:
[...]
Thanks!
> I'll get around to extending this to allow the "base timeout" to be
> configurable as a command-line option, too, if this seems like a good
> way to go.
>
> > 3. Mark rtc_time64_to_tm_test_date_range_1000 slow,
> > 4. Mark rtc_time64_to_tm_test_date_range_160000 very slow.
>
> Hmm... these are definitely fast enough on my "modern" machine that
> they probably only warrant "slow", not "very slow". But given they're
> definitely causing problems on older machines, I'm happy to go with
> marking the large ones very slow. (I've been waiting for them for
> about 45 minutes so far on my 486.)
>
> Do the time tests in kernel/time/time_test.c also need to be marked
> very slow, or does that run much faster on your setup?
Hmm, I did run time_test (insmod took (+7 minutes), but I don't
seem to have pass/fail output. Will rerun...
Indeed:
# time64_to_tm_test_date_range.speed: slow
Another test that wanted to be marked as slow was:
# kunit_platform_device_add_twice_fails_test: Test should be
marked slow (runtime: 30.788248702s)
I will rerun all, as it seems I have lost some logs...
> Is this causing you enough strife that you want it in as-is, straight
> away, or would you be happy with it being split up and polished a bit
> first -- particularly around supporting the more configurable timeout,
> and shifting the test changes into separate patches? (I'm happy to do
> that for you if you don't want to dig around in the somewhat messy
> KUnit try-catch stuff any further.)
This is definitely not something urgent for me.
Thanks!
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- ge...@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
