nlopes added a comment. In D98179#2639565 <https://reviews.llvm.org/D98179#2639565>, @davezarzycki wrote:
> In D98179#2639554 <https://reviews.llvm.org/D98179#2639554>, @nlopes wrote: > >> In D98179#2639491 <https://reviews.llvm.org/D98179#2639491>, @davezarzycki >> wrote: >> >>> In D98179#2639476 <https://reviews.llvm.org/D98179#2639476>, @nlopes wrote: >>> >>>> Why are timeouts important? Our use case is running Alive2 with the test >>>> suite. Alive2 is heavy machinery and runs into timeouts. Running the tests >>>> in roughly the same order every time is important to avoid timeout tests >>>> flipping to failed or vice-versa. Plus slow tests = tests that consume a >>>> lot of memory (in our scenario), so we can't bundle slow tests together. >>>> Adding a `--ignore-timing-data` would be great, yes! But I still feel that >>>> sorting the list of failed tests is important for user experience. I diff >>>> these all the time. >>> >>> That still sounds incredibly brittle. If there is any variety in test >>> machine performance, then any and all attempts at sorting should be futile >>> because the underlying hardware will perturb different timeouts. Is this >>> not your experience? How do you reconcile hardware performance and >>> configuration details (like SMT) with timeout settings? >> >> Of course it's brittle :) Changing from a time-based setting to a >> ticks-based system is ongoing work, such that resource exhaustion becomes >> deterministic. >> Nevertheless, on a same machine, we don't see many test flips. It's quite >> stable most of the times (just one test flip once in a while). > > This seems really beyond the scope and purpose of sorting the tests. > > If you don't mind and given that the workaround is trivial (delete the timing > data), I'd like to hold off on adding `--ignore-timing-data`. If enough > people complain then we can add that option. Is that okay with you? Sounds good. Thanks for adding support for the summary sort! Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D98179/new/ https://reviews.llvm.org/D98179 _______________________________________________ lldb-commits mailing list lldb-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
