On Mon, Mar 14, 2022 at 4:44 PM Eduardo de Souza Cruz < eduardo.c...@kdemail.net> wrote:
> Hi, > > Regarding the krunner timer-based test which I authored: > > We could just delete the few lines that are enforcing those upper-limit > timeouts and add some comments explaining that in real life those timeouts > should have been met, I don't think we need to delete the entire test. I > was explicitly asked to write a test when I submitted the MR to enforce > that this functionally wouldn't regress in the future. > > Also, I'm wondering... if we happen to have a compilation directive that's > #defined in this CI environment only, or some framework function that > returns weather we are in the CI environment or not, we could put some > #ifndef (or if's) on those few upper-limit timeout lines and avoid > compiling/running just those lines in the CI environment. I'm not familiar > with this environment so I don't know if there is such a thing, I'm just > wondering... > Not at this time i'm afraid. There are some environment variables you can look to but those differ between our Jenkins and Gitlab setups. > > I'm available to submit a quick MR if you'd like me to, just give me some > directions for what you'd like me to do. > > [], > Eduardo > Cheers, Ben > > ------------------------------ > *From:* Ben Cooksley <bcooks...@kde.org> > *Sent:* Sunday, March 13, 2022 1:53 PM > *To:* KDE Frameworks <kde-frameworks-devel@kde.org> > *Cc:* eduardo.c...@kdemail.net <eduardo.c...@kdemail.net> > *Subject:* Re: Unit tests all pass in Jenkins on Linux > > On Mon, Mar 14, 2022 at 4:40 AM David Faure <fa...@kde.org> wrote: > > After the recent discussions on state of CI, I fixed the last unittest > failures (kio, purpose... + apol fixed ECM) so that > https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/ > is all green^H^Hblue again. > Please keep it that way! > > > Thanks for looking into and fixing all of these David. > > > > Note however that > > * kwayland has a flaky test: > > > https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/job/kwayland/job/kf5-qt5%20SUSEQt5.15/171/testReport/junit/projectroot.autotests/client/kwayland_testDataDevice/ > > FAIL! : TestDataDevice::testReplaceSource() Compared values are not the > same > Actual (selectionOfferedSpy.count()): 1 > Expected (2) : 2 > Loc: [autotests/client/test_datadevice.cpp(557)] > > Who can look at this one? git log mostly shows Martin Flöser < > mgraess...@kde.org> > who I think isn't active anymore? > > > Not sure if it applies to KWayland as well, but I know that KWin has load > sensitive tests (which is why the Gitlab .kde-ci.yml files support the flag > tests-load-sensitive) > If this test appears to be flaky, then it is quite possible that it is > load sensitive as well. > > > > * krunner has a flaky test [2] because it measures time spent and expects > small values like 65ms > (I changed that one to 100ms), 250ms, 300ms. With only 10% safety margins. > On a busy CI system, > this is bound to fail regularly, even with bigger safety margins. In my > experience this kind of test > is just not possible (we're not running on a real time OS), I vote for > removing the test. > CC'ing Eduardo. > > > https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/job/krunner/job/kf5-qt5%20SUSEQt5.15/325/testReport/junit/projectroot/autotests/runnermanagertest/ > > > Yes, that will definitely fail more often than not - your only way to make > sure tests like this pass on our CI system is to > set tests-load-sensitive=True (in Gitlab CI) > Note however that option should be avoided where possible as it means your > build will stop and wait for load to fall to low levels before proceeding > with running tests - which blocks a CI worker slot from being used by > another project. > > I'd also be in favour of removing this test. > > > > -- > David Faure, fa...@kde.org, http://www.davidfaure.fr > Working on KDE Frameworks 5 > > > > > Cheers, > Ben >
