Hi, seems that Gitlab CI is currently configured to show the green "Success" checkmark for pipeline runs even if unit tests are failing. Reasons seems to be that there Gitlab only knows Yay or Nay, without the warning state level known from Jenkins. And given that quite some projects (sadly) maintain a few long-time failing unit tests, having the pipeline fail on unit tests seems to have been seen as too aggressive
This of course harms the purpose of the unit tests, when their failures are only noticed weeks later, not e.g. at MR discussion time. Seeing how at least in KDE Frameworks first regressions sneaked in without someone noticing (nobody looks at logs when the surface shows a green checkmark, e.g. kcoreaddons, kwidgetsaddons, kio, purpose, krunner on openSUSE and possibly more have regressed in recent weeks, see build.kde.org) this should be something to deal with better, right? Bhushan gave two first ideas just now on #kde-sysadmin: > Well we can add a switch that repos can commit to saying test failure is build failure > Another alternative is we use bot to write a comment on MR IMHO, to give unit tests the purpose they have, we should by default to let test failures be build failures. And have projects opt out if they need to have some unit tests keep failing, instead of e.g. tagging them with expected failures or handling any special environment they run into on the CI. Your opinions? Cheers Friedrich
