> The original example that you came up with was a false negative,
> namely a missing object that stayed unnoticed. Now we're discussing
> all kinds of complicated algorithms to reduce the probability of
> false negatives, while also trying to avoid false positives. My
> question is: Do we really have a problem with false positives?
Yes, it can happen that a code change suddenly make notes visible that
were suppressed before.
> I had a quick glance on the last MRs that had their artifacts still
> available and found no example except !2391 that had scores "below
> threshold". Does this happen frequently?
'Below threshold' happens quite often, yes.
> If not, wouldn't it suffice to improve on the sensitivity of the
> comparison process and not introduce stuff that tries to
> discriminate between "good" and "bad" changes? We could render
> tests with unclear results a second time with higher resolution,
> e.g.
In almost all cases, a higher resolution doesn't help with the problem
at hand, namely the appearance or disappearance of objects in
regression tests that accidentally go unnoticed.
I ask you and others to not complicate the issue. What's really
needed (and nothing more) is an improved algorithm that can be either
used as an addition or as a replacement for the current one: In
comparison to what we currently have, it should yield smaller demerits
for slightly shifted images and/or objects and much larger demerits
for appearing or disappearing objects.
It doesn't make any sense right now to muse about completely changing
the process.
Werner
