On 1/6/2021 6:10 AM, Volker Hilsheimer wrote:
Hi Max and Adam,
What can do better to avoid such regressions from making it into a release, or
preferably into the code, in the first place? Nobody, not even the Qt Company
management :P *wants* to release crappy quality on time.
What we know about those bugs is that they passed all code reviews, and didn’t
get caught by any of the thousands of tests we run for every change on half a
dozen platforms. And we know that the only way they were discovered is real
users testing real applications against the released version of Qt.
So, what we have is clearly not good enough, but if the last 15 years of
writing unit tests etc hasn’t gotten us to a better place, then maybe “more of
the same” can’t be the only strategy.
Is your experience that we release stuff “come hell or high water" in spite of
severe bugs being reported during beta testing? We do spend a lot of time triaging
incoming bug reports, and a severe enough bug can always block a release.
Or do we not discover the issues until the .0 release because few people test
the pre-releases? That seems to be supported by the data we have about
downloads and general activity in response to pre-releases.
Volker
Hi Voker,
Thanks for your thoughtful response. You bring up good points, all of
which certainly merit serious discussion.
Honestly I don't care what the releases are called or how they're
numbered. The main issue right now is the locking down of stable
release branches. It should be no surprise that this is "still" a
sensitive issue and calling Qt6 the next usable version (or however it's
stated officially) is pretty much rubbing salt in the wounds. And who
knows what's next.
As for getting people to test things... obviously there's no magic
bullet here. Any project which relies on their users to test
functionality is going to have this issue. These days this seems to
include just about everything, like commercial applications, firmware,
and airplanes. I think maybe people are getting tired of it and burnt
out. Also there's only so much time in a day. So on one hand it's a
popularity contest... how to get users excited to be your testers? OTOH
it's a catch-22 because the obvious way to get users excited is with New
Features, which may not leave much time or motivation to fix stuff that
doesn't seem vital. But users who don't need the exciting NFs may not
bother. So a few versions go by before those others catch up and then
start discovering new bugs from several version ago. At that point
you're lucky to get a P3, because hey, no one noticed this for over a
year, "obviously" not a big deal.
I would also say it's discouraging when there are already 5K+ open P2+
QTBUGs in the system. Double that with P3. Some going way back and
still, seemingly, relevant. (e.g. a P1 from 2014: "Affects Version/s:
5.4.0 Alpha, 5.6.2, 5.9, 5.12.3, 5.14") This was brought up here just
recently, of course.
Even if half of them are irrelevant now, this is time people spent doing
"half" the work of finding the bug in the first place. Stuff unit tests
obviously couldn't/didn't catch, for whatever reason. In some cases
there are even patches, or suggested fixes, or pointers to exact source
of the issue. So one could argue that there's already plenty to work on
before, say, new features or higher version numbers.
Look, when one can control the severity level of an issue, and one knows
a certain level will block the on-time release, it seems logical that
one gets to pick and choose what constitutes "severe" or "critical" or
whatever. It's a circular argument. "Well, we don't want to delay
release _just_ for this, so it's not critical." Someone (or group) gets
to make the choice, any way you look at it. Which is fine really,
someone has to drive the boat. It's inevitable that some of the
passengers like the direction, and some might not. And some won't know
until they get there... :-) Whether the drivers are happy with the
outcome, and who they care to listen to on the way, is up to them.
Personally I prefer to clear out backlogs of known issues before
building further upon that foundation. I mean real issues, not feature
requests or nice-to-haves, or pixel-perfect everywhere. Fresh paint on
a house with crumbling walls will only last so long.
Thanks again for listening,
-Max
_______________________________________________
Development mailing list
[email protected]
https://lists.qt-project.org/listinfo/development
