On Thu, 21 Jul 2005, MikeM wrote:
Your comment presupposes that most of the bugs are specific to one piece
of hardware, I doubt that is a valid assertion. I would offer that most
of the bugs are not present in source code specific to a certain piece
of hardware, but are present in source code that is run across much of
the hardware that FreeBSD runs on. As such, it is just a matter of
setting up the correct QA testing scripts to catch the bugs.
Once a bug is reported, and that bug can be reproduced on the hardware
of the development team, then that bug should not reappear again,
because there should be a testing script written for it.
Additionally, every software bug is not only a defect in the software,
but it also represents a defect in the process that created the
software. Bugs should be looked at to analyze why they occurred, and
what in the process might be changed to prevent the same or similar bugs
from recurring.
Some of us have actually spent quite a bit of time looking at the defect
sets reported for 5.x. Depending on the release they fall into a number
of categories, but here are the major ones I've identified:
- ACPI-related hardware probe issues, especially in earlier 5.x releases
when the ACPI code (especially Intel vendor code) started knowing how to
work around common ACPI BIOS bugs. The source of these problems was
often that BIOS ACPI code contained work-arounds for Windows ACPI bugs.
Newer 5.x releases have blacklists of known bad BIOSes, workarounds for
bugs, etc, and this is a much less reported problem now. These problems
weren't present in 4.x because ACPI wasn't supported in 4.x; on the
other hand, there's a broad range of modern server hardware that now
requires ACPI to boot, so 4.x didn't run on that hardware, or supported
it poorly. After a very large effort, ACPI problems are massively
reduced.
- ATA problems. Many of these, while a symptom of bugs in the ATA code
running without Giant, were very specific to timing, or divergent/poor
ATA hardware. As a result, they were difficult to reproduce in any
environment but the original reporting environment. The same hardware
might perform fine in a FreeBSD developer's system. Many of these
problems have now been resolved, but some have not. Often as not, the
problems have to do with retrying requests to drives. As I mentioned,
we believe the ATA code in 6.x is much more resilient, but right now
what it needs is testing, not merging to 5.x yet. Fixes require just as
much testing as any other change, since a fix for one issue may well
trigger another issue, especially in the world of cheap PC hardware.
- Network stack stability under high load, especially on SMP. Many of
these bugs had to do with exercising timing and race conditions
"precisely right", and involved workloads not in the standard set of
testing performed. In many cases, those workloads have now been added
to the regression test suite. For example, there were a number of race
conditions relating to the closing of sockets and network stack teardown
in the protocols. These tended to turn up on systems running tens of
thousands of rapidly opening and closing TCP connections on SMP
hardware. Reproducing those conditions is difficult, and not something
most FreeBSD developers have the resources to do, so have to wait for
bug reports from people who do have those resources.
However, over the past 12 months we've been working to put together a
"netperf" test cluster, using hardware donated by a number of
organizations, including the FreeBSD Foundation, FreeBSD Systems,
IronPort Systems, as well as network connectivity and management donated
by Sentex Communications. This has allowed us to apply network tests in
higher performance environments, and make high end SMP hardware
available to a broader range of developers.
- Storage/file system related buffer starvation, deadlocks, etc, most a
result of the development of snapshots and bgfsck support, changes in
the I/O path, and so on. A number of these have turned out to be driver
bugs, but a fair number (especially in the 5.2 time frame) had to do
with resource management in the UFS code. Some still remain.
- Lock and resource leak crashes, especially with 5.2 and 5.3, when large
parts of the system moved from running under Giant to running without
it. Our process has definitely improved here, through improved lock
debugging tools, increased use of assertions, and the advent of things
like Coverity's static analysis tools being run over the source tree.
- ACPI-like problems having to do with migrating interrupt and hardware
configuration models. These usually manifest as interrupt storms. They
are required changes to support modern server class SMP hardware, but
often trigger bugs in a range of motherboard revisions from about 2-3
years ago. Sometimes, fixing these problems has required figuring out
how Windows does the same thing, as only the behavior used by Windows is
tested by the hardware vendor. Go figure.
- Threading bugs associated with the creation of a new threading model and
library, especially from 5.3. Many of these have now been resolved,
although further work on performance is on-going.
While there's a lot to be said for software engineering and improving
practices, and that we're working on that in a number of directions, I
think it is inaccurate to claim that software defects are always represent
solvable process defects. The reality is that all known software
development processes involve software defects, especially with complex
software systems. In operating system development, detecting bugs is
often a property of very specific environments that are hard or impossible
to reproduce or test in a formal way, at least not without huge expense.
Many bugs cannot be reproduced on hardware owned by developers, because
the cost (not to mention heat, time, and power constraints) would be
excessive. Many testing network environments require several computers in
order to reproduce even simple scenarios, as well as substantial
configuration work.
Combinatorics is also a practical reality, and doesn't work in our favor.
For example: Soren does most of our ATA development. Last I checked, he
had hundreds of ATA adapters, hundreds of storage devices, and a good
dozen or more test computers with various properties (speed, notebook or
not, SMP, hardware architecture, bus topology, BIOS revisions, and so on).
Assuming the general accuracy of the above numbers, and assuming it took
only five minutes per configuration to perform all testing, we're talking
about 83 days of non-stop 24-hour testing. Of course, the basic
assumptions are flawed, because it takes at least five minutes to set up a
hardware configuration, let alone run the hours of testing that you'd want
to run in the configuration. These are not problems that are solved
through test scripts, they are fundamental to the general problem of
testing device drivers and device driver frameworks. By doing a subset of
the above testing, you get software that is reliable on most hardware.
And indeed, that's what FreeBSD ships: software that is reliable on most
hardware.
A number of people have been working to improve FreeBSD testing. If
you're interested in working on it, as you obviously have clear views on
how it should work, we'd welcome your help. Especially if it comes to
defining test cases, producing scripts or test tools that can be run
mechanically, etc. There have been a couple of concerted efforts to
formalize our regression test framework, and in particular, to standardize
how tests are included, documented, and run. Most of these efforts peter
out, because the work is hard, offers few rewards, and the attention spans
of developers are limited by reality (jobs, etc). I think everyone is in
agreement that this is an area where there's more work to be done, and
contribution is welcomed.
Robert N M Watson
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
