> The community block I/O test suite is qemu-iotests: > http://git.kernel.org/?p=linux/kernel/git/hch/qemu-iotests.git;a=summary > If you have tests that you'd like to contribute, please put them into > that framework so other developers can run them as part of their > regular testing.
Hi Stefan, What I described is not a qemu-io test case. I also use qemu-io, which is very helpful, but I observed that qemu-io has several limitations in discovering elusive bugs: B1) qemu-io cannot trigger many race condition bugs, because it does not fully control the timing of events. For example, qemu-io cannot test this scenario: three concurrent writes a, b, and c are processed by bdrv_aio_writev() in the order of Pa, Pb, and Pc; their writes are actually persisted on disk in another order of Wc, Wa, and Wb; and finally their callbacks are invoked in yet another order of Vb, Vc, and Va. Some race condition bugs may exist in the code (e.g., inappropriate locking), because it does not anticipate these orders of events are possible. This is just one example. In theory, there can be 100 concurrent reads or writes, and their events can happen in an arbitrary permutation order. It is nearly impossible to manually generating test cases for all of them. B2) Even if a race condition bug is triggered by chance, its behavior depends on subtle event timing that is hard to repeat and hence hard to debug. B3) With qemu-io, it is hard to test code paths that handle I/O failures. For example, a disk write may fail due to disk media error. Because these errors are rare, the failure handling code paths may never be tested, which for example may contain a null pointer bug that can crash the entire VM or gradually leaks resources (e.g., memory) due to incomplete cleanup. B4) qemu-io requires manually creating test cases, which is not only time consuming but also leads a low coverage in testing. This is because many bugs happen in scenarios that the developers do not anticipate, and hence do not know how to create test cases in the first place. The FVD patch includes a new testing framework that addresses the above issues. This testing framework is orthogonal to FVD and can be used to test other block device drivers as well. This testing framework includes two components that can be used both separately and in a combination T1) To address the problems of B1- B3, I implemented an emulated disk in block/sim.c, which allows a full control of event timings, either manually or automatically. Given the three concurrent writes example above, their 9 events (Pa, Pb, Pc, Wa, Wb, Wc, Va, Vb, and Vc) can be precisely controlled to be executed in any given order. Moreover, the emulated disk can inject disk I/O errors in a controlled manner. For example, it can fail a specific read or write to test how the code handles that, or it can even fail as many as 90% of the reads/writes to test if the code has resource leaks. qemu-io is extended with a module qemu-io-sim.c to work with the emulated disk block/sim.c, so that the tester can use the qemu-io console to manually control the order of events or fail disk reads or writes. T2) The solution in T1 still does not address the problem of B3), i.e., manually generating test cases is time consuming and has a low coverage. This problem is solved by a new testing tool called qemu-test. qemu-test can 1) automatically generate an unlimited number of randomized test cases that, e.g., execute 1,000 concurrent disk reads or writes on overlapping disk regions; 2) automatically generate the corresponding anticipated correct results, automatically run the tests, and automatically compare the actual test results with the anticipated correct results. Once it discovers a difference, which indicates a bug, it halts testing and waits for the developer to debug. The randomized test cases created by qemu-test are controlled by a pseudo random number generator, and hence the behavior is completely repeatable. Therefore, once a bug is triggered, it can be precisely repeated for an unlimited number of times to facilitate debugging, even if this bug happens extremely rare in real runs of a VM. qemu-test is fully automated. Once started, it can continuously run, e.g., for months to test an enormous number of test cases. The implementation of qemu-test is actually not that complicated. It opens two virtual disks, the so-called truth image and test image, respectively. The truth image is served by a trivial synchronous block device driver so that its behavior is guaranteed to be correct. The test image is served a real block device driver (e.g., FVD or QCOW2) that we want to test. qemu-test submits the same sequence of disk I/O requests (which is randomly generated) to the truth image and the test image, and expect that the two images’ contents never diverge. Otherwise, it indicates a bug in the test image’s block device driver. qemu-test works with the emulated disk block/sim.c so that it can randomize event timings in a controlled manner and can inject disk I/O errors randomly. I found qemu-test extremely powerful in discovering elusive bugs that I never anticipated, and using qemu-test is effortless. Whenever I completed some major code upgrade, I simply started qemu-test in the evening and came back in the morning to collect bugs, if any. Debugging them is also easy because the bugs are precisely repeatable even if they are hard to trigger. As for the QCOW2 bug I mentioned previously, it can be triggered by test-qcow2.sh. A faster way to trigger it is to bypass those correct test runs by executing the commands below: dd if=/dev/zero of=/var/ramdisk/truth.raw count=0 bs=1 seek=1155683840 dd if=/dev/zero of=/var/ramdisk/zero-500M.raw count=0 bs=1 seek=609064448 ./qemu-img create -f qcow2 -b /var/ramdisk/zero-500M.raw /var/ramdisk/test.qcow2 1155683840 ./qemu-test --seed=116579177 --truth=/var/ramdisk/truth.raw --test=/var/ramdisk/test.qcow2 --verify_write=true --compare_before=false --compare_after=true --round=100000 --parallel=100 --io_size=10485760 --fail_prob=0 --cancel_prob=0 --instant_qemubh=true As for the FVD patch that includes the new testing framework, I tried to post it on the mailing list twice but it always got bounced back, either because the message is too big or because of a Notes client configuration issue. Until I figure it out, please down the FVD patch from https://researcher.ibm.com/researcher/files/us-ctang/FVD-01-14-2011.patch . Best regards, ChunQiang (CQ) Tang, Ph.D. Homepage: http://www.research.ibm.com/people/c/ctang
