On Thu, 07/27 11:09, Daniel P. Berrange wrote: > On Thu, Jul 27, 2017 at 05:19:57PM +0800, Fam Zheng wrote: > > On Thu, 07/27 10:14, Markus Armbruster wrote: > > > This brings some advantages of "verify output with diff" to tests that > > > verify with code. Improvement if it simplifies the verification code. > > > > > > I'd still prefer *no* verification code (by delegating the job to diff) > > > for tests where I can get away wit it. > > > > Python based iotests can be (re)done in such a way that they print actual > > logs > > (interactions with qtest/monitor, stdout/stderr of QEMU, etc) instead of the > > current dot dot dot summary, then we automatically have diff based > > verification, > > no? > > The python test 149 that I wrote does exactly that. There's no reason why > the others couldn't do the same.
Yes, and IMO it should be the default and recommended way. > > > One thing I feel painful with bash iotests is how harder it is to write > > complicated test scenarios such as migration, incremental backup, etc. > > Yes, shell is an awful language if you need non-trivial control > logic or data structures > > > On the other hand the iotests are more difficult to debug when things go > > wrong > > because it eats the output which, if done with shell, should be very easy to > > get. > > Even if the python tests are not doing verify-by-diff, it should be fairly > easy to wire up an env variable IOTESTS_DEBUG=1 which would force them > to propagate stdout/err of all commands run (or at least save it to a > log file somewhere). Good point. Maybe we should extend the -d option of ./check for more log, if the above "full output" still isn't enough. Fam
