On Mon, Jul 24, 2017 at 08:56:50AM +0200, Markus Armbruster wrote: > Cleber Rosa <cr...@redhat.com> writes: > > > On 07/21/2017 11:33 AM, Stefan Hajnoczi wrote: > >>> Output testing style delegates checking ouput to diff. I rather like it > >>> when text output is readily available. It is when testing QMP. A > >>> non-trivial example using this style could be useful, as discussing > >>> ideas tends to be more productive when they come with patches. > >> > >> Yes, I was considering how many of the Python iotests could be rewritten > >> comfortably in shell. It is nice when the test simply executes commands > >> and the output file shows the entire history of what happened. Great > >> for debugging. > >> > >> Stefan > >> > > I'd like to have a better understanding of the major pain points here. > > > > Although this can be seen as a matter of taste, style preferences and > > even religion, I guess it's safe to say that Python can scale better > > than shell. The upside of shell based tests is the "automatic" and > > complete logging, right? Running "bash -x /path/to/test.sh" will give > > much more *useful* information than "python -v /path/to/test.py" will, fact. > > > > I believe this has to do with how *generic* Python code is written, and > > how builtin functions and most of the standard Python libraries work as > > they do. Now, when writing code aimed at testing, making use of testing > > oriented libraries and tools, one would expect much more useful and > > readily available debug information. > > > > I'm biased, for sure, but that's what you get when you write basic tests > > using the Avocado libraries. For instance, when using process.run()[1] > > within a test, you can choose to see its command output quite easily > > with a command such as "avocado --show=avocado.test.stdout run test.py". > > > > Using other custom logging channels is also trivial (for instance for > > specific QMP communication)[2][3]. > > > > I wonder if such logging capabilities fill in the gap of what you > > describe as "[when the] output file shows the entire history of what > > happened". > > Test code language is orthogonal to verification method (with code > vs. with diff). Except verifying with shell code would be obviously > nuts[*]. > > The existing iotests written in Python verify with code, and the ones > written in shell verify with diff. Doesn't mean that we have to port > from Python to shell to gain "verify with diff".
Nb, not all the python tests verify with code. The LUKS test 149 that I wrote in python verifies with diff. I chose python because shell is an awful programming language if the code needs conditionals, non-scalar data structures, or is more than 10 lines long in total :-) Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
