On Fri, 01 Nov 2013 13:45:08 +1000 Nick Coghlan <ncogh...@redhat.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 11/01/2013 03:53 AM, Tim Flink wrote: > > On Thu, 31 Oct 2013 07:17:03 -0400 (EDT) Josef Skladanka > > <jskla...@redhat.com> wrote: > >> I do not really feel that this is a good match for our needs (at > >> least without any significant hacking) > > > > Yeah, this matches what I remember finding when I looked into this > > a couple of months ago. It looks like a better protocol overall > > but it's going to be quite a bit more upfront work to support than > > TAP. I know that openstack's test suite uses subunit for nova and > > neutron [1] > > > > [1] https://wiki.openstack.org/wiki/Testr > > > > Subunit has been recently added to the fedora repos but I suppose > > that's of little utility for us if we end up having to re-implement > > parts to get away from UnitTest. > > As Tim noted, subunit development is tied in to the OpenStack > project's "test repository" (testr) project. To add confusion, they're using an upstream project called testrepository: https://testrepository.readthedocs.org/en/latest/MANUAL.html I remember running into problems actually running testr a month or two ago but I got pulled away by something else and never finished looking into it. I just ran it again and it didn't have any issues, so I'm either mis-remembering or my OS upgrade since then ended up fixing the issue. > So, if you were going to go down the subunit path, I'd suggest looking > at leveraging (at least pieces of) testr for result storage as well. > > While it's not an urgent requirement, we also plan to explore adding > subunit support to Beaker. Ah, I seem to recall talking about that at Flock and it slipped my mind. Thanks for the reminder as this could end up being important for integration. > Robert Collins gave a decent talk on testr (et al) at the PyCon AU > 2013 OpenStack miniconf: https://www.youtube.com/watch?v=Oe_HhBBbqbw > > Looking at the list of Producers and Consumers on the TAP wiki doesn't > give me any confidence that *anyone* has successfully used TAP at > scale or as part of a distributed continuous integration system, nor > does it look like it has anyone providing the level of investment HP > and others are putting into the OpenStack testing infrastructure > (Robert works for HP, mostly on the OpenStack-on-OpenStack project). I think that AMD uses or was using TAP but yeah, I'm not really seeing much of anyone else who is using TAP on a large scale. My main concern is how tightly coupled SubUnit and testrepository are to UnitTest. I took another quick look at their code and one of the first things I noticed is the description in a header [1]: # subunit: extensions to Python unittest to get test results from subprocesses. There are other things that raise flags for our usage at first glance, but I only looked at it for a few minutes and could easily be very wrong :) I'm intrigued by some of the things that subunit can do/support (embedding files and the idea of replaying tests in particular - I think they have the potential to solve some problems we're eventually going to hit). I think that the short term question may end up being whether we want to focus on better stuff from the beginning or to "rough in" semi-disposable components (knowing or ignoring the fact that those are the types of things which tend to live longer than they should) to get a better idea of where the unknown pain points are. I can't really argue with xUnit's success as a unit testing paradigm but I'm not crazy about getting too locked into that right now. One of the goals that I have for taskbot is to make it flexible enough to be able to do stuff that isn't traditionally in the domain of a test system. However, I'm not quite crazy enough to try being everything to everyone so that goal may not end up feasible. [1]http://bazaar.launchpad.net/~subunit/subunit/trunk/view/head:/python/subunit/v2.py > Even if subunit is more work initially, you'll likely enjoy flow-on > effects in the future from those OpenStack investments (similar to the > way LMR decided to outsource the "inventory management and scaling to > multiple labs" problems for autotest to Beaker, so autotest gets to > avoid doubling up on work we were going to do anyway). Assuming that what we're trying to do is similar enough, yes. If it's different enough that we end up modifying their code or re-implementing it ourselves, that big investment could end up being a pain point. The documentation states that v2 is not yet a stable protocol and current implementers should be prepared to deal with non-backwards-compatible changes. I'm not that familiar with what exactly OpenStack is doing with subunit and testr, so I can't say how much overlap there actually is. > With subunit as the more expressive protocol, it also means it would > be easier to convert TAP streams to subunit streams than the other way > around, so building the back end to be subunit based doesn't rule out > the possibility of permitting TAP based input in the future. By > contrast, using TAP as the base *will* almost certainly rule out > accepting subunit streams from testr or Beaker. Yeah, it does sound like this topic needs more investigation and maybe some example/poc code. Most of my concerns will end up being academic if using subunit ends up being easier than or at least no more difficult than the alternatives. Thanks for the discussion, I really appreciate your input. This has shaken loose a few things from the back of my mind from when I was looking into this a couple of months ago before handing the task off to Josef. Tim > Regards, > Nick. > > - -- > Nick Coghlan > Red Hat Infrastructure Engineering & Development, Brisbane > > Testing Solutions Team Lead > Beaker Development Lead (http://beaker-project.org/) > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.14 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQEcBAEBAgAGBQJScyPEAAoJEHEkJo9fMO/LR8QH/iLHZCA0f87FT1nWzZe9UKOM > H0V2Pd+NdubHrHruxsNAhzwdfgZhc3JhW+515hR53LkYti5XlceP/cqUtS1o2YfQ > U2qFjkEJ1Fu4+jTVD6a7XO1CejeDHN9SJ+lAKQr+yw7lzCrsmb+GkfMkYPhw/8/5 > 4B0S9soOCs9Jasl3PwhFvx+1eso3r/+ZYUt6w8IpcDM4PSHhgjoZdQ/Xg7Tmi/js > 3qcpEDS9eJCpKJ1KlCG01gFEAxmKBC86dku/Rb3Dy7FYPCBwOFkerYAD89QLaPrX > 1pEbIKM77F0e6FoxthEjweALAztdI20uuH0mYisd6HfQAThPPyzOmjuRNY9VJVE= > =pNIi > -----END PGP SIGNATURE----- > _______________________________________________ > qa-devel mailing list > qa-de...@lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/qa-devel
signature.asc
Description: PGP signature
_______________________________________________ qa-devel mailing list qa-de...@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/qa-devel
