On 01/07/2016 07:38 PM, Jim Rollenhagen wrote:
On Thu, Jan 07, 2016 at 07:18:15PM -0500, Jay Pipes wrote:
On 01/07/2016 06:42 PM, Jim Rollenhagen wrote:
On Thu, Jan 07, 2016 at 03:32:39PM -0500, Jay Pipes wrote:
On 01/07/2016 03:01 PM, Jim Rollenhagen wrote:
On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote:
On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:
Hi all,
A change to global-requirements[1] introduces mimic, which is an http
server that can mock various APIs, including nova and ironic, including
control of error codes and timeouts. The ironic team plans to use this
for testing python-ironicclient without standing up a full ironic
environment.
Here's the catch - mimic is built on twisted. I know twisted was
previously removed from OpenStack (or at least people said "pls no", I
don't know the full history). We didn't intend to stealth-introduce
twisted back into g-r, but it was pointed out to me that it may appear
this way, so here I am letting everyone know. lifeless pointed out that
when tests are failing, people may end up digging into mimic or twisted
code, which most people in this community aren't familiar with AFAIK,
which is a valid point though I hope it isn't required often.
So, the primary question here is: do folks have a problem with adding
twisted here? We're holding off on Ironic changes that depend on this
until this discussion has happened, but aren't reverting the g-r change
until we decide one way or another.
// jim
[1] https://review.openstack.org/#/c/220268/
What is the advantage of running another server like this over using
requests-mock (which is used by other OpenStack projects for testing
today)? The only difference here seems to be that you actually execute
requests code in one case and not in the other.
Requests-mock debugging when things go wrong seems a bit simpler.
This is less about twisted and more about trying to not introduce yet
another way to mock code in the tree that people need to understand.
-Sean
We'd be using this for functional tests, not unit, where we can't really
inject mocks. The idea is that we could run a full functional suite
against either mimic or a full ironic environment, just by changing a
test setting.
I don't really see the point of a separate project like Mimic that has a
whole bunch of reimplementations (mocked out) of all sorts of OpenStack (and
RAX-specific) API services. It's just a great way to introduce a larger
surface area for bugs to creep in -- since you have to keep the Mimic
interfaces up to date with the real interfaces. Better to keep something
like this -- if it is TRULY needed -- in-tree with the API service itself,
so that the chances of divergence are reduced. This is similar to the
fakevirt driver in Nova. It's in tree for good reason: when someone changes
the virt driver interface, the fakevirt driver goes boom and needs to be
changed in a corresponding fashion in the same patch.
I tend to think our REST APIs are much more stable than internal python
APIs, and so we can manage the divergence much easier.
Also, mimic can simulate:
* old versions (less needed with all the microversion stuff)
* old bugs that were shipped
* misconfigurations
* networking errors
* the passage of time (including timeouts)
We probably don't want to keep a catalog of old bugs and misconfigurations in
tree forever.
What value does a functional test against an HTTP API service that does
nothing (other than introduce greater surface area for bugs) actually offer
over unit tests anyway?
Testing the full stack of the client instead of mocking the bottom
half of requests is a big one.
Lekha and Glyph gave a talk on Mimic in Paris; it may be enlightening.
https://www.youtube.com/watch?v=HKUUQme3dwA
OK, I just watched that. Sorry, still don't see the value that Mimic
provides over unit testing the client interfaces and mocking out the HTTP
payloads so you have strict control over the expectations.
The problem that Glyph noted in the video about the unit test that mocked
out os.chdir to improperly return a value isn't solved whatsoever by Mimic,
so I find it odd that that example was used in discussing the value of
Mimic. Bad mocks are just that: bad mocks. The same false positive failure
could easily be introduced with a typo in the "metadata injection" that
Mimic does to inject failures into the system.
Mimic isn't testing anything on the server side at all so I'm not sure why
folks call it "integration testing". It isn't testing the integration of
anything at all. All it enables is multi-language client library testing,
and see my response to Ben, the surface area it introduces for bugs in the
test platform itself in my opinion outweigh the multi-language value it
might have.
FWIW, I've only been calling it functional testing. I also don't care
about the multi-language parts here. The sole purpose we have for mimic
(so far) is to functionally test python-ironicclient without mocking the
world in ironicclient.
Final question on this... if Mimic is *only* for testing clients, why not
make it just a dependency for python-ironicclient and not ironic itself?
We haven't made it a dep for anything yet, only added to g-r.
According to Dims, not to g-r, but to u-c, right Dims? Not sure if that
makes functionally any difference, though (pun intended).
However, now that you mention that, a really ambitious goal would be to
add a rabbit interface to mimic, and functionally test the API server
(that it sends the right messages, etc). Another would be to mimic
(Neutron, Glance) and test Ironic by itself.
So you would reimplement AMQP communication protocols using an in-memory
data store for queues. Sounds like an even greater surface area for bugs
to be introduced.
Last, I frankly don't understand why there's
such heavy opposition to the ironic team using an additional tool for
testing.
Since you asked, I'll be blunt. This isn't a personal attack on you,
Jim, though.
a) Because it fractures the testing and QA processes used by upstream
contributors that work on OpenStack projects by requiring them to learn
another system -- and one that potentially would require them to
understand a whole new surface area for potential bugs
b) Because it represents yet another RAX-driven divergence in the QA
space. CloudCafe took essentially all of the RAX folks that were at one
point working on Tempest and upstream QA and siloed them into a totally
different organization, in true RAX fashion. Instead of pulling the
OpenStack QA community along together, RAX QA continues to just do its
own thing and there's still bitterness on the tips of tongues.
<snip>
There was *more* than sufficient time for "I don't think this is useful"
to be posted on the review.
Sorry, this was the first I'd heard of it all.
> If anyone has a hard technical reason why
mimic should not be used to test an OpenStack project, I'm happy to
listen. Otherwise I'd like to work on actually getting things done.
I've listed my hard technical reason: it introduces, IMHO, too large of
a surface area for bugs to creep in to the testing process versus the
benefit it provides.
Best,
-jay
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
