On 22/10/2015 08:06, Alex Harvey wrote:
On 21 October 2015 at 23:28, Gareth Rushgrove
<gar...@morethanseven.net <mailto:gar...@morethanseven.net>> wrote:
On 20 October 2015 at 08:49, Alex Harvey <alexharv...@gmail.com
<mailto:alexharv...@gmail.com>> wrote:
> Hi all,
>
> I am investigating whether or not I can use Beaker to do
acceptance testing
> on roles and profiles.
>
> I've had a look at Liam Bennett's excellent blog posts -
>
http://tech.opentable.co.uk/blog/2014/09/01/testing-puppet-with-beaker-pt-dot-3-testing-roles/
>
http://www.slideshare.net/liamjbennett/cfgmgmt2015-testing-with-beaker
>
> I need to handle a situation in my tests where, say, a role that
I am
> testing will apply a base class which will cause the node, for
instance, to
> join a FreeIPA domain. But I don't want Beaker to actually
build a FreeIPA
> box. And I don't want my short-lived node to join a real
FreeIPA domain.
>
> I would hope that Beaker could either build Mock Services
> https://en.wikipedia.org/wiki/Mock_object
>
> Or better still, tell Beaker to expect the base class to try to
apply the
> FreeIPA class, and just pretend it succeeds. Just as you can
stub out
> methods in rspec etc.
>
> Has anyone done anything like this before?
>
You can actually do something like that :)
Beaker is going to run you Puppet code on the node(s) it spins up,
probably using apply manifest like so:
apply_manifest(pp, :catch_failures=>true)
If this sees an error while doing so (because your FreeIP class
doesn't apply) then it will throw an error. However...
If you check the API docs you can see:
http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Helpers/PuppetHelpers#apply_manifest_on-instance_method
apply_manifest(pp, :accept_all_exit_codes => true)
This will run your puppet code and not throw an error.
Now, whether this works for you really depends on if any other classes
depend on the FreeIPA class. And you can't easily test for idempotent
runs easily with this.
Another approach if you can change the puppet code is to introduce a
switch in the profile, for instance if a fact exists like MOCK_FREEIPA
you could not include the class. And then drop that fact into your
beaker test environment.
Also see the other points about testing at different levels but you
can absolutely do it in your acceptance tests.
Thanks Gareth.
Both of these solutions are a bit messy but it is good to know of all
the options. One of the big issues I would encounter immediately I
used :accept_all_exit_codes is that the FreeIPA code takes several
minutes to time out if the server can't be contacted. Or if I moved
all this logic associated with testing into the code, that would work,
but it is not really where test logic belongs, and it would rather
clutter my base classes.
I feel that there is some space here for a new approach to acceptance
testing. I am not sure that I agree that acceptance tests should be
end-to-end tests. Consider:
* spinning up a FreeIPA system - not to mention all the other
external services that a base class might depend upon - takes a
long time. The longer tests take to run, the less likely people
will actually run them.
* not all developers will have workstations that can run such
resource-intensive tests.
* I actually don't want to test that every single role can join the
FreeIPA domain - I only want to test that once, and preferably in
tests associated with my FreeIPA modules and classes.
Another approach might be to could create a long-lived acceptance test
environment that only contains a DNS, FreeIPA, etc, however:
* this would introduce a management burden - sysadmins needed to
keep them running, clean up junk related to broken tests etc.
* It would also introduce a cost - we would have to pay Amazon to
always have these instances running.
I actually think deploying into a real UAT environment is the only
real end-to-end test of the code that matters before it goes to
production.
Now, if I am testing a web server role, rspec-puppet allows me to do
very useful testing on catalogs - but if I am testing a front end web
server role, I only want to know that it builds a properly configured
Apache. I suppose, therefore, that I could just test the front end
web server profile; although it would be good to also test as much of
the integration as is practical.
What if Beaker was extended to provide a framework for provisioning
fakes - if it accepted plugins somehow for a mocked up FreeIPA service
etc. Would that be unrealistically complicated? In the development
world, developers must be mocking up these sorts of fakes all the
time. How do they do it - do they just continually reinvent the wheel
or are there tools for this?
fakes/mocks in s/w development have a very limited scope: usually only
one or two methods with pre-defined return values. If you really do not
want to test integration with a certain service, perhaps making that
explicit and having a test version of a profile that does not include
the FreeIPA support makes sense? It would also ensure that other
services (like SSO on the apache ??) do not depend too heavily on FreeIPA.
It reminds me a bit of Aspect Oriented Programming, where cross-cutting
concerns are extracted from the code and re-injected by the compiler in
a uniform way across the board.
Cheers, D.
--
You received this message because you are subscribed to the Google Groups "Puppet
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/5628A835.5040701%40dasz.at.
For more options, visit https://groups.google.com/d/optout.
