I've been thinking about using Hudson for this, so its good to know that there are others out there doing this.
3 projects that I've bookmarked to look at for unit testing frameworks for the OS are: https://fedoraproject.org/wiki/AutoQA http://autotest.kernel.org/ http://www.linux-kvm.org/page/Main_Page I haven't had time to do much more than scan the web site, but they seem to be worth watching. I've also started including test.pp scripts as part of each module. These scripts check directories or do some execs (i.e. wgets, pings, lookups). These scripts then run with every puppet update push. However, this isn't ideal for production and there needs to be a better way to trigger the running of unit tests. But it has been helpful in validating iptables configuration and ensuring services and their clients are on the same network. On 11/8/09 7:43 AM, Julian Simpson wrote: > I use Hudson. > > I've used Test::Unit to wrap validations of generated config files in > the past. I want to know as early as possible if something isn't > going to work. > > Functionally testing the config from the outside in might be easy but > slow. You can deploy to a VM and actually test it using the goodness > of cucumber. I'm still looking for the equivalent of unit tests: how > do you quickly test function from the inside to give enough confidence > and a fast feedback loop? Maybe we need to build out Patrick's > approach with a library of command-line tools to verify that > everything works. > > J. > > > 2009/11/7 Isaac Christoffersen<isaac.christoffer...@gmail.com>: > >> I'd be interested in hearing more about this. I've been working on a >> project in which we do a lot of automated server builds and >> configuration using various OSS tools, including puppet. However, one >> thing that I've been looking for is a way to write a set of test suites >> to validate the server configuration. The combination of TDD and BDD >> with server rollouts is very attractive. >> >> I'd also be curious to know what CI server people are using to help >> coordinate these intermediate builds and tests. >> >> >> >> On 11/7/09 11:00 AM, Patrick Debois wrote: >> >>> for what it's worth: >>> >>> for a while I was experimenting with cucumber to test scripts running >>> on linux machines. And yes , IMHO it is the way to go . >>> >>> It worked well with the behavior testing just a you guys described. >>> still just as in regular development, bdd can be complemented with >>> tdd. >>> The way i did the tdd, was by using the exit codes of the shell >>> commands. I created an abstraction in ruby for execute (local, remote) >>> and upload, download of files. >>> >>> If the command did not have exit 0 (the default) then i imagined the >>> test being failed. >>> Using snapshots of the virtual machines, I ran the scripts against a >>> virtual machine (used virtualbox), with the ability to do a rollback. >>> I know this is not the way puppet normally operates, but I used the >>> (if i recall correctly) the puppetrun command to execute it and see >>> what happened >>> If all of these exit codes where ok (similar to unit tests), i ran the >>> bdd tests to see if everything was correct. >>> >>> By integrating the tdd, and the bdd within a CI-system, I was able to >>> have it run continuously and rebuild the system over and over again. >>> >>> Another thing I want to mention is that you can easily abstract >>> commands in a custom dsl, but I really like to see what actual >>> commands got executed. >>> This is slightly off topic maybe, but by logging/running the actual >>> commands and have the whole environment build like that (including >>> making a floppy, booting the vms) >>> it allowed me to create a kind of install document with the actual >>> commands instead of pointing to a newly invented DSL. >>> So I say yes to an abstraction layer, but no to hiding the actual >>> commands. >>> >>> just my 2 c >>> >>> On Nov 6, 3:56 pm, Lindsay Holmwood<lind...@holmwood.id.au> wrote: >>> >>> >>>> 2009/11/6 Martin Englund<martin.engl...@sun.com>: >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>>> Folks, >>>>> >>>>> >>>> >>>>> I've been struggling a bit with how we're using puppet (at my job): >>>>> how do you validate that puppet has done what it is supposed to, and >>>>> even troublesome, how you validate that it has done what you intended >>>>> it to do? >>>>> >>>>> >>>> >>>>> Since I'm the only one who is writing the puppet profiles and working >>>>> with it on a daily basis, I'm the only one who can decipher the puppet >>>>> logs. I often get the question: how do we know when the system is >>>>> ready for production? >>>>> >>>>> >>>> >>>>> I've been playing around[1] with cucumber& webrat, and have pieced >>>>> together a way to do behavior driven infrastructure testing. Puppet >>>>> takes care of getting the system configured correctly, but there are >>>>> often other pieces involved, like opening firewall ports, adding DNS >>>>> entries, sendmail routing, etc. Which must be done outside of puppet >>>>> to get the system ready for release. >>>>> >>>>> >>>> >>>>> When you write code, you always use unit testing& integration testing >>>>> to verify that the application is working as expected, but why don't >>>>> we use that when we install a system? >>>>> >>>>> >>>> >>>>> What are you using to verify that your system is correctly configured >>>>> and behaves the way you want? >>>>> >>>>> >>>> >>>>> [1]<http://blogs.sun.com/martin/entry/behavior_driven_infrastructure> >>>>> >>>>> >>>> [although i've commented on the excellent blog entry, i'm posting here] >>>> >>>> Hi Martin, >>>> It looks like there's a bit of crossover here with a project i've been >>>> working on the last few months called cucumber-nagios[0]. It takes the >>>> result of a Cucumber run and outputs it in the Nagios plugin format. >>>> Essentially you use it to express your intentions in plain language, >>>> and verify your intentions periodically through your monitoring >>>> system. Just like what you've posted about. :-) >>>> >>>> Anyhow, I spoke about cucumber-nagios at the excellent Devopsdays in >>>> Belgium last weekend, and I got talking with people about expanding >>>> the library of steps to cover things like logins over SSH, file >>>> manipulation, and mail delivery. It would be cool if we could >>>> centralise our efforts and focus on building an awesome library of >>>> reusable steps to test our infrastructure. >>>> >>>> Your point about doing behaviour driven development when writing >>>> software is right on the mark. From an infrastructure perspective, I >>>> like to think of Cucumber as the testing tool, and Puppet as the >>>> programming language. >>>> >>>> Anyhow, i'd be interested to hear what other people think about this idea! >>>> >>>> Cheers, >>>> Lindsay >>>> >>>> [0]http://auxesis.github.com/cucumber-nagios >>>> >>>> --http://holmwood.id.au/~lindsay/(me) >>>> >>>> >>>> >>> >> >>> >> > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
