Hello, On Tue, Mar 18, 2008 at 4:15 AM, David Schmidt <[EMAIL PROTECTED]> wrote: > Hello fellow RSpec users. > > Before you all start warming up your flame throwers please let me explain my > Subject line. > > I've been working over 4 months on a large Rails project with a few other > developers. Test coverage was spotty at best, though they *were* RSpec > tests. One of the other developers and I had started adding more tests, > mostly controller tests using the methodology given at rspec.info for > writing controller tests isolated from the model and view layers using stubs > and mocks. > > Recently a new project manager was put in place and he brought in another > developer. This developer promptly started to re-write all the *existing* > controller (and later view) tests, removing all mocks and stubs and > replacing them with code to use fixtures. (He also deletes many comments he > finds in the code if *he* thinks they're obvious, but that's another > story...). His commit messages include comments like "Stop mocking around" > and "More fixes due to our test mockery". > > When challenged on why he's re-writing these tests instead of writing new, > missing tests (even tests using fixtures) he replied with this e-mail with > the subject "Why not MockEverything". (Note that I *do* use fixtures for > model tests but follow the RSpec documentation and use mocks/stubs for > controller and view tests for isolation.) In the email this developer > mentions tests broken by the addition of conditional to the view. This > conditional used a model method not previously used in the view, and the > addition of one stub was sufficient to fix the view test in question. > > Here is his email to me, less his signature as I don't want to make this > personal. I'd like to see what the RSpec user community has to say in > response to his comments, below: > > --- Why not MockEverything --- > David I've removed the mocks on purpuse. Not that I have sufficient ills > with them to meddle without a *need*. We committed simple template fixes > adding a conditional and there, yet the tests broke. > > Now this was to be expected, the tests were constructed by exhaustively > mocking out all methods called on the object. Add a simple conditional be it > harmless as it is now means another method needs to be mocked out. > > The MockEverything approach is not healthy, judicious use is preferable. One > thing is to write a short sample in a blog and another is to have a working > app with lots of tests. From all my apps that I have worked on this has by > far the lowest coverage both in profile and in test value. There is no > discussion we are all committed to tests. > > To better see what constitutes good practice I recommend you to inspect > the source of RadiantCMS a beautiful and well engineered app recently > rewrote to use rspec instead of Test::Unit: > > http://dev.radiantcms.org/browser/trunk/radiant/spec > > Observe how the code is restrained in mocking, real objects are preferred > wherever possible. Incidentally they don't use fixtures rather factories to > create *real* objects. Now the factory part is a separate issue I'll don't > discuss here, as it has its own disadvantages especially a project with many > models ... > > With real objects your test will not be brittle, and their value will be > kept even after adjusting the templates or doing other small refactorings. > Contrary to common misconception test speed will not be affected either. > Especially for view tests where you don't even have to save to the db upon > preparing the test rig. > > Beside Radiant there where efforts to rspec Typo and Mephisto (both noted > rails blog engines). Still these were half harted conversions so my > arguments based on them would not have the same weight. RadiantCMS is enough > - they are used on ruby-lang.org and have converted 100% to rspec ... plus > they also have good coverage showing that they actually believe in tests. So > please look into Radiant, you'll find it most helpful I think. > --- END OF EMAIL--- > > Thank you, > > David Schmidt
There are a few bad assumptions in your colleague's response, so to set the record straight: * test coverage and tests which use the interaction-based test approach are not mutually exclusive * you can have crappy tests which take the state-based approach and crappy tests which use a interaction-based approach * interaction-based testing is not merely limited to contrived examples on people's blogs, it is a real practice which adds value on lots of "real-world" projects * using factories to generate required objects in tests has several pros over the use of fixtures, and very very very few cons State-based testing and interaction-based testing both have their place. There are number of reasons why they are both useful, but I'm going to pick two: object decomposition (and coordinators) and integration testing. Others have mentioned the value of writing tests with the interface you want so I'm going to leave that out. As an application grows in features and complexity (business logic of course) good developers will decompose the problem into a number of simple objects. Some of these objects are responsible for doing the work and others are responsible for coordinating other objects to do the work. Objects which are responsible for coordinating are great candidates for using interaction-based testing, because you are concerned in the interaction, not the "state". If you don't have integration tests then using an interaction-based testing approach is not worth it because you need something that is going to test the real objects working with real objects. In Rails you can write integration tests as Rail's ActionController::IntegrationTests, Rail's functional tests, RSpec stories, or RSpec controller tests w/view isolation turned off. IMO, one false benefit of only using a state-based approach when writing a full-fledged application is that every object is essentially an integration test at some level. You are always testing everything with everything that it touches. This can lead to having one failure in one model make several other model tests fail, and it can make several controller tests failing (as well as any other object which touches the model that is failing). I see this has a big negative because it makes it more difficult to pinpoint the issue. People will end up tracking it down, but it can be time consuming and frustrating. Now on the flip side people will complain that they renamed a model method and re-ran all of their tests and everything passed, but when running the application a bug exists. Doh, we forgot to update the controller that relied on calling that model method. It is normal to say/think, "well that should have failed because the method doesn't exist on the model". (It sounds like David Chelimsky may have something in trunk to help with this.) The main problem here though is that an integration test didn't fail exposing that you weren't done with your change. Thinking back to coordinating objects, my controllers don't contain business logic in them because they are application layer classes, they aren't apart of the domain of my software. They are only used by the application to allow the software to fulfill the requirements of my customer. Controllers are coordinators, not DOERS. They ask other objects to fulfill a business requirement for them like moving stocks from one portfolio to the another. So I used interaction-based testing here to ensure that my controller is finding a stock, finding a portfolio and asking a portfolio manager to move the stock to the designed portfolio. I don't need to have those things written or even fully implemented to ensure my controller works as I expect. I should be able to see that my controller does what it should be doing, even if the pieces it will use to do the work in the application aren't finished. Now if those aren't implemented I should have an integration test which fails showing me that the feature for moving stocks from one portfolio to another is not completed, but that isn't what I'm testing in my controller. Also after my controller works as expected I can go make sure the PortfolioManager works as expected, and then I can go down and make sure the Stock model does what I expect. When these objects are working correctly individual I run my integration tests to ensure they work well together. Another drawback of only using state-based testing is that you always have to develop bottom up. You have to start with the low level components and work your way out. I used to write code this way. I think I have progressed beyond that, and now I write things in a Acceptance Test Driven Development style. I start by writing an integration test from the user's perspective proving that the feature doesn't work, and then I move to the view, and then to the controller, then to any manager/factory/presenter/service objects that are required, and then down to any domain level objects (models and non-models alike). You can't do this approach with state-based testing only. There is a lot of value that can be gained by developing software this way. In short: Interaction-based testing allows you to ensure that an object is doing what you expect, without the underlying implementation having to exist yet at all or in full. It is great for application layer objects which typically only coordinate domain layer objects where the correct interaction is what is important. It also helps you develop interfaces, and it can scream loudly when you have an object doing way too much. * "Blaming "brittleness" of tests upon interaction-based testing is a red herring. Both interaction-based tests and state-based tests become brittle if they make assertions upon implementation details and overly constrain the interfaces between modules." - Nat Pryce * http://nat.truemesh.com/archives/000342.html - a wonderful read on interaction-based vs state-based testing -- Zach Dennis http://www.continuousthinking.com _______________________________________________ rspec-users mailing list rspec-users@rubyforge.org http://rubyforge.org/mailman/listinfo/rspec-users
