Thank you for all the responses! To be honest, I hoped that someone
would explain why this mocking style is a good thing, and I just
misunderstand something about the "top-down development" that the Midje
wiki suggests:
https://github.com/marick/Midje/wiki/The-idea-behind-top-down-development
While this idea sounds perfectly reasonable when you write the code for
the first time, it seems to me that it will make your life very hard
when you try to modify that code later.
I agree with Timothy that the main problem with my example is that these
tests are checking almost nothing. In essence, the tests are just a
"shadow implementation" of the real code: every function call has a
corresponding mock call in the test, so whenever you modify the
implementation (without even changing the top-level result) you have to
modify the tests as well, and vica-versa. I once encountered a codebase
that had thousands of lines of such "tests", and it was a nightmare to
modify anything in it.
The thing that I like in the Midje article though is the process how the
code is written: I usually design things top-down, but implement it
bottom-up because that's what works easily with the tools. It would be
great (at least for me) if top-down thinking could be applied during
implementation as well.
Maybe the "big trick" is to use the mocking features for unfinished code
(they are indeed convenient in Midje), but remove them immediately after
the lower-level components are implemented. In some way it is similar to
the classic TDD cycle when you first write a fake method that satisfies
your test, and then "refactor" it to have a real implementation.
BTW my current approach is to test mostly without mocks, from as
high-level as reasonably possible:
- If possible, test a module from its public interface
- If this is too much pain (e.g. the execution has too many branches and
it is hard to exercise all of them), descend one level: take the
high-level components of the module and test these parts individually.
Descend another level if it's still too painful, (recurse). Always keep
a few integration tests that exercises the components together.
- Try to find balance between not going too deep vs being able to test
effectively. I usually end up with unit-testing a few complex low-level
functions, and have much higher-level tests for the rest of the code.
- Use mocks for undeterministic functions like (rand) or (current-time)
only
Although this approach is maybe far from classic TDD, I found that it
usually results in a test suite that is easy to maintain: the public
interfaces rarely change, so you can refactor most of the implementation
details without touching the tests. It also helps you think in
"contracts", as your focus is on the interface that is used between
components.
Regards,
Akos
On Tue, Jan 6, 2015, at 03:41 PM, Colin Yates wrote:
> +1 - I think we are saying the same thing (not sure if you meant to
> reply to me?)
>
> On 6 January 2015 at 14:35, Timothy Baldridge <tbaldri...@gmail.com>
> wrote:
> > I think the answer to questions like this is that you are testing the wrong
> > thing, or more correctly you are writing incomplete tests.
> >
> > In your example, you stubbed out check-pw. But calling check-pw has a
> > contract, a contract that (at the moment) only exists in your head, but a
> > contract none-the-less. That contract needs to be tested, from both sides.
> > Tests should invoke all instances of check-pw. In addition you should have a
> > test that pairs a login form with a check-pw and runs tests against this
> > system.
> >
> > Some people call these tests "integration tests" or "system tests". But I
> > think of them as contract tests.
> >
> > Here's a diagram of the problem:
> >
> > login -----> check-pw
> >
> > I've found that most code that uses mocking will test the login and the
> > check-pw bits, but completely neglect testing the "arrow" between them, and
> > when that happens, you get exactly the experience you described.
> >
> > The other thing I'd like to mention is that I have found it very valuable to
> > sit down and think about what code is actually being hit by a test. In your
> > example, if check-pw and the db are both mocked, what is actually being
> > tested? In your example all you are testing with those functions mocked is
> > that Clojure is capable of compiling a function that calls two other
> > functions. I can't tell you how many times I've looked at mocked tests and
> > realized that the only thing being tested is something like read-string,
> > get, or destructuring.
> >
> > So my personal approach is this: write very coarse tests that exercise the
> > entire system. These will catch the protocol mis-matches. Then if you want
> > more detail for when tests do fail, write more specific tests. In short:
> >
> > System (integration) tests: so I feel good about my codebase
> > Unit (smaller) tests: so I can figure out what went wrong when the larger
> > tests fail.
> >
> > Timothy
> >
> > On Tue, Jan 6, 2015 at 6:26 AM, Colin Yates <colin.ya...@gmail.com> wrote:
> >>
> >> I don't think there is an easy answer here, and note that this is a
> >> problem generic to mocking (i.e. not clojure or midje specific).
> >>
> >> The usual advice applies though:
> >> - do you really need to mock? Unit testing is about the coarseness of
> >> granularity which is defined more by cohesion and abstractions than "one
> >> function and only this function" (e.g. remove the problem by not overly
> >> mocking)
> >> - make everything fail then fix rather than fix and then upgrade (i.e.
> >> update every instance of the call/mock to check-pw before the
> >> implementation
> >> of check-pw).
> >> - Clojure's lack of types means the compiler can't help. Schema or
> >> core.typed can. This isn't *the* answer, but I have found it very helpful.
> >>
> >> As mentioned elsewhere, mocking in general is a very powerful tool, but it
> >> is does need wielding carefully. These problems are easier to swallow in
> >> strongly typed languages because of IDE support (changing parameters around
> >> in Java with IntelliJ is a matter of a few key presses for example).
> >>
> >> Hope this helps.
> >>
> >>
> >> On Tuesday, 6 January 2015 08:22:36 UTC, Akos Gyimesi wrote:
> >>>
> >>>
> >>> On Sat, Jan 3, 2015, at 02:46 AM, Brian Marick wrote:
> >>> >
> >>> > > I use TDD and mocking/stubbing (conjure) to test each layer of my
> >>> > > code.
> >>> > > The problem is when I change the function signature and the tests do
> >>> > > not
> >>> > > break, because the mocks/stubs do not know when their argument lists
> >>> > > no
> >>> > > longer agree with the underlying function they are mocking. Is there
> >>> > > a
> >>> > > way to catch this? Short of a test suite that eschews stubbing in
> >>> > > favor
> >>> > > of full setup/teardown of DB data for each test?
> >>> >
> >>> > Could you give an example? I use mocks fairly heavily, and I don't seem
> >>> > to have this problem. Perhaps it's because I change the tests before
> >>> > the
> >>> > code?
> >>>
> >>> Although the subject changed a little bit, I would be still interested
> >>> in your approach to refactoring if there is heavy use of mocking. Let me
> >>> give you an example:
> >>>
> >>> Let's say I am writing a login form, trying to use the top-down approach
> >>> you described. My approach could be the following:
> >>>
> >>> (unfinished check-pw)
> >>>
> >>> (fact "login-form succeeds if user enters the correct password"
> >>> (login-form-success? {:username "admin" :password "secret"}) => true
> >>> (provided
> >>> (db/get-user "admin") => (contains (:password "my-secret-hash"))
> >>> (check-pw "my-secret-hash" "secret") => true))
> >>>
> >>> (defn login-form-success? [user-input]
> >>> (let [user (db/get-user (:username user-input))]
> >>> (check-pw (:password user) (:password user-input))))
> >>>
> >>> Then I finish the check-pw function and everything works.
> >>>
> >>> Now, later that day I decide that I pass the whole user object to the
> >>> check-pw function. Maybe I want to use the user ID as a salt, or maybe I
> >>> just want to leave the possibility for checking password expiration,
> >>> etc. So I modify the test and the implementation of check-pw so that the
> >>> first parameter is the user object, not the password hash.
> >>>
> >>> Suddenly my co-worker comes to me saying "hey, I need you on a meeting
> >>> right now!" I close my laptop, and an hour later I think "where were
> >>> we?..." I run all the tests, and they all pass, so I commit.
> >>>
> >>> Except... I forgot to modify all the invocations of check-pw in both the
> >>> test and the implementation. Every test pass, so I have no way of
> >>> finding out the problem without careful code review or by examining the
> >>> stack traces from the live code.
> >>>
> >>> While this bug is easy to catch, what if my function is mocked in
> >>> several places, and I fail to rewrite all of them properly?
> >>>
> >>> Do you have any advice on what you would have done differently here to
> >>> avoid this bug?
> >>>
> >>> Regards,
> >>> Akos
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> >> Groups "Clojure" group.
> >> To post to this group, send email to clojure@googlegroups.com
> >> Note that posts from new members are moderated - please be patient with
> >> your first post.
> >> To unsubscribe from this group, send email to
> >> clojure+unsubscr...@googlegroups.com
> >> For more options, visit this group at
> >> http://groups.google.com/group/clojure?hl=en
> >> ---
> >> You received this message because you are subscribed to the Google Groups
> >> "Clojure" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an
> >> email to clojure+unsubscr...@googlegroups.com.
> >> For more options, visit https://groups.google.com/d/optout.
> >
> >
> >
> >
> > --
> > “One of the main causes of the fall of the Roman Empire was that–lacking
> > zero–they had no way to indicate successful termination of their C
> > programs.”
> > (Robert Firth)
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Clojure" group.
> > To post to this group, send email to clojure@googlegroups.com
> > Note that posts from new members are moderated - please be patient with your
> > first post.
> > To unsubscribe from this group, send email to
> > clojure+unsubscr...@googlegroups.com
> > For more options, visit this group at
> > http://groups.google.com/group/clojure?hl=en
> > ---
> > You received this message because you are subscribed to a topic in the
> > Google Groups "Clojure" group.
> > To unsubscribe from this topic, visit
> > https://groups.google.com/d/topic/clojure/T8fIW27kDYE/unsubscribe.
> > To unsubscribe from this group and all its topics, send an email to
> > clojure+unsubscr...@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
