I think the answer to questions like this is that you are testing the wrong
thing, or more correctly you are writing incomplete tests.
In your example, you stubbed out check-pw. But calling check-pw has a
contract, a contract that (at the moment) only exists in your head, but a
contract none-the-less. That contract needs to be tested, from both sides.
Tests should invoke all instances of check-pw. In addition you should have
a test that pairs a login form with a check-pw and runs tests against this
system.
Some people call these tests "integration tests" or "system tests". But I
think of them as contract tests.
Here's a diagram of the problem:
login -----> check-pw
I've found that most code that uses mocking will test the login and the
check-pw bits, but completely neglect testing the "arrow" between them, and
when that happens, you get exactly the experience you described.
The other thing I'd like to mention is that I have found it very valuable
to sit down and think about what code is actually being hit by a test. In
your example, if check-pw and the db are both mocked, what is actually
being tested? In your example all you are testing with those functions
mocked is that Clojure is capable of compiling a function that calls two
other functions. I can't tell you how many times I've looked at mocked
tests and realized that the only thing being tested is something like
read-string, get, or destructuring.
So my personal approach is this: write very coarse tests that exercise the
entire system. These will catch the protocol mis-matches. Then if you want
more detail for when tests do fail, write more specific tests. In short:
System (integration) tests: so I feel good about my codebase
Unit (smaller) tests: so I can figure out what went wrong when the larger
tests fail.
Timothy
On Tue, Jan 6, 2015 at 6:26 AM, Colin Yates <colin.ya...@gmail.com> wrote:
> I don't think there is an easy answer here, and note that this is a
> problem generic to mocking (i.e. not clojure or midje specific).
>
> The usual advice applies though:
> - do you really need to mock? Unit testing is about the coarseness of
> granularity which is defined more by cohesion and abstractions than "one
> function and only this function" (e.g. remove the problem by not overly
> mocking)
> - make everything fail then fix rather than fix and then upgrade (i.e.
> update every instance of the call/mock to check-pw before the
> implementation of check-pw).
> - Clojure's lack of types means the compiler can't help. Schema or
> core.typed can. This isn't *the* answer, but I have found it very helpful.
>
> As mentioned elsewhere, mocking in general is a very powerful tool, but it
> is does need wielding carefully. These problems are easier to swallow in
> strongly typed languages because of IDE support (changing parameters around
> in Java with IntelliJ is a matter of a few key presses for example).
>
> Hope this helps.
>
>
> On Tuesday, 6 January 2015 08:22:36 UTC, Akos Gyimesi wrote:
>>
>>
>> On Sat, Jan 3, 2015, at 02:46 AM, Brian Marick wrote:
>> >
>> > > I use TDD and mocking/stubbing (conjure) to test each layer of my
>> code.
>> > > The problem is when I change the function signature and the tests do
>> not
>> > > break, because the mocks/stubs do not know when their argument lists
>> no
>> > > longer agree with the underlying function they are mocking. Is there
>> a
>> > > way to catch this? Short of a test suite that eschews stubbing in
>> favor
>> > > of full setup/teardown of DB data for each test?
>> >
>> > Could you give an example? I use mocks fairly heavily, and I don't seem
>> > to have this problem. Perhaps it's because I change the tests before
>> the
>> > code?
>>
>> Although the subject changed a little bit, I would be still interested
>> in your approach to refactoring if there is heavy use of mocking. Let me
>> give you an example:
>>
>> Let's say I am writing a login form, trying to use the top-down approach
>> you described. My approach could be the following:
>>
>> (unfinished check-pw)
>>
>> (fact "login-form succeeds if user enters the correct password"
>> (login-form-success? {:username "admin" :password "secret"}) => true
>> (provided
>> (db/get-user "admin") => (contains (:password "my-secret-hash"))
>> (check-pw "my-secret-hash" "secret") => true))
>>
>> (defn login-form-success? [user-input]
>> (let [user (db/get-user (:username user-input))]
>> (check-pw (:password user) (:password user-input))))
>>
>> Then I finish the check-pw function and everything works.
>>
>> Now, later that day I decide that I pass the whole user object to the
>> check-pw function. Maybe I want to use the user ID as a salt, or maybe I
>> just want to leave the possibility for checking password expiration,
>> etc. So I modify the test and the implementation of check-pw so that the
>> first parameter is the user object, not the password hash.
>>
>> Suddenly my co-worker comes to me saying "hey, I need you on a meeting
>> right now!" I close my laptop, and an hour later I think "where were
>> we?..." I run all the tests, and they all pass, so I commit.
>>
>> Except... I forgot to modify all the invocations of check-pw in both the
>> test and the implementation. Every test pass, so I have no way of
>> finding out the problem without careful code review or by examining the
>> stack traces from the live code.
>>
>> While this bug is easy to catch, what if my function is mocked in
>> several places, and I fail to rewrite all of them properly?
>>
>> Do you have any advice on what you would have done differently here to
>> avoid this bug?
>>
>> Regards,
>> Akos
>>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
“One of the main causes of the fall of the Roman Empire was that–lacking
zero–they had no way to indicate successful termination of their C
programs.”
(Robert Firth)
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
