Thanks for the reply Stuart, I thought about similar things. I'm still not
sure which approach is better myself, but let me clarify a few things.
In the current design of test-is, every test must be attached to
> the :test metadata of a Var. That's why "deftest" currently requires
> a symbol, even a generated one -- it needs a Var to attach the
> metadata to. The "run-tests" function iterates through every Var in a
> namespace, looks for a function in the :test metadata, and calls it.
> I like and use RSpec, but I felt it was better suited to the Object-
> Oriented style of Ruby than to the Lispy style of Clojure. But maybe
> not.
The fact that this is approach is implemented with anamorphic forms instead
of just function calls is something that has troubled me as well. However,
right now I'm leaning towards it being the better decision because a common
testing pattern is encapsulated with it, and the production namespaces
remain clean (I'll explain what I mean below.) I definitely encourage using
vanilla pure functions over side effects, but Clojure does have references
types and I think the anamorphic approach lends itself to testing these more
easily, while keeping the simple case of testing the pure functions easy
(not using context, and only using test.)
> Trying to add "contexts" confuses things a bit. What is a "context",
> exactly? In RSpec, it maps conceptually to a class, with external
> state handled by before/after functions. You can do that in test-is
> now:
>
> (deftest in-context-foo
> ... setup context ...
> ... call each test function ...
> ... teardown context ...)
>
> This way, you can re-use the same test function in multiple contexts.
> In RSpec, tests are defined in a single context, so it's a little less
> flexible (without the addition of share-test and use-test, as in
> Specjure).
One thing that should be clarified here is that share-test does create a
test, but rather creates a shareable function that is named with a string
instead of a symbol. That's all it is, you could just use functions if you
wanted to. However, the reasoning behind using it is to avoid creating
functions in your production namespace. As test-is is currently implemented,
when you deftest you create vars in the namespace you are in. Although these
are gensym'd, it seems like a smell to me to be creating clojure vars in
your produciton namespace (ie: what if production code happened to do
something with all tests in namespaces and inadvertently gathered the test
functions?) Now, I admit that this may be a rare occurrence, but when
testing software I prefer to keep the test and production environments as
close as possible to avoid potentially hard to track down production bugs
(in this case where your test tells you one thing because of those extra
functions existing during test time, but they don't exist in production.)
One thing you can do to make this less of a problem is create specific
test-namespaces for your tests. But, here you lose the ability to test
private functions, and production functions may still be trying to do
something with all namespaces (rather than just their own namespace.)
How I implemented "context/test" and "share-test" may clarify this:
share-test is just a reference to struct that contains contains a :name
string key, and a :fn function key. The *shared-tests* symbol is kept in the
specjure namespace, and when you use-test it does a (get *shared-tests*
:name "my shared test name") and calls the functions with any extra
arguments passed to use-test.
The other reference in the specjure namespace is *tests* which contains an
array of context structs. This struct has the context doc string, an array
of before functions, an array of after functions, and an array of test
functions (there is also one context with nil as its doc string that is used
for test functions defined without being in a context.) Whenver you run a
test, each test function is iterated through. Each iteration binds the
*parameters* var to an empty map for use with $get/$assoc!, calls the before
functions in order, then calls the function, and finally calls the after
functions in order.
Another advantage of having these 2 pieces of data be separate from
production namespaces is test cleanup. If you develop in a repl, you often
reload files and run your tests as you make changes to your files and tests.
To avoid keeping around tests that you've deleted, you can simply ref-set
*tests* and *shared-tests* to a fresh array and map, respectively. In
test-is right now, you have the arguably more complex task of going through
all symbols in all namespaces, and clearing their test metadata.
>
>
> So, finally, here are the two directions I feel test-is could go next:
> 1. RSpec-style: Abandon the Var/metadata framework, use strings for
> test names, and store tests in a global data structure.
> 2. Keep tests as functions, add doc strings, use nested tests for
> context.
>
> I prefer 2, because I think it's more functional and requires less
> bookkeeping code. For example, in specjure, you added share-test, use-
> test, $assoc!, and $get. The first two are unnecessary if tests are
> functions, and the latter two can be handled by binding, like this:
>
> (declare *stack*)
>
> (deftest peek-returns-nil
> (is (nil? (peek *stack*)))
>
> (deftest with-empty-stack
> (binding [*stack* (list)]
> (peek-returns-nil))))
>
> Basically, I think tests are easier to write when they look more like
> ordinary code. You also get automatic bonuses, like source line
> numbers, from the compiler.
>
> One last thing: it seems to be often overlooked that the "is" macro
> can take a second argument, a doc string:
> (is (= 4 (+ 2 2)) "two and two make four")
Although in general it is a good idea to follow the principle of "one
assertion per test", sometimes it is more useful to break the rule. For
example, you may be testing a complex data structure or a function that
creates more than one side effect. In such cases your doc string is more
useful when it describes the behavior rather than the implementation. For
example, say you have some function that manipulates an array. That array is
only valid when the first element is a prime number and the rest of the
elements are even numbers. In this obviously contrived example, this would
be the preferred test:
(context baz "when passed a valid foo"
(test "returns a valid foo"
(let [result (baz (valid-foo))]
(is (prime? (first result)))
(is (all-even? (rest result))))))
Now, this type of complex structure testing is not the norm. It could also
be argued that you should just create a "valid?" function anyway, but
sometimes creating too many of these helper functions makes your library too
big and complex. What this is getting to is that a doc string in the "is"
form (the assertion which may only be part of multiple assertions required
for the particular test doc strying to be valid) is not really the
appropriate location, but rather in the "test" form.
Another point is that although you don't need a separate context form that
allows arbitrary testing with before/after hooks, I think it is a common
enough abstraction to be useful (this is similar to the
language-that-does-not-have-a-feature-but-is-still-turing-complete
argument.) If you create functions that you manually call then this is
custom behavior that is not part of the library. A user will have to
recognize what you are trying to do with these functions, and track them
down to reason about them. If forms such as context/before/after are used
instead, then they are already generally known. Personally, I also prefer
the nested grouping of a context which has before, after, and tests inside
of it easier to grasp visually. The alternative is functions defined outside
that you must "look outside" to see. (Note, I still advocate only using the
test form for simple cases like the addition function example, but when a
context is needed I think nested visual grouping helps.)
> That doc string will be included in failure reports. If I add doc
> strings to deftest, you've basically got all the necessary
> infrastructure for RSpec-style comment strings.
The following assumes that you mean to keep the symbol as a required
argument, sorry if this is not what you meant:
One issue with adding a docstring option to deftest while keeping the
required symbol is that it confuses what should be written in the symbol and
what should be written in the string. Sometimes you may not need to test a
specific function, for example in a webframework:
(test "GET request to /users/:name contains the name of the user"
(create-user! :name "Bob Brown")
(is (contains-str? (http-get "/users/bob-brown") "Bob Brown")))
In this case you would use some mechanism of the framework to create a
function that will be used by the framework. Since the framework is using
this function but doing other things as well, you don't want to test the
function used to add that particular web path, but rather the framework
itself by issuing a real http request. This means you only want to use a
string for deftest/test (not a symbol showing the intention of testing a
particular function, nor a random symbol just to satisfy the required
symbol.)
Again, sorry if you did not mean that a symbol is still required by deftest,
but I think the example is a valuable one to point out for people that
haven't thought of such a scenario.
Everything mentioned here is not meant as an attack of any sort. I like your
idea of keeping it simpler a lot, but would rather play devil's advocate and
present as many counter-points as possible to end up with the most ideal
library. Using it the way I've described versus the more simple way you've
described is an issue that I've gone both ways on many times. Hopefully
we'll come to a good conclusion now that it's out in the open, what are
everyone elses thoughts on this?
--
Respectfully,
Larry Diehl
www.larrytheliquid.com
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To post to this group, send email to clojure@googlegroups.com
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~----------~----~----~----~------~----~------~--~---
