Ken,
Thanks for your response! Those are some great suggestions. Some
look quite promising, a few don't fit my purposes. For instance, I
can't use pcalls because the user needs to be able to control the
number of threads. Your method to skip tests using 'and' works, and
is interesting, but I want the tree of tests to be easily printable
and for users to be able to write them that way (nested map/vector),
so that dependencies are visible at a glance without skipping around
in the code.
Part of the reason I embarked on this project was that midje, lazytest
etc are used almost exclusively for unit testing. I need to do black
box testing, where the app being tested is remote and it is only
accessible via HTTP or some other network protocol.
Some of the features I wanted are:
* Self-documenting tests. There should be no need to write english
testcases AND automate them, the automation IS the testcase.
However, management likes to know that tests can be read and run by
someone besides the person who wrote them, even when they are
automated. So I want
"(defn duplicate-name-disallowed-test []
(let [name "testitem"]
(create-widget name)
(expect NameTakenError
(create-widget name)))"
to be the canonical testcase. If necessary I can process the form
into more human readable text (although to me it's plenty readable
already, but non programmers might disagree). Unfortunately automating
a testcase doesn't absolve it from being in the testcase tracking db.
* Self-logging (I use clojure.contrib.trace with some modification -
writing log4j statements is a waste of time IMO). Automated tests are
software, and so is the app under test. The automation simply flags
when it doesn't get what it expects. The engineer's job is to figure
out which software is wrong, so logs are important.
* Fast execution (hence multithreaded) - the more often tests can run,
the fewer suspects you have for the cause of the breakage.
* Ability to control depth of testing (this is why I used a tree
structure where dependent tests are children - it forces the automator
to lay out tests such that the most severe bugs are found first. If
you can't log in, the whole app is broken, so "login test" goes at the
root of the tree. In most cases, bug severity <= depth in tree.
Quick testing means maxdepth = 2 or 3, detailed testing means maxdepth
= unlimited).
On Aug 8, 2:12 pm, Ken Wesson <kwess...@gmail.com> wrote:
> If the local bindings will never change, then why not just use
> (binding [whatever-setup ...] ...) wrapping the individual test bodies
> that need such setup? (Where explicit tear-down is required, you'd
> need try ... finally as well, or better yet a macro like with-open,
> but using binding instead of let, to abstract out the repeated aspects
> of such code. Common setups could become more macros, e.g.
> (with-foo-whatsit [some-symbol some-other-symbol] ...).)
>
> If the bindings are to stateful stuff that a sequence of tests will
> alter, though, you have more problems. Possible improvements:
>
> * Redesign the whole thing to be more functional and less stateful.
>
> * At least, move as much into pure functions as possible; the pure
> functions are easy to test.
>
> * The (remaining) sequences of tests on state will have to run
> sequentially, so combine them into a single test that calls the
> smaller ones.
>
> You'll end up with something like:
>
> (def results (atom {}))
>
> (def tests (atom []))
>
> (def foo nil)
>
> (def bar nil)
>
> (defmacro deftest ... )
>
> (deftest foo nil
> [foo (initialize-some-resource)
> bar (initialize-another)]
> (the test goes here)))
>
> and that produces something like
>
> (do
> (defn foo []
> (binding [foo (initialize-some-resource)]
> (try
> (binding [bar (initialize-another)]
> (try
> (deliver (@results foo) (the test goes here))
> (finally (.close bar))))
> (catch Throwable _ (deliver (@results foo) false))
> (finally (.close foo)))))
> (swap! results assoc foo (promise))
> (swap! tests conj foo))
>
> whereas
>
> (deftest bar foo ...)
>
> is similar, but with foo instead of nil for the second argument the
> bar function body gets wrapped in:
>
> (if @(@results foo)
> (do
> (body that would have been)
> (with nil second arg))
> (deliver (@results bar) false))
>
> so when bar is run it blocks until foo has run, and short-circuits to
> failing if foo failed, but runs if foo succeeded.
>
> And then there'd be
>
> (deftest baz :subtest ...)
>
> which expands without swap!s or a wrapper -- it just becomes a
> function like foo and nothing else. A later test body can do setup,
> call baz as a function along with several other subtests, and do
> teardown, e.g.
>
> (deftest quux ...
> ...
> (and (baz) (baz2) (baz3)))
>
> which, obviously, short-circuits to failure if any of these fails and
> otherwise runs them all and returns baz3's result.
>
> Lastly, you'd have
>
> (doall (apply pcalls @tests))
>
> to run the tests and
>
> (report)
>
> after that, with
>
> (defn report
> (doseq [[test result] @results]
> (println (:name (meta test)) " - " @result)))
>
> or whatever. If you want more detailed reports, deftest could put
> additional stuff into the metadata of the functions (and have
> additional arguments, if necessary).
>
> The above framework seems like it would do what you want: allow tests
> to have setup and teardown and simplify that, make the stuff thus set
> up be dynamic bindings visible to subsidiary functions, allow tests to
> depend on earlier tests and block until the earlier tests complete,
> and allow a sequence of tests to be run within a single set-up
> environment run sequentially, in a single thread, in that environment.
> The major downside is that pcalls is optimized for cpu-bound jobs; if
> the tests are I/O-bound you'll want to change pcalls to something
> based around Executor instead, with a thread pool however large you
> want. The other is that tests that must be run in the same
> environment, with just one setup before all the tests and one teardown
> after, will be a single unit as far as parallelism goes and will stop
> on the first failed test. The framework can be modified to let
> subtests create separate result entries, though, and using (let [a
> (baz) b (baz2) c (baz3)] (and a b c)) will let you run more subtests
> even if one fails, when you know the setup environment won't have been
> borked by the failures. Also, tests that should get an exception
> require you to explicitly catch the exception exception and return
> true, e.g. (try (this-should-throw-something) false (catch
> FooException _ true)). You could add a bunch of (expect ...) macros to
> simplify further the writing of tests.
>
> If you're trying to use an existing testing framework like midje,
> though, you're probably SOL unless the framework developer provides
> their own parallel testing support or you can understand the framework
> code well enough to marry it to something like the above.
>
> --
> Protege: What is this seething mass of parentheses?!
> Master: Your father's Lisp REPL. This is the language of a true
> hacker. Not as clumsy or random as C++; a language for a more
> civilized age.
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
