test-equal: actual returned value is #f when tested expression raises execption

[Jérémy Korwin-Zmijowski]

Dear Schemers,

As a Guile user, I rely mostly on srfi-64 to write tests.

Recently, a Guile fellow pointed out a strange behavior from one of my
code :

Say I write a test suite :

   ;; char-sets-test.scm

   (use-modules (srfi srfi-64)
                (char-sets))

   (test-begin "harness-char-sets")

   (test-equal "empty password is not valid"
     #f
     (password-valid? ""))

   (test-end "harness-char-sets")

Running `guile -L . char-sets-test.scm` in the file location will
produce the following output :

   $ guile -L . char-sets-test.scm 
   ;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
   ;;;       or pass the --no-auto-compile argument to disable.
   ;;; compiling /tmp/char-sets-test.scm
   ;;; WARNING: compilation of /tmp/char-sets-test.scm failed:
   ;;; no code for module (char-sets)
   Backtrace:
              9 (primitive-load "/tmp/char-sets-test.scm")
   In ice-9/eval.scm:
      721:20  8 (primitive-eval (use-modules (srfi srfi-64) (char-
   sets)))
   In ice-9/psyntax.scm:
     1241:36  7 (expand-top-sequence ((use-modules (srfi srfi-64) (#)))
   …)
     1233:19  6 (parse _ (("placeholder" placeholder)) ((top) #(# # …))
   …)
      285:10  5 (parse _ (("placeholder" placeholder)) (()) _ c&e
   (eval) …)
   In ice-9/boot-9.scm:
     3898:20  4 (process-use-modules _)
      222:29  3 (map1 (((srfi srfi-64)) ((char-sets))))
      222:17  2 (map1 (((char-sets))))
     3899:31  1 (_ ((char-sets)))
      3300:6  0 (resolve-interface (char-sets) #:select _ #:hide _ # _
   # …)

   ice-9/boot-9.scm:3300:6: In procedure resolve-interface:
   no code for module (char-sets)

All good so far. Then I create the `(char-sets)` module in a file next
to the test file :

   ;; char-sets.scm

   (define-module (char-sets))

And now, I rerun the tests :

$ guile -L . char-sets-test.scm 
;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
;;;       or pass the --no-auto-compile argument to disable.
;;; compiling /tmp/char-sets-test.scm
;;; compiling ./char-sets.scm
;;; compiled /home/jeko/.cache/guile/ccache/3.0-LE-8-4.4/tmp/char-
sets.scm.go
;;; char-sets-test.scm:10:2: warning: possibly unbound variable
`password-valid?'
;;; compiled /home/jeko/.cache/guile/ccache/3.0-LE-8-4.4/tmp/char-sets-
test.scm.go
%%%% Starting test harness-char-sets  (Writing full log to "harness-
char-sets.log")
# of expected passes      1

Here, the result of the test feel weird to me. As the tested procedure
is not defined I was expecting the test to fail.

Is there a way to get a failing test in such situation ? I fear to miss
things like those and so build non working softwares.

Cheers,

Jérémy