On Sat, Jan 07 2023, Leo Butler <leo.but...@umanitoba.ca> wrote: > On Fri, Jan 06 2023, Ihor Radchenko <yanta...@posteo.net> wrote: > >> >> Leo Butler <leo.but...@umanitoba.ca> writes: >> >>>> https://builds.sr.ht/~bzg/job/914954 >>>> 2 unexpected results: >>>> FAILED ob-octave/graphics-file ((should-not (get-buffer "*Org-Babel >>>> Error Output*")) :form (get-buffer "*Org-Babel Error Output*") :value >>>> #<killed buffer>) >>>> FAILED ob-octave/graphics-file-space ((should-not (get-buffer >>>> "*Org-Babel Error Output*")) :form (get-buffer "*Org-Babel Error >>>> Output*") :value #<killed buffer>) >>>> >>>> As you can see *Org-Babel Error Output* buffer does not exist when >>>> running the test. >>>> >>>> Leo, could you please take a look? >>> >>> An earlier test is creating that *Org Babel Error Output* buffer. > > I will try to look into improving the tests so that we can trap the test(s) > that is(are) creating that error buffer.
See the attachment. There are four test failures that are currently untrapped. I also see the `buffer-live-p' bug. Leo
#+AUTHOR: Leo Butler #+DATE: 10 Jan 2023 #+TITLE: False positives in ~org~ tests #+STARTUP: show2levels * Summary Applying the [[patch-to--org-test-with-temp-text]] reveals 4 tests that fail, but which are passed with the unpatched ~org-test-with-temp-text~ macro. The next sections show the test failures. References: - *Org-Babel Error Output* :: [[https://orgmode.org/list/87bknh5nva.fsf@localhosthttps://orgmode.org/list/87bknh5nva.fsf@localhost]] - buffer-live-p :: [[https://debbugs.gnu.org/cgi/bugreport.cgi?bug=60626]] #+name: patch-to--org-test-with-temp-text #+begin_example diff --git a/testing/org-test.el b/testing/org-test.el index 22ac60670..7909f36cc 100644 --- a/testing/org-test.el +++ b/testing/org-test.el @@ -196,7 +196,18 @@ otherwise place the point at the beginning of the inserted text." (insert inside-text) (goto-char (point-min)))) (font-lock-ensure (point-min) (point-max)) - ,@body))) + (prog1 + (progn ,@body) + (unwind-protect + (should-not (buffer-live-p (get-buffer "*Org-Babel Error Output*"))) + (when (get-buffer "*Org-Babel Error Output*") + (message "Detected: *Org-Babel Error Output*") + (message "Contents:") + (message (with-current-buffer "*Org-Babel Error Output*" + (buffer-substring-no-properties (point-min) (point-max)))) + (message "End:") + (kill-buffer "*Org-Babel Error Output*")))) + ))) (defmacro org-test-with-temp-text-in-file (text &rest body) "Run body in a temporary file buffer with Org mode as the active mode. #+end_example * Command The tests are run in-place like so: #+begin_src sh :exports none :results raw drawer make test-dirty #+end_src * Test Failures ** Failure 1 #+begin_example Detected: *Org-Babel Error Output* Contents: /tmp/tmp-orgtest/fortran-src-ym6m0X.F90:5:17: 5 | write (*, â(i2)â), nint(s(1,2)) | 1 Warning: Legacy Extension: Comma before i/o item list at (1) [ Babel evaluation exited with code 0 ] /tmp/tmp-orgtest/fortran-src-EYVgCk.F90:5:17: 5 | write (*, â(i2)â), nint(s(2,3)) | 1 Warning: Legacy Extension: Comma before i/o item list at (1) [ Babel evaluation exited with code 0 ] /tmp/tmp-orgtest/fortran-src-SwTuHS.F90:5:20: 5 | write (*, â(3f5.2)â), s | 1 Warning: Legacy Extension: Comma before i/o item list at (1) [ Babel evaluation exited with code 0 ] /tmp/tmp-orgtest/fortran-src-xnH1cO.F90:5:20: 5 | write (*, â(2f5.2)â), s | 1 Warning: Legacy Extension: Comma before i/o item list at (1) [ Babel evaluation exited with code 0 ] End: Test ob-java/args-quoted-string backtrace: ert-fail(((should-not (buffer-live-p (get-buffer "*Org-Babel Error O (if (not (unwind-protect (setq value-1015 (apply fn-1013 args-1014)) (let (form-description-1017) (if (not (unwind-protect (setq value-10 (let ((value-1015 'ert-form-evaluation-aborted-1016)) (let (form-des (let* ((fn-1013 #'buffer-live-p) (args-1014 (condition-case err (let (unwind-protect (let* ((fn-1013 #'buffer-live-p) (args-1014 (conditi (prog1 (progn (let* ((fn-1008 #'string=) (args-1009 (condition-case (progn (org-mode) (let ((point (string-match "<point>" inside-text)) (unwind-protect (progn (org-mode) (let ((point (string-match "<point (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current (let ((inside-text (if (stringp "#+begin_src java :dir 'nil :results (let ((lexical-binding t)) (let ((inside-text (if (stringp "#+begin_ (closure (t) nil (let ((lexical-binding t)) (let ((inside-text (if ( ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test ert-run-test(#s(ert-test :name ob-java/args-quoted-string :documenta ert-run-or-rerun-test(#s(ert--stats :selector "\\(org\\|ob\\|ox\\... ert-run-tests("\\(org\\|ob\\|ox\\)" #f(compiled-function (event-type ert-run-tests-batch("\\(org\\|ob\\|ox\\)") ert-run-tests-batch-and-exit("\\(org\\|ob\\|ox\\)") (let ((org-id-track-globally t) (org-test-selector (if org-test-sele org-test-run-batch-tests("\\(org\\|ob\\|ox\\)") command-line-1(("--eval" "(setq vc-handled-backends nil org-startup- command-line() normal-top-level() Test ob-java/args-quoted-string condition: (ert-test-failed ((should-not (buffer-live-p (get-buffer "*Org-Babel Error Output*"))) :form (buffer-live-p #<killed buffer>) :value t)) FAILED 71/1116 ob-java/args-quoted-string (0.313545 sec) #+end_example ** Failure 2 #+begin_example Detected: *Org-Babel Error Output* Contents: [ Babel evaluation exited with code 1 ] [ Babel evaluation exited with code 2 ] End: Test ob-shell/remote-with-stdin-or-cmdline backtrace: ert-fail(((should-not (buffer-live-p (get-buffer "*Org-Babel Error O (if (not (unwind-protect (setq value-2299 (apply fn-2297 args-2298)) (let (form-description-2301) (if (not (unwind-protect (setq value-22 (let ((value-2299 'ert-form-evaluation-aborted-2300)) (let (form-des (let* ((fn-2297 #'buffer-live-p) (args-2298 (condition-case err (let (unwind-protect (let* ((fn-2297 #'buffer-live-p) (args-2298 (conditi (prog1 (progn (org-trim (org-babel-execute-src-block))) (unwind-prot (progn (org-mode) (let ((point (string-match "<point>" inside-text)) (unwind-protect (progn (org-mode) (let ((point (string-match "<point (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current (let ((inside-text (if (stringp (mapconcat #'identity (list "#+name: (let* ((result (let ((inside-text (if (stringp ...) (mapconcat ... . (let ((default-directory (or (plist-get spec :dir) default-directory (let ((spec (car --dolist-tail--))) (let ((default-directory (or (pl (while --dolist-tail-- (let ((spec (car --dolist-tail--))) (let ((de (let ((--dolist-tail-- (cons nil (cons (list ':dir remote-dir) (cons (closure (t) (remote-dir) (let ((--dolist-tail-- (cons nil (cons (li funcall((closure (t) (remote-dir) (let ((--dolist-tail-- (cons nil ( (let ((tramp-methods (cons '("mock" (tramp-login-program "sh") (tram (cond (env-def (funcall body env-def)) ((eq system-type 'windows-nt) (let ((env-def (getenv "REMOTE_TEMPORARY_FILE_DIRECTORY"))) (cond (e org-test-with-tramp-remote-dir--worker((closure (t) (remote-dir) (le (let ((lexical-binding t)) (let* ((fn-2292 #'not) (args-2293 (condit (closure (t) nil (let ((lexical-binding t)) (let* ((fn-2292 #'not) ( ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test ert-run-test(#s(ert-test :name ob-shell/remote-with-stdin-or-cmdline ert-run-or-rerun-test(#s(ert--stats :selector "\\(org\\|ob\\|ox\\... ert-run-tests("\\(org\\|ob\\|ox\\)" #f(compiled-function (event-type ert-run-tests-batch("\\(org\\|ob\\|ox\\)") ert-run-tests-batch-and-exit("\\(org\\|ob\\|ox\\)") (let ((org-id-track-globally t) (org-test-selector (if org-test-sele org-test-run-batch-tests("\\(org\\|ob\\|ox\\)") command-line-1(("--eval" "(setq vc-handled-backends nil org-startup- command-line() normal-top-level() Test ob-shell/remote-with-stdin-or-cmdline condition: (ert-test-failed ((should-not (buffer-live-p (get-buffer "*Org-Babel Error Output*"))) :form (buffer-live-p #<killed buffer>) :value t)) FAILED 190/1116 ob-shell/remote-with-stdin-or-cmdline (0.013323 sec) #+end_example ** Failure 3 #+begin_example Detected: *Org-Babel Error Output* Contents: [ Babel evaluation exited with code 2 ] End: Test ob-tangle/jump-to-org backtrace: ert--should-signal-hook(ert-test-failed (((should-not (buffer-live-p ert-fail(((should-not (buffer-live-p (get-buffer "*Org-Babel Error O (if (not (unwind-protect (setq value-2414 (apply fn-2412 args-2413)) (let (form-description-2416) (if (not (unwind-protect (setq value-24 (let ((value-2414 'ert-form-evaluation-aborted-2415)) (let (form-des (let* ((fn-2412 #'buffer-live-p) (args-2413 (condition-case err (let (unwind-protect (let* ((fn-2412 #'buffer-live-p) (args-2413 (conditi (prog1 (progn (org-babel-tangle-jump-to-org) (buffer-string)) (unwin (progn (org-mode) (let ((point (string-match "<point>" inside-text)) (unwind-protect (progn (org-mode) (let ((point (string-match "<point (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current (let ((inside-text (if (stringp (format ";; [[file:%s][H:1]]\n<point (let ((file (buffer-file-name))) (let ((inside-text (if (stringp (fo (progn (let ((file (buffer-file-name))) (let ((inside-text (if (stri (progn (setq buffer (find-file file)) (if (re-search-forward "<point (unwind-protect (progn (setq buffer (find-file file)) (if (re-search (let ((file (make-temp-file "org-test")) (inside-text (if (stringp " (list "* H\n#+begin_src emacs-lisp\n1\n#+end_src" (let ((file (make- (let ((signal-hook-function #'ert--should-signal-hook)) (list "* H\n (condition-case err (let ((signal-hook-function #'ert--should-signal (let* ((fn-2417 #'equal) (args-2418 (condition-case err (let ((signa (let ((org-file-apps '((t . emacs)))) (let* ((fn-2417 #'equal) (args (let ((lexical-binding t)) (let ((org-file-apps '((t . emacs)))) (le (closure (t) nil (let ((lexical-binding t)) (let ((org-file-apps '(( ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test ert-run-test(#s(ert-test :name ob-tangle/jump-to-org :documentation ert-run-or-rerun-test(#s(ert--stats :selector "\\(org\\|ob\\|ox\\... ert-run-tests("\\(org\\|ob\\|ox\\)" #f(compiled-function (event-type ert-run-tests-batch("\\(org\\|ob\\|ox\\)") ert-run-tests-batch-and-exit("\\(org\\|ob\\|ox\\)") (let ((org-id-track-globally t) (org-test-selector (if org-test-sele org-test-run-batch-tests("\\(org\\|ob\\|ox\\)") command-line-1(("--eval" "(setq vc-handled-backends nil org-startup- command-line() normal-top-level() Test ob-tangle/jump-to-org condition: (ert-test-failed ((should-not (buffer-live-p (get-buffer "*Org-Babel Error Output*"))) :form (buffer-live-p #<killed buffer>) :value t)) FAILED 206/1116 ob-tangle/jump-to-org (0.007505 sec) #+end_example ** Failure 4 #+begin_example Detected: *Org-Babel Error Output* Contents: ls: cannot access âNoSuchFileOrDirectory.txtâ: No such file or directory [ Babel evaluation exited with code 2 ] End: Test test-ob/allow-spaces-around-=-in-var-specs backtrace: ert-fail(((should-not (buffer-live-p (get-buffer "*Org-Babel Error O (if (not (unwind-protect (setq value-3233 (apply fn-3231 args-3232)) (let (form-description-3235) (if (not (unwind-protect (setq value-32 (let ((value-3233 'ert-form-evaluation-aborted-3234)) (let (form-des (let* ((fn-3231 #'buffer-live-p) (args-3232 (condition-case err (let (unwind-protect (let* ((fn-3231 #'buffer-live-p) (args-3232 (conditi (prog1 (progn (let* ((fn-3226 #'=) (args-3227 (condition-case err (l (progn (org-mode) (let ((point (string-match "<point>" inside-text)) (unwind-protect (progn (org-mode) (let ((point (string-match "<point (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current (let ((inside-text (if (stringp "#+begin_src emacs-lisp :var a = 1 b (let ((lexical-binding t)) (let ((inside-text (if (stringp "#+begin_ (closure (t) nil (let ((lexical-binding t)) (let ((inside-text (if ( ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test ert-run-test(#s(ert-test :name test-ob/allow-spaces-around-=-in-var- ert-run-or-rerun-test(#s(ert--stats :selector "\\(org\\|ob\\|ox\\... ert-run-tests("\\(org\\|ob\\|ox\\)" #f(compiled-function (event-type ert-run-tests-batch("\\(org\\|ob\\|ox\\)") ert-run-tests-batch-and-exit("\\(org\\|ob\\|ox\\)") (let ((org-id-track-globally t) (org-test-selector (if org-test-sele org-test-run-batch-tests("\\(org\\|ob\\|ox\\)") command-line-1(("--eval" "(setq vc-handled-backends nil org-startup- command-line() normal-top-level() Test test-ob/allow-spaces-around-=-in-var-specs condition: (ert-test-failed ((should-not (buffer-live-p (get-buffer "*Org-Babel Error Output*"))) :form (buffer-live-p #<killed buffer>) :value t)) FAILED 315/1116 test-ob/allow-spaces-around-=-in-var-specs (0.003319 sec) #+end_example ** Test Summary #+begin_example Ran 1116 tests, 1102 results as expected, 4 unexpected, 10 skipped (2023-01-10 11:36:40-0600, 64.394581 sec) 2 expected failures 4 unexpected results: FAILED ob-java/args-quoted-string ((should-not (buffer-live-p (get-buffer "*Org-Babel Error Output*"))) :form (buffer-live-p #<killed buffer>) :value t) FAILED ob-shell/remote-with-stdin-or-cmdline ((should-not (buffer-live-p (get-buffer "*Org-Babel Error Output*"))) :form (buffer-live-p #<killed buffer>) :value t) FAILED ob-tangle/jump-to-org ((should-not (buffer-live-p (get-buffer "*Org-Babel Error Output*"))) :form (buffer-live-p #<killed buffer>) :value t) FAILED test-ob/allow-spaces-around-=-in-var-specs ((should-not (buffer-live-p (get-buffer "*Org-Babel Error Output*"))) :form (buffer-live-p #<killed buffer>) :value t) 10 skipped results: SKIPPED org-missing-dependency/test-ob-R ((skip-unless nil) :form nil :value nil) SKIPPED org-missing-dependency/test-ob-clojure ((skip-unless nil) :form nil :value nil) SKIPPED org-missing-dependency/test-ob-eshell ((skip-unless nil) :form nil :value nil) SKIPPED org-missing-dependency/test-ob-julia ((skip-unless nil) :form nil :value nil) SKIPPED org-missing-dependency/test-ob-lua ((skip-unless nil) :form nil :value nil) SKIPPED org-missing-dependency/test-ob-ruby ((skip-unless nil) :form nil :value nil) SKIPPED org-missing-dependency/test-ob-scheme ((skip-unless nil) :form nil :value nil) SKIPPED org-missing-dependency/test-ob-sql ((skip-unless nil) :form nil :value nil) SKIPPED org-missing-dependency/test-ob-sqlite ((skip-unless nil) :form nil :value nil) SKIPPED org-missing-dependency/test-org-attach-git ((skip-unless nil) :form nil :value nil) make: *** [mk/targets.mk:100: test-dirty] Error 1 #+end_example
