Am 13.03.2025 um 12:49 schrieb Richard Biener:
On Thu, 13 Mar 2025, Sam James wrote:
Simon Sobisch <simonsobi...@gnu.org> writes:
Thanks for your work on adding a testsuite. Can you please explain why
you do this when a complete testsuite exists in autoconf (autotest)
format (which roots back to decade of work in GnuCOBOL, with all
copyrights for that already with the FSF)?
I don't think any of us were aware of it ("we" being "the general GCC
developer community", not the COBOL folks, for the purposes of this
email) until yesterday when richi mused about it on IRC maybe existing
and we went looking out of curiosity.
I agree that having that testsuite integrated would be fantastic.
Is the existence of this in upstream [1] just unknown (because it was
not part of the initial patches [for reasons I not understood])?
I would've personally liked to see the NIST testsuite integration at
least in the initial patches, but it is what it is. I don't think the
GnuCOBOL testsuite was brought up at all (and I think most of us weren't
aware of it) in the patch upstreaming discussions.
Now that we *are* aware of it, it seems desirable to have for sure.
Is the format such a big issue (note: previous discussions elaborated
"a test suite is very important and other frontends also use a
framework other than dejagnu)?
If dejagnu is the way to go:
* Shouldn't there be deprecation of autotest in autoconf (of course
only if that preference is also outside of gcc)?
It's a GCC / GNU toolchain-only preference because it allows easily
doing cross + simulator testing, and all of our tools are used to its
format.
That's indeed the main reason.
Thanks for the explanation. That's totally fine.
It's definitely not perfect. Years ago (way before I followed GCC),
there was talk of replacing dejagnu, just efforts failed.
* Shouldn't there be a (at least semi automated) script / migration
tool (at least for this specific time in place to convert the "UAT"
once into dejagnu format)?
Yes. Having testsuite integration is seen as critical at this
point. richi just wanted to present this as a non-COBOL person to give
us something to play with.
Yes, and to give people familiar with how GCC tests are done a place
to put regression tests going forward.
I do think that integrating the testsuites the COBOLworx folks have
is important and of course integrating tests from GNU Cobol is desirable
as well. Whether we can or want to integrate tests based on autotest
is another question - I'd probably avoid that, even as short-term
solution, as such tend to stay forever.
I agree. Note: COBOLworx started by using the GnuCOBOL testsuite; even
with the current UAT's state it would be a lot of manual work to
re-synchronize them, so going one step further to dejagnu seems to not
make it much harder either.
It will definitely be useful if the "original test file names" (like
run_subscripts.at, or at least run_subscripts) are kept somewhere - a
comment like "auto-translated from run_subscripts.at" is enough - and
maybe they can stay in one file each (I don't know enough about dejagnu
to comment on that).
The main point is that it seems most reasonable to convert those files
into dejagnu format once (so obviously a "script working good enough,
not installed" comes into mind), instead of writing it from scratch.
What would be nice is to have a common separate test harness you can
test an installed compiler against - I'm not sure whether the GNU
Cobol test harness or the COBOLworx one qualifies here. The NIST
one probably does, but it seems to require "plumbing" that's not
part of NIST and that, in implementation, might differ from GNU Cobol
to COBOLworx.
That's a good opportunity to be picky: it is GnuCOBOL (one word, COBOL
in upper-case) :-)
And yes: a common separate test harness is most reasonable and that's
exactly what the idea of NIST was.
If you ever wonder: GnuCOBOL uses make (with one sub-directory per
"Module") along with perl [2].
This allows to not only do testing (or just extraction of the files)
along with counting and tracking time, but also to automate some of the
required "needs manual inspection".
And given gcobc, I'd argue that gcobol should not fail the following
(and ideally show its superior compile and run time):
$> tar -xvf gnucobol-3.*.tar.*
$> cd gnucobol-3.*/
$> ./configure # for automake and autoconf doing the setup
$> cd tests/cobol85
$> make test COBC=gcobc-15
... just tried that:
gcobol: error: unrecognized command-line option ‘-std=cobol85’
--> seems like the gcobc should drop that and set the right flags for
gcobol here (I know, should be on bugzilla, or just fixed)
$> make test COBC=gcobc-15 COBC_FLAGS=--debug
Compiling EXEC85 program
warning: --debug implies -fstack-check: ignored
EXEC85.cob:1:73: error: syntax error, unexpected NAME, expecting
FUNCTION or PROGRAM-ID
1 | 000100 IDENTIFICATION DIVISION.
EXEC84.2
|
^
cobol1: error: failed compiling EXEC85.cob
--> seems the recognition in cobol1 for the reference-format can be
improved (that's no *problem* in general)
$> make test COBC=gcobc-15 COBC_FLAGS="-fec=all -fformat=fixed -g"
Compiling EXEC85 program
Building module directory NC ...
/home/build/gcobol-test/gnucobol-3.2/pre-inst-env: 69: exec: ../EXEC85:
not found
--> that is definitely another problem of the gcobc driver script - if
no output file is given this must be deduced from the first input file
to match cobc behavior
$> mv a.out EXEC85
$> make test COBC=gcobc-15 COBC_FLAGS="-fec=all -fformat=fixed -g"
Building module directory NC ...
... that runs until crash.
/bin/bash: Line 6: 6503 Killed ( cd NC && COB_UNIX_LF=Y
"/home/build/gcobol-test/gnucobol-3.2/pre-inst-env" ../EXEC85 )
because of some looping (single core completely used, memory fast
increasing, here's the output from "top" shortly before the
system-internal kill
simon 20 0 19,1g 15,1g 960 R 15,0 97,0 1:52.12 EXEC85
Side note: even when the system kills the process, it should have some
time to write a reasonable stacktrace or something to know where the
error happened.
I'm quite sure the looping part is related to file handling (rooting in
configuration that is different for gcobol and GnuCOBOL [which it may
shouldn't]) but that's for Bob to inspect/fix (I've used the COBOLworx
debian package generated yesterday, BTW).
... ok, re-executed and attached with GDB - it loops around the READ at
EXEC.cob:2202, presumably for the non-open file.
With the guessed configuration issue:
* the program should not hog memory during looping (Bob, please inspect
this one *first*)
* the program should abort somewhere, likely at the OPEN long before it
gets to the READ where it loops, but that should be fixed after this
specific "loop creates a memory problem" is fixed (end then maybe in
reverse order, first let the READ [directly after the OPEN] of the
not-opened file abort, then the OPEN itself) [note: there is neither
a USE section nor an io-status for the file defined and even the
-fec-io runtime checks were requested to be enabled, so ISO says that
this READ (and the OPEN before) should result in a fatal runtime
abort]
Thanks,
Simon
Richard.
[1]:
https://gitlab.cobolworx.com/COBOLworx/gcc-cobol/-/tree/master+cobol/gcc/cobol/UAT
[2]:
https://sourceforge.net/p/gnucobol/code/HEAD/tree/branches/gnucobol-3.x/tests/cobol85/
