> > > -----Original Message----- > From: Gavin Smith [mailto:gavinsmith0...@gmail.com] > Sent: Sunday, April 26, 2015 9:03 AM > To: Arthur Schwarz > Cc: Automake Mailing List > Subject: Re: Question on --test-name=NAME > > On 26 April 2015 at 16:30, Arthur Schwarz <aschwarz1...@att.net> wrote: > > > > > > 15.3.3.1 Command-line arguments for test drivers > > > > --test-name=NAME > > > > The name of the test, with VPATH prefix (if any) removed. This can > > have a suffix and a directory component (as in e.g., sub/foo.test), and is > > mostly meant to be used in console reports about testsuite advancements and > > results (see Testsuite progress output). > > > > > > If test-name is _mostly meant to be used in console reports_ why is there > > any concern for a path? One can use any string in a report, paths are > > unnecessary, and you haven't specified any other use. > > > > So let me guess what you might be trying to say. From 15.3.1 Overview of > > Custom Test Drivers Support it appears that the test handler (responsible > > for test suite summaries and I suspect for executing test drivers) executes > > the _custom test driver_ which executes a test program, which must be a > > _test script_. The test script usage is directly contradicted in 15.3.2 > > Declaring Custom Test Drivers where _ext_ may or may not be a script suffix, > > but whose counting. > > > > The _custom test driver_ is responsible for interpreting command line > > arguments. One assumption is that when this argument is received the _custom > > test driver_ executes the NAME given in the command line argument. Of > > course, if this were true then it is possible to input a NAME which is not > > one of the known tests given by the developer. I don't know that this makes > > much sense, to execute a random script not defined by the developer. > > After the list of options it says: > > "The first non-option argument passed to the test driver is the program > to be run, and all the following ones are command-line options and > arguments for this program." > > By "program" it means test script. > > Maybe this could be clearer with a synopsis, like "DRIVER > DRIVER-OPTIONS -- TEST [TEST-OPTIONS]". It could be more explicit that > the test driver has to run the program it is given: maybe > > "The first non-option argument passed to the test driver is the test > script for it to run". >
In Apologia: If I sound (come across as) sarcastic, acerbic and short its because I'm frustrated. I've been at this since February and have gotten to the point where I have to parse each sentence to determine what it means. The document does not say what it means nor does it mean what it says. Your document quote at the end is an example (see http://lists.gnu.org/archive/html/automake/2015-04/msg00022.html thread). So now, grab a whiskey, light a candle, find a chair and I'll begin. In the beginning there was a test handler calling test drivers calling test cases. Now there are several test handlers, each one identified by automake in a different fashion. Only one test handler is active in any given generated script. The custom test driver is a test driver. It can be a script or a program (see the description of TEST_EXTENSIONS and their relation to LOG_DRIVERS). The custom test driver can call a test case. A test case can be a script or an executable program (see the descriptions of TEST_EXTENSIONS and LOG_DRIVERS and think on a windows *.exe test case being executed). So at this point we have discovered that the manual is incorrect is specifying that we must have a test script. Now for the case you mentioned in identifying a program, which is really a test case renamed in the manual for mystifying and unknowable reasons. Well, there is no mechanism for "The first non-option argument passed to the test driver is the program to be run, and all the following ones are command-line options and arguments for this program." Let's go through some of options. Suppose command line arguments are input on the command line (seems reasonable huh). Note that none of the command line inputs are identified or described in the make help file (make --help) or the make man pages (man make). I haven't looked at info make but, things being what they are, their probably not there either. So, since the manual doesn't describe how or what to do I guess we get to make it all up. In http://lists.gnu.org/archive/html/automake/2015-04/msg00022.html the question was asked as to how a program (test script) is to be entered. Let's see. Option 1: make program check Option 2: make program Option 3: make --test-name=NAME program check Option 4: make --test-name=NAME check Option 5: make program --test-name=NAME check Option 6: you can't do it Pick one or invent your own! That's how the command line options must be input. I can't think of another way. Now, once we input the command line options (and programs/test case) we pass it to the driver. Well, now we have something we can determine. testDriver [options] program That's what it (seems to) says and that's what it is. Only I can't figure how to get to this point. Remember that ext_LOG_DRIVER associates a driver (whatever that is) with a class of things to be driven. We are not dealing with scripts. We are dealing with everything, see 15.2.3 Parallel Test Harness parallel test example, which is really a custom driver example. And note that an example of custom test driver usage does not appear with the description of custom test drivers. Oh, about the example. The automake variable can't be used for anything but scripts, see 15 Support for test suites "The other (and most used) form is based on the use of generic test scripts, and its activation is triggered by the definition of the special TESTS variable." This quite clearly says that anytime that a TESTS variable is seen then generic scripts are used. But the example has "TESTS = foo.pl bar.py baz" which are not generic scripts. It must mean that the test driver is a generic script, but oh wait. The test drivers are for PERL and PYTHON and scripts. And (last time I looked) PERL and PYTHON were not generic scripts. One more little teenie-weenie thingie. Test handler types are recognized by all of: 1: Nothing. If there is nothing then the test handler executes bash on the test cases. 2: LOG anything. The test handler is for custom test drivers. And scripts handlers must be provided. 3: AUTOMAKE_OPTIONS = serial-tests: The test handler is a serial test handler (oh, by the way, the option is not in alphabetic order). 4: AUTOMAKE_OPTIONS = parallel-tests or missing, then the test handler is a parallel test handler. 5: AUTOMAKE_OPTIONS = dejagnu: The test handler handles a dejagnu test drivers? 6: TEST_LOG_DRIVER = env AM_TAP: The test handler is for TAP something. And oh, I'm frustrated. I'm sorry if the verbiage is sarcastic. I hope you will see how contradictory the manual is; how it lack precision and how it appears as an ad hoc product. art
