Changes in directory llvm/docs:
TestingGuide.html updated: 1.40 -> 1.41 --- Log message: For PR1319: http://llvm.org/PR1319 : Rewrite much of the DejaGnu section to bring it in line with the new facilities in llvm.exp. --- Diffs of the changes: (+192 -74) TestingGuide.html | 266 ++++++++++++++++++++++++++++++++++++++---------------- 1 files changed, 192 insertions(+), 74 deletions(-) Index: llvm/docs/TestingGuide.html diff -u llvm/docs/TestingGuide.html:1.40 llvm/docs/TestingGuide.html:1.41 --- llvm/docs/TestingGuide.html:1.40 Thu Feb 8 11:00:55 2007 +++ llvm/docs/TestingGuide.html Sat Apr 14 16:46:15 2007 @@ -275,81 +275,199 @@ <!--=========================================================================--> <div class="doc_section"><a name="dgstructure">DejaGNU Structure</a></div> <!--=========================================================================--> - <div class="doc_text"> -<p>The LLVM test suite is partially driven by DejaGNU and partially -driven by GNU Make. Specifically, the Features and Regression tests -are all driven by DejaGNU. The <tt>llvm-test</tt> -module is currently driven by a set of Makefiles.</p> - -<p>The DejaGNU structure is very simple, but does require some -information to be set. This information is gathered via <tt>configure</tt> and -is written to a file, <tt>site.exp</tt> in <tt>llvm/test</tt>. The -<tt>llvm/test</tt> -Makefile does this work for you.</p> - -<p>In order for DejaGNU to work, each directory of tests must have a -<tt>dg.exp</tt> file. This file is a program written in tcl that calls -the <tt>llvm-runtests</tt> procedure on each test file. The -llvm-runtests procedure is defined in -<tt>llvm/test/lib/llvm-dg.exp</tt>. Any directory that contains only -directories does not need the <tt>dg.exp</tt> file.</p> - -<p>In order for a test to be run, it must contain information within -the test file on how to run the test. These are called <tt>RUN</tt> -lines. Run lines are specified in the comments of the test program -using the keyword <tt>RUN</tt> followed by a colon, and lastly the -commands to execute. These commands will be executed in a bash script, -so any bash syntax is acceptable. You can specify as many RUN lines as -necessary. Each RUN line translates to one line in the resulting bash -script. Below is an example of legal RUN lines in a <tt>.ll</tt> -file:</p> -<pre> -; RUN: llvm-as < %s | llvm-dis > %t1 -; RUN: llvm-dis < %s.bc-13 > %t2 -; RUN: diff %t1 %t2 -</pre> -<p>There are a couple patterns within a <tt>RUN</tt> line that the -llvm-runtest procedure looks for and replaces with the appropriate -syntax:</p> - -<dl style="margin-left: 25px"> -<dt>%p</dt> -<dd>The path to the source directory. This is for locating -any supporting files that are not generated by the test, but used by -the test.</dd> -<dt>%s</dt> -<dd>The test file.</dd> - -<dt>%t</dt> -<dd>Temporary filename: testscript.test_filename.tmp, where -test_filename is the name of the test file. All temporary files are -placed in the Output directory within the directory the test is -located.</dd> - -<dt>%prcontext</dt> -<dd>Path to a script that performs grep -C. Use this since not all -platforms support grep -C.</dd> - -<dt>%llvmgcc</dt> <dd>Full path to the llvm-gcc executable.</dd> -<dt>%llvmgxx</dt> <dd>Full path to the llvm-g++ executable.</dd> -</dl> - -<p>There are also several scripts in the llvm/test/Scripts directory -that you might find useful when writing <tt>RUN</tt> lines.</p> + <p>The LLVM test suite is partially driven by DejaGNU and partially driven by + GNU Make. Specifically, the Features and Regression tests are all driven by + DejaGNU. The <tt>llvm-test</tt> module is currently driven by a set of + Makefiles.</p> + + <p>The DejaGNU structure is very simple, but does require some information to + be set. This information is gathered via <tt>configure</tt> and is written + to a file, <tt>site.exp</tt> in <tt>llvm/test</tt>. The <tt>llvm/test</tt> + Makefile does this work for you.</p> + + <p>In order for DejaGNU to work, each directory of tests must have a + <tt>dg.exp</tt> file. DejaGNU looks for this file to determine how to run the + tests. This file is just a Tcl script and it can do anything you want, but + we've standardized it for the LLVM regression tests. It simply loads a Tcl + library (<tt>test/lib/llvm.exp</tt>) and calls the <tt>llvm_runtests</tt> + function defined in that library with a list of file names to run. The names + are obtained by using Tcl's glob command. Any directory that contains only + directories does not need the <tt>dg.exp</tt> file.</p> + + <p>The <tt>llvm-runtests</tt> function lookas at each file that is passed to + it and gathers any lines together that match "RUN:". This are the "RUN" lines + that specify how the test is to be run. So, each test script must contain + RUN lines if it is to do anything. If there are no RUN lines, the + <tt>llvm-runtests</tt> function will issue an error and the test will + fail.</p> + + <p>RUN lines are specified in the comments of the test program using the + keyword <tt>RUN</tt> followed by a colon, and lastly the command (pipeline) + to execute. Together, these lines form the "script" that + <tt>llvm-runtests</tt> executes to run the test case. The syntax of the + RUN lines is similar to a shell's syntax for pipelines including I/O + redirection and variable substitution. However, even though these lines + may <i>look</i> like a shell script, they are not. RUN lines are interpreted + directly by the Tcl <tt>exec</tt> command. They are never executed by a + shell. Consequently the syntax differs from normal shell script syntax in a + few ways. You can specify as many RUN lines as needed.</p> + + <p>Each RUN line is executed on its own, distinct from other lines unless + its last character is <tt>\</tt>. This continuation character causes the RUN + line to be concatenated with the next one. In this way you can build up long + pipelines of commands without making huge line lengths. The lines ending in + <tt>\</tt> are concatenated until a RUN line that doesn't end in <tt>\</tt> is + found. This concatenated set or RUN lines then constitutes one execution. + Tcl will substitute variables and arrange for the pipeline to be executed. If + any process in the pipeline fails, the entire line (and test case) fails too. + </p> + + <p> Below is an example of legal RUN lines in a <tt>.ll</tt> file:</p> + <pre> + ; RUN: llvm-as < %s | llvm-dis > %t1 + ; RUN: llvm-dis < %s.bc-13 > %t2 + ; RUN: diff %t1 %t2 + </pre> +</div> -<p>Lastly, you can easily mark a test that is expected to fail on a -specific platform or with a specific version of llvmgcc by using the - <tt>XFAIL</tt> keyword. Xfail lines are -specified in the comments of the test program using <tt>XFAIL</tt>, -followed by a colon, and one or more regular expressions (separated by -a comma) that will match against the target triplet or llvmgcc version for the -machine. You can use * to match all targets. You can specify the major or full - version (i.e. 3.4) for llvmgcc. Here is an example of an -<tt>XFAIL</tt> line:</p> -<pre> -; XFAIL: darwin,sun,llvmgcc4 -</pre> +<!-- _______________________________________________________________________ --> +<div class="doc_subsection"><a name="dgvars">Vars And Substitutions</a></div> +<div class="doc_text"> + <p>With a RUN line there are a number of substitutions that are permitted. In + general, any Tcl variable that is available in the <tt>substitute</tt> + function (in <tt>test/lib/llvm.exp</tt>) can be substituted into a RUN line. + To make a substitution just write the variable's name preceded by a $. + Additionally, for compatibility reasons with previous versions of the test + library, certain names can be accessed with an alternate syntax: a % prefix. + These alternates are deprecated and may go away in a future version. + </p> + Here are the available variable names. The alternate syntax is listed in + parentheses.</p> + <dl style="margin-left: 25px"> + <dt><b>$test</b> (%s)</dt> + <dd>The full path to the test case's source. This is suitable for passing + on the command line as the input to an llvm tool.</dd> + <dt><b>$srcdir</b></dt> + <dd>The source directory from where the "<tt>make check</tt>" was run.</dd> + <dt><b>objdir</b></dt> + <dd>The object directory that corresponds to the </tt>$srcdir</tt>.</dd> + <dt><b>subdir</b></dt> + <dd>A partial path from the <tt>test</tt> directory that contains the + sub-directory that contains the test source being executed.</dd> + <dt><b>srcroot</b></dt> + <dd>The root directory of the LLVM src tree.</dd> + <dt><b>objroot</b></dt> + <dd>The root directory of the LLVM object tree. This could be the same + as the srcroot.</dd> + <dt><b>path</b><dt> + <dd>The path to the directory that contains the test case source. This is + for locating any supporting files that are not generated by the test, but + used by the test.</dd> + <dt><b>tmp</b></dt> + <dd>The path to a temporary file name that could be used for this test case. + The file name won't conflict with other test cases. You can append to it if + you need multiple temporaries. This is useful as the destination of some + redirected output.</dd> + <dt><b>llvmlibsdir</b> (%llvmlibsdir)</dt> + <dd>The directory where the LLVM libraries are located.</dd> + <dt><b>target_triplet</b> (%target_triplet)</dt> + <dd>The target triplet that corresponds to the current host machine (the one + running the test cases). This should probably be called "host".<dd> + <dt><b>prcontext</b> (%prcontext)</dt> + <dd>Path to the prcontext tcl script that prints some context around a + line that matches a pattern. This isn't strictly necessary as the test suite + is run with its PATH altered to include the test/Scripts directory where + the prcontext script is located. Note that this script is similar to + <tt>grep -C</tt> but you should use the <tt>prcontext</tt> script because + not all platforms support <tt>grep -C</tt>.</dd> + <dt><b>llvmgcc</b> (%llvmgcc)</dt> + <dd>The full path to the <tt>llvm-gcc</tt> executable as specified in the + configured LLVM environment</dd> + <dt><b>llvmgxx</b> (%llvmgxx)</dt> + <dd>The full path to the <tt>llvm-gxx</tt> executable as specified in the + configured LLVM environment</dd> + <dt><b>llvmgcc_version</b> (%llvmgcc_version)</dt> + <dd>The full version number of the <tt>llvm-gcc</tt> executable.</dd> + <dt><b>llvmgccmajvers</b> (%llvmgccmajvers)</dt> + <dd>The major version number of the <tt>llvm-gcc</tt> executable.</dd> + <dt><b>gccpath</b></dt> + <dd>The full path to the C compiler used to <i>build </i> LLVM. Note that + this might not be gcc.</dd> + <dt><b>gxxpath</b></dt> + <dd>The full path to the C++ compiler used to <i>build </i> LLVM. Note that + this might not be g++.</dd> + <dt><b>compile_c</b> (%compile_c)</dt> + <dd>The full command line used to compile LLVM C source code. This has all + the configured -I, -D and optimization options.</dd> + <dt><b>compile_cxx</b> (%compile_cxx)</dt> + <dd>The full command used to compile LLVM C++ source code. This has + all the configured -I, -D and optimization options.</dd> + <dt><b>link</b> (%link)</dt> + <dd>This full link command used to link LLVM executables. This has all the + configured -I, -L and -l options.</dd> + <dt><b>shlibext</b> (%shlibext)</dt> + <dd>The suffix for the host platforms share library (dll) files. This + includes the period as the first character.</dd> + </dl> + <p>To add more variables, two things need to be changed. First, add a line in + the <tt>test/Makefile</tt> that creates the <tt>site.exp</tt> file. This will + "set" the variable as a global in the site.exp file. Second, in the + <tt>test/lib/llvm.exp</tt> file, in the substitute proc, add the variable name + to the list of "global" declarations at the beginning of the proc. That's it, + the variable can then be used in test scripts.</p> +</div> + +<!-- _______________________________________________________________________ --> +<div class="doc_subsection"><a name="dgfeatures">Other Features</a></div> +<div class="doc_text"> + <p>To make RUN line writing easier, there are several shell scripts located + in the <tt>llvm/test/Scripts</tt> directory. For example:</p> + <dl> + <dt><b>ignore</b></dt> + <dd>This script runs its arguments and then always returns 0. This is useful + in cases where the test needs to cause a tool to generate an error (e.g. to + check the error output). However, any program in a pipeline that returns a + non-zero result will cause the test to fail. This script overcomes that + issue and nicely documents that the test case is purposefully ignoring the + result code of the tool</dd> + <dt><b>not</b></dt> + <dd>This script runs its arguments and then inverts the result code from + it. Zero result codes become 1. Non-zero result codes become 0. This is + useful to invert the result of a grep. For example "not grep X" means + succeed only if you don't find X in the input.</dd> + </dl> + + <p>Sometimes it is necessary to mark a test case as "expected fail" or XFAIL. + You can easily mark a test as XFAIL just by including <tt>XFAIL: </tt> on a + line near the top of the file. This signals that the test case should succeed + if the test fails. Such test cases are counted separately by DejaGnu. To + specify an expected fail, use the XFAIL keyword in the comments of the test + program followed by a colon and one or more regular expressions (separated by + a comma). The regular expressions allow you to XFAIL the test conditionally + by host platform. The regular expressions following the : are matched against + the target triplet or llvmgcc version number for the host machine. If there is + a match, the test is expected to fail. If not, the test is expected to + succeed. To XFAIL everywhere just specify <tt>XFAIL: *</tt>. When matching + the llvm-gcc version, you can specify the major (e.g. 3) or full version + (i.e. 3.4) number. Here is an example of an <tt>XFAIL</tt> line:</p> + <pre> + ; XFAIL: darwin,sun,llvmgcc4 + </pre> + + <p>To make the output more useful, the <tt>llvm_runtest</tt> function wil + scan the lines of the test case for ones that contain a pattern that matches + PR[0-9]+. This is the syntax for specifying a PR (Problem Report) number that + is related to the test case. The numer after "PR" specifies the LLVM bugzilla + number. When a PR number is specified, it will be used in the pass/fail + reporting. This is useful to quickly get some context when a test fails.</p> + + <p>Finally, any line that contains "END." will cause the special + interpretation of lines to terminate. This is generally done right after the + last RUN: line. This has two side effects: (a) it prevents special + interpretation of lines that are part of the test program, not the + instructions to the test case, and (b) it speeds things up for really big test + cases by avoiding interpretation of the remainder of the file.</p> </div> @@ -638,7 +756,7 @@ John T. Criswell, Reid Spencer, and Tanya Lattner<br> <a href="http://llvm.org";>The LLVM Compiler Infrastructure</a><br/> - Last modified: $Date: 2007/02/08 17:00:55 $ + Last modified: $Date: 2007/04/14 21:46:15 $ </address> </body> </html> _______________________________________________ llvm-commits mailing list [EMAIL PROTECTED] http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
