| 1 | /*
|
|---|
| 2 | * Project: MoleCuilder
|
|---|
| 3 | * Description: creates and alters molecular systems
|
|---|
| 4 | * Copyright (C) 2010 University of Bonn. All rights reserved.
|
|---|
| 5 | * Please see the LICENSE file or "Copyright notice" in builder.cpp for details.
|
|---|
| 6 | */
|
|---|
| 7 |
|
|---|
| 8 | /**
|
|---|
| 9 | * \file regression-tests.dox
|
|---|
| 10 | *
|
|---|
| 11 | * This file contains explanation how to write, launch and use regression tests.
|
|---|
| 12 | *
|
|---|
| 13 | * Created on: Oct 11, 2011
|
|---|
| 14 | * Author: heber
|
|---|
| 15 | */
|
|---|
| 16 |
|
|---|
| 17 |
|
|---|
| 18 | /**
|
|---|
| 19 | * \page regressiontest "Regression tests"
|
|---|
| 20 | *
|
|---|
| 21 | * Regression test with this project are used to check on the functionality of
|
|---|
| 22 | * all Actions. The launch the MoleCuilder from the command line with a given
|
|---|
| 23 | * action and option and basically diff the output against a stored file with
|
|---|
| 24 | * the desired result. This should be the behavior of more than 90% of all
|
|---|
| 25 | * regression tests.
|
|---|
| 26 | *
|
|---|
| 27 | * \section regressiontest-structure Directory structure
|
|---|
| 28 | *
|
|---|
| 29 | * They are contained in the source folder \b tests/regression. There are
|
|---|
| 30 | * categories containing a distinct folder for each action. The folder for the
|
|---|
| 31 | * test of a specific Action has the following structure:
|
|---|
| 32 | * \li a folder \b pre contains all files required as input to the test.
|
|---|
| 33 | * \li a folder \b post contains all output files against which we compare all
|
|---|
| 34 | * or a number of resulting files.
|
|---|
| 35 | * \li a test script \b testsuite-....at which is included in a similarly-
|
|---|
| 36 | * named test script one directory level higher.
|
|---|
| 37 | *
|
|---|
| 38 | * \section regressiontest-launch-all Launching all tests
|
|---|
| 39 | *
|
|---|
| 40 | * Launching all regression tests is as simple as:
|
|---|
| 41 | * -# Enter the build directory
|
|---|
| 42 | * -# There, enter \b tests/regression
|
|---|
| 43 | * -# Run
|
|---|
| 44 | * \code make check \endcode
|
|---|
| 45 | * (at you liberty with option \a -j8 or similar for running the tests in
|
|---|
| 46 | * parallel.
|
|---|
| 47 | *
|
|---|
| 48 | * \section regressiontest-launch-some Launching a specific tests
|
|---|
| 49 | *
|
|---|
| 50 | * Launching a single or just some of the tests is only a little bit more
|
|---|
| 51 | * complicated. There are two options: either by the test number which however
|
|---|
| 52 | * changes when new tests are added, and by keywords.
|
|---|
| 53 | *
|
|---|
| 54 | * Then proceed as follows:
|
|---|
| 55 | * -# Enter the build directory
|
|---|
| 56 | * -# There, enter \b tests/regression
|
|---|
| 57 | * -# Run
|
|---|
| 58 | * \code ../../../tests/regression/testsuite <option> AUTOTEST_PATH="<buildpath>/src" \endcode,
|
|---|
| 59 | * where \a <option> is explained in the subsections below and \a <buildpath>
|
|---|
| 60 | * is the build path (i.e. the variable \a AUTOTEST_PATH should contain the
|
|---|
| 61 | * path to the executable).
|
|---|
| 62 | *
|
|---|
| 63 | * \subsection regressiontest-launch-by-number ... by number
|
|---|
| 64 | *
|
|---|
| 65 | * Tests can be launched by specifying their test number, e.g. then \a <option>
|
|---|
| 66 | * might be \a 283 or \a 283-284 or \a 283,285-286 or alike
|
|---|
| 67 | *
|
|---|
| 68 | * \subsection regressiontest-launch-by-keyword .. by keyword
|
|---|
| 69 | *
|
|---|
| 70 | * Tests may as well be launched by some keywords, e.g. each Action has a
|
|---|
| 71 | * specific token which is also one of its keyword (see above for the
|
|---|
| 72 | * policy). I.e. we may launch the test on fill-void via the \a <option>
|
|---|
| 73 | * \a -k \a fill-void. Also multiple keywords may be given.
|
|---|
| 74 | *
|
|---|
| 75 | * \section regressiontest-results Inspecting test results
|
|---|
| 76 | *
|
|---|
| 77 | * The testsuite can be launched with the additional option of \a -d which
|
|---|
| 78 | * leaves the directory of the test present even though the test has passed
|
|---|
| 79 | * for inspection.
|
|---|
| 80 | *
|
|---|
| 81 | * If a test fails, in whatever way it was launched, will leave in the build
|
|---|
| 82 | * directory a folder \b tests/regression/testsuite.dir/<nr> where \a <nr> is
|
|---|
| 83 | * the number of the test (padded maybe with some zeros).
|
|---|
| 84 | *
|
|---|
| 85 | * \section regressiontest-add Adding a new test
|
|---|
| 86 | *
|
|---|
| 87 | * \attention Name convention of files and directories, e.g.
|
|---|
| 88 | * \b tests/regression/Parser/Pdb with \b testsuite-parser-pdb-save.at
|
|---|
| 89 | * - the test directory should either be called as the token of the Action it
|
|---|
| 90 | * tests or a unique and brief description but with no spaces, no dashes,
|
|---|
| 91 | * but CamelCase (i.e. Capitalize each new word)
|
|---|
| 92 | * - the test script file should be called as follows:
|
|---|
| 93 | * -# testsuite-...
|
|---|
| 94 | * -# ...each directory (non-capital letters) with a dash...
|
|---|
| 95 | * -# ..the name of the test directory...
|
|---|
| 96 | * -# a further description of there are multiple test scripts in the
|
|---|
| 97 | * test directory.
|
|---|
| 98 | *
|
|---|
| 99 | * Adding a new regression tests consists of the following items:
|
|---|
| 100 | * -# Create a new folder in a matching category
|
|---|
| 101 | * -# Create therein subfolders \b pre and \b post
|
|---|
| 102 | * -# Create a test script where the name begins with \b testsuite- and contains
|
|---|
| 103 | * category and the action token. See other test scripts to get an idea on how
|
|---|
| 104 | * to write these and also look here (http://www.gnu.org/s/hello/manual/autoconf/Using-Autotest.html#Using-Autotest).
|
|---|
| 105 | * -# In the command \a AT_KEYWORD which must be present you should given the
|
|---|
| 106 | * token of the Action as it would be called from the command line (see below for
|
|---|
| 107 | * the reason).
|
|---|
| 108 | * -# Include your testscript in the one present one directory-level up.
|
|---|
| 109 | * -# Also include your script in \b tests/scripts/Makefile.am such that the
|
|---|
| 110 | * testsuite is automatically re-compiled when one of the test files has changed.
|
|---|
| 111 | *
|
|---|
| 112 | *
|
|---|
| 113 | * \date 2011-10-31
|
|---|
| 114 | *
|
|---|
| 115 | */
|
|---|
