source: src/documentation/tests/regression-tests.dox@ 1cde364

/*
 * Project: MoleCuilder
 * Description: creates and alters molecular systems
 * Copyright (C)  2010 University of Bonn. All rights reserved.
 * Please see the LICENSE file or "Copyright notice" in builder.cpp for details.
 */

/**
 * \file regression-tests.dox
 *
 * This file contains explanation how to write, launch and use regression tests.
 *
 * Created on: Oct 11, 2011
 *    Author: heber
 */


/**
 * \page regressiontest "Regression tests"
 *
 * Regression test with this project are used to check on the functionality of
 * all Actions. They launch MoleCuilder from the command line with a given
 * action and option and basically diff the output against a stored file with
 * the desired result. This should be the behavior of more than 90% of all
 * regression tests. 
 * For some we additionally catch the output from stdout and stderr and diff against
 * it, but this is actually not a recommended way of testing as output depends very
 * much on the logging level.
 *
 * \section regressiontest-structure Directory structure
 *
 * They are contained in the source folder \b tests/regression. There are
 * categories containing a distinct folder for each action. The folder for the
 * test of a specific Action has the following structure:
 *  \li a folder \b pre contains all files required as input to the test.
 *  \li a folder \b post contains all output files against which we compare all
 *  or a number of resulting files.
 *  \li a test script \b testsuite-....at which is included in a similarly-
 *  named test script one directory level higher.
 *
 *  \section regressiontest-launch-all Launching all tests
 *
 *  Launching all regression tests is as simple as:
 *  -# Enter the build directory
 *  -# There, enter \b tests/regression
 *  -# Run
 *  \code make check \endcode
 *  (at you liberty with option \a -j8 or similar for running the tests in
 *  parallel.
 *
 *  \section regressiontest-launch-some Launching a specific tests
 *
 *  Launching a single or just some of the tests is only a little bit more
 *  complicated. There are two options: either by the test number which however
 *  changes when new tests are added, and by keywords.
 *
 *  Then proceed as follows:
 *  -# Enter the build directory
 *  -# There, enter \b tests/regression
 *  -# Run
 *  \code ../../../tests/regression/testsuite <option> AUTOTEST_PATH="<buildpath>/src" \endcode,
 *  where \a <option> is explained in the subsections below and \a <buildpath>
 *  is the build path (i.e. the variable \a AUTOTEST_PATH should contain the
 *  path to the executable).
 *
 *  \subsection regressiontest-launch-by-number ... by number
 *
 *    Tests can be launched by specifying their test number, e.g. then \a <option>
 *    might be \a 283 or \a 283-284 or \a 283,285-286 or alike
 *
 *  \subsection regressiontest-launch-by-keyword .. by keyword
 *
 *    Tests may as well be launched by some keywords, e.g. each Action has a
 *    specific token which is also one of its keyword (see above for the
 *    policy). I.e. we may launch the test on fill-void via the \a <option>
 *    \a -k \a fill-void. Also multiple keywords may be given.
 *
 * \section regressiontest-results Inspecting test results
 *
 *  The testsuite can be launched with the additional option of \a -d which
 *  leaves the directory of the test present even though the test has passed
 *  for inspection.
 *
 *  If a test fails, in whatever way it was launched, will leave in the build
 *  directory a folder \b tests/regression/testsuite.dir/<nr> where \a <nr> is
 *  the number of the test (padded maybe with some zeros).
 *
 * \subsection regressiontest-results-failures Typical failure causes
 *
 *  Here, we note down some typical failure causes:
 *  -# A test fails on installcheck on \code make distcheck \endcode: In the test
 *     a file is probably copied and then some work is done on it. Due to autoconf's
 *     distcheck policy, files copied from the archive are write-protected, hence
 *     cannot be modified. And so is the copy. The remedy is to add the line
 *     \code
 *     AT_CHECK([chmod u+w $file], 0)
 *     \endcode
 *     just after the copying to modify the write permission of the copied file \a $file
 *     and allow its modification.
 *
 * \section regressiontest-add Adding a new test
 *
 *  The basic working is: have a input file, do something with it and compare
 *  to output file against a stored one.
 *
 *  \attention Name convention of files and directories, e.g.
 *  \b tests/regression/Parser/Pdb with \b testsuite-parser-pdb-save.at
 *  - the test directory should either be called as the token of the Action it
 *    tests or a unique and brief description but with no spaces, no dashes,
 *    but CamelCase (i.e. Capitalize each new word)
 *  - the test script file should be called as follows:
 *    -# testsuite-...
 *    -# ...each directory (non-capital letters) with a dash...
 *    -# ..the name of the test directory...
 *    -# a further description of there are multiple test scripts in the
 *      test directory.
 *
 *  Adding a new regression tests consists of the following items:
 *  -# Create a new folder in a matching category
 *  -# Create therein subfolders \b pre and \b post
 *  -# Create a test script where the name begins with \b testsuite- and contains
 *  category and the action token. See other test scripts to get an idea on how
 *  to write these and also look here (http://www.gnu.org/s/hello/manual/autoconf/Using-Autotest.html#Using-Autotest).
 *  -# In the command \a AT_KEYWORD which must be present you should given the
 *  token of the Action as it would be called from the command line (see below for
 *  the reason).
 *  -# Include your testscript in the one present one directory-level up.
 *  -# Also include your script in \b tests/scripts/Makefile.am such that the
 *  testsuite is automatically re-compiled when one of the test files has changed.
 *
 *  On how to write a testsuite test, we refer you to one of these .at files to get a
 *  notion and  here to have a reference of the possible autotest commands at hand.
 *  The scheme is always the same basically:
 *  \code
 * AT_SETUP([General test part - description of this testsuite section])
 * AT_KEYWORDS([<some keywords>,<actionname>,[undo/redo]])
 * ...
 * AT_CHECK([this], <return value>, [ignore], [ignore])
 * AT_CHECK([that], <return value>, [stdout], [stderr])
 * AT_CHECk([fgrep "test fine" stdout], 0, [ignore], ignore])
 * ...
 * AT_CLEANUP #remove all temporary files
 * \endcode
 * where <return value> is some number the code returns to indicate everything
 * worked fine. The global theme is specified only once per .at file, file
 * AT_SETUP and AT_CLEANUP sort of embrace every specific test that you want to do.
 * Note that it is required to list the action name under AT_KEYWORDS and also
 * give undo oder redo as an additional keyword if the undo or redo of the action
 * is tested. It is advised to give further keywords, e.g. the directory name
 * giving the general theme of the tests (selection, analysis, ...). Also note
 * that all keywords are always lower-case!
 *
 * Note that testing the undo/redo-functionality of an Action is always placed
 * into the same test file along with the normal functionality but in different
 * tests (i.e. undo and redo each have their own AT_SETUP .. AT_CLEANUP wrapping).
 *
 *
 * \date 2011-10-31
 *
 */