source: branches/stable/UNIT_TESTER/README.txt

How to use ARB unit testing
---------------------------

1. Invocation

   Edit config.makefile. Set UNIT_TESTS to 1

   cd $ARBHOME
   make unit_tests

   'make all' runs tests as well ('make build' doesn't)

2. Add unit test code

   Go to the unit containing your_function() and add something like
   the following to the end of the file.

       #if (UNIT_TESTS == 1)

       #include <test_unit.h>

       void TEST_your_function() {
       }

       #endif

   The names of the test functions are important:
       - they have to start with 'TEST_' and
       - need to be visible at global scope.

   (see also 'Building tests' at end of file)


   Test result:

      If TEST_your_function() does nothing special, it is assumed the unit test passed "ok".

      If your function fails an assertion (TEST_ASSERT) or raises SIGSEGV in any other manner,
      it is assumed the unit test failed and an error message is printed to the console.

      (Any other test function in the same unit will nevertheless get called.)


   Post conditions:

      If a unit defines one or more tests called
         void TEST_POSTCOND_xxx()
      each of these post condition tests will be run after EACH single unit test.

      These post condition tests are intended to check for unwanted global side-effects of
      test-functions. Such side effects often let other (unrelated) test-functions fail
      unexpectedly. Use them to undo these side-effects.

      If the post-condition fails, the test itself will be reported as failure.

      If the test itself has failed for other reasons, post condition tests will nevertheless
      be executed (and may do their cleanups), but their failures will not be shown in the log.


   Failing tests:

      If you have some broken behavior that you cannot fix now, please do NOT leave
      the test as "failing". Instead change TEST_ASSERT into

          TEST_ASSERT_BROKEN(failing_condition);

      That will print a warning as reminder as long as the condition fails.
      When the behavior was fixed (so that the condition is fulfilled now),
      it will abort the test as failed!
      Just change TEST_ASSERT_BROKEN back into TEST_ASSERT then.

      Set WARN_LEVEL to 0 in Makefile.setup.local to disable warnings.

      Random failures:

             Some tests tend to fail randomly if started from inside jenkins (due to
             parallel builds), whereas they behave proper and provide safety during
             normal development. You should disable such tests for jenkins by enclosing
             the whole test with:

                #if !defined(DEVEL_JENKINS)
                #endif

             One example for such a test is ../CORE/arb_cs.cxx@TEST_open_socket

             Another possibility to handle random failures is to allow multiple tries.
             This can be done using the macro TEST_ALLOW_RETRY(count). If called from
             testcode this will allow the test to run 'count' times, failing only if
             all 'count' calls of the test function result in failure.
             Note: TEST_ALLOW_RETRY has no effect if a test already gets retried!

   Missing tests:

      You may drop a reminder like

          MISSING_TEST(describe what is missing);

      which will appear as warning during test run.


   Order of tests:

      Tests are executed in the order they appear in the code.
      Multiple Files are sorted alphabethically.

      Exceptions:
      - If test name starts with 'TEST_###' (where ### are digits), it declares a test with priority ###.
        The default priority is 100. Lower numbers mean higher priority, i.e. start more early. 
      - Other predefined priorities are
        - TEST_BASIC_       (=20)
        - TEST_EARLY_       (=50)
        - TEST_             (=100 default)
        - TEST_LATE_        (=200)
        - TEST_SLOW_        (=900)
        - TEST_AFTER_SLOW_  (=910)

        TEST_SLOW_... is meant to indicate "slow" tests (i.e. tests which need more than
        a second to execute).

   Order of test units:

      Tests from each unit (aka library) run consecutive.
      Tests of different units are executed asynchronously.

   Code differences when compiled with UNIT_TESTS=1

      Some unit tests require differences in production code, e.g.
      - to inspect class internals
      - to provoke special conditions (e.g. reduce some buffer size to simulate big
        amount of processed data)

      Such differences should be marked with a comment containing 'UT_DIFF'.


3. Valgrinding test code

   Note: valgrind was not used for some time and might have rotten.
         I've completely switched to using sanitizers -> see ../config.makefile


   If you set

      VALGRIND=A

   in UNIT_TESTER/Makefile.setup.local, valgrind will be started on the test-binary after the normal
   unit tests passed with success.
   Valgrind errors/warnings will not raise an error or abort testing.


   If you set

      VALGRIND=B

   valgrind will be started BEFORE running tests.
   Useful if test fail unexpectedly and you suspect a memory bug.

   See also ENABLE_CRASH_TESTS in test_unit.h (all crash tests are reported by valgrind)


   If you set

      VALGRIND=E

   some external tools like arb_pt_server will be spawned valgrinded and the generated
   reports will be printed on test termination.

   Any combination of the above works as well.

4. Check test-coverage

   set 'COVERAGE=1' in config.makefile and 'make rebuild'

   - prints out uncovered code to compile log
   - leaves info for partly covered code files in yourcode.cxx.cov

   Note:
        You can silence lines which are never reached while testing
        with a comment containing 'NEED_NO_COV'.

   call 'make clean_coverage_results' to get rid of .cov files

   (see also showCoverageForAll in gcov2msg.pl)

5. Define which tests shall run

   [look into Makefile.setup.local for examples]

   a. Restrict tests to one or several libraries:

      Set 'RESTRICT_LIB=lib1[:libN]*' to run only tests defined in the
      specified libraries.

      Set 'RESTRICT_LIB=' to test all libraries.

   b. Restrict tests to specific modules:

      Set "RESTRICT_MODULE='regexpr'" to run only tests located in
      matching modules. Use '.' to match any modules.
      Prefix whole regexpr with '!' to match all but listed modules.

   c. Restrict tests by name:

      Set "RESTRICT_FUN='regexpr'" to run only tests whose function-name
      matches the regexpr. Use '.' to match any test.
      Prefix whole regexpr with '!' to match all but listed tests.

   d. Restrict test frequency:

      Set 'SKIP_SLOW=min' to restrict frequency of "slow tests". Slow tests
      are all tests named 'TEST_SLOW_.*'. These tests will not run more often
      than the specified amount of minutes.
      Set 'SKIP_SLOW=0' to always run all tests (default).

   e. Run unchanged tests:

      By default only changed unit-tests will run. Set 'ONLY_CHANGED_TESTS=0'
      to force testing unchanged code.

      Tests will re-run if ..
      - the test code was changed,
      - any needed library was changed or if
      - Makefile.setup.local or anything in the UnitTester itself was changed.

      Calling 'make clean' in $ARBHOME will also force all tests to re-run.


6. Global test environments

   Environments provide a test situation which takes some reasonable time
   for startup.

   Multiple tests will be permitted to use one environment, but they will
   be serialized (even if tests are running in multiple processes).

   These environments have to be coded directly in TestEnvironment.cxx
   (for existing environments see TestEnvironment.cxx@ExistingEnvironments)

   If you write a test that wishes to use one of the environments simply write
      TEST_SETUP_GLOBAL_ENVIRONMENT("name"); // starts environment 'name'


7. Auto-patch generation

   Whenever unit tests complete successfully, a patch (svn diff) is generated and
   stored inside $ARBHOME/patches.arb/

   After PATCHES_KEEP_HOURS seconds the patch will be deleted automatically
   (see Makefile.setup.local), considering at least PATCHES_MIN_KEPT patches remain.

8. Building tests

   To build and execute a test-executable for a previously untested subdir
   the following steps are necessary:

   - add the test to the list of performed tests in ../Makefile@UNITS_TESTED
   - if the test subject is a shared library, be sure to mention it a Makefile.test@SHAREDLIBS
   - if the test subject is an executable, be sure to mention it a Makefile.test@EXEOBJDIRS