source: trunk/UNIT_TESTER/README.txt

Last change on this file was 16778, checked in by westram, 3 months ago
  • unittests may permit retries using TEST_ALLOW_RETRY
    • workaround for randomly failing tests
File size: 8.1 KB
Line 
1
2How to use ARB unit testing
3---------------------------
4
51. Invocation
6
7   Edit config.makefile. Set UNIT_TESTS to 1
8
9   cd $ARBHOME
10   make unit_tests
11
12   'make all' runs tests as well ('make build' doesn't)
13
142. Add unit test code
15
16   Go to the unit containing your_function() and add something like
17   the following to the end of the file.
18
19       #if (UNIT_TESTS == 1)
20
21       #include <test_unit.h>
22
23       void TEST_your_function() {
24       }
25
26       #endif
27
28   The names of the test functions are important:
29       - they have to start with 'TEST_' and
30       - need to be visible at global scope.
31
32   (see also 'Building tests' at end of file)
33
34
35   Test result:
36
37      If TEST_your_function() does nothing special, it is assumed the unit test passed "ok".
38
39      If your function fails an assertion (TEST_ASSERT) or raises SIGSEGV in any other manner,
40      it is assumed the unit test failed and an error message is printed to the console.
41
42      (Any other test function in the same unit will nevertheless get called.)
43
44
45   Post conditions:
46
47      If a unit defines one or more tests called
48         void TEST_POSTCOND_xxx()
49      each of these post condition tests will be run after EACH single unit test.
50
51      These post condition tests are intended to check for unwanted global side-effects of
52      test-functions. Such side effects often let other (unrelated) test-functions fail
53      unexpectedly. Use them to undo these side-effects.
54
55      If the post-condition fails, the test itself will be reported as failure.
56
57      If the test itself has failed for other reasons, post condition tests will nevertheless
58      be executed (and may do their cleanups), but their failures will not be shown in the log.
59
60
61   Failing tests:
62
63      If you have some broken behavior that you cannot fix now, please do NOT leave
64      the test as "failing". Instead change TEST_ASSERT into
65
66          TEST_ASSERT_BROKEN(failing_condition);
67
68      That will print a warning as reminder as long as the condition fails.
69      When the behavior was fixed (so that the condition is fulfilled now),
70      it will abort the test as failed!
71      Just change TEST_ASSERT_BROKEN back into TEST_ASSERT then.
72
73      Set WARN_LEVEL to 0 in Makefile.setup.local to disable warnings.
74
75      Random failures:
76
77             Some tests tend to fail randomly if started from inside jenkins (due to
78             parallel builds), whereas they behave proper and provide safety during
79             normal development. You should disable such tests for jenkins by enclosing
80             the whole test with:
81
82                #if !defined(DEVEL_JENKINS)
83                #endif
84
85             One example for such a test is ../CORE/arb_cs.cxx@TEST_open_socket
86
87             Another possibility to handle random failures is to allow multiple tries.
88             This can be done using the macro TEST_ALLOW_RETRY(count). If called from
89             testcode this will allow the test to run 'count' times, failing only if
90             all 'count' calls of the test function result in failure.
91             Note: TEST_ALLOW_RETRY has no effect if a test already gets retried!
92
93   Missing tests:
94
95      You may drop a reminder like
96
97          MISSING_TEST(describe what is missing);
98
99      which will appear as warning during test run.
100
101
102   Order of tests:
103
104      Tests are executed in the order they appear in the code.
105      Multiple Files are sorted alphabethically.
106
107      Exceptions:
108      - If test name starts with 'TEST_###' (where ### are digits), it declares a test with priority ###.
109        The default priority is 100. Lower numbers mean higher priority, i.e. start more early.
110      - Other predefined priorities are
111        - TEST_BASIC_       (=20)
112        - TEST_EARLY_       (=50)
113        - TEST_             (=100 default)
114        - TEST_LATE_        (=200)
115        - TEST_SLOW_        (=900)
116        - TEST_AFTER_SLOW_  (=910)
117
118        TEST_SLOW_... is meant to indicate "slow" tests (i.e. tests which need more than
119        a second to execute).
120
121   Order of test units:
122
123      Tests from each unit (aka library) run consecutive.
124      Tests of different units are executed asynchronously.
125
126   Code differences when compiled with UNIT_TESTS=1
127
128      Some unit tests require differences in production code, e.g.
129      - to inspect class internals
130      - to provoke special conditions (e.g. reduce some buffer size to simulate big
131        amount of processed data)
132
133      Such differences should be marked with a comment containing 'UT_DIFF'.
134
135
1363. Valgrinding test code
137
138   If you set
139
140      VALGRIND=A
141
142   in UNIT_TESTER/Makefile.setup.local, valgrind will be started on the test-binary after the normal
143   unit tests passed with success.
144   Valgrind errors/warnings will not raise an error or abort testing.
145
146
147   If you set
148
149      VALGRIND=B
150
151   valgrind will be started BEFORE running tests.
152   Useful if test fail unexpectedly and you suspect a memory bug.
153
154   See also ENABLE_CRASH_TESTS in test_unit.h (all crash tests are reported by valgrind)
155
156
157   If you set
158
159      VALGRIND=E
160
161   some external tools like arb_pt_server will be spawned valgrinded and the generated
162   reports will be printed on test termination.
163
164   Any combination of the above works as well.
165
1664. Check test-coverage
167
168   set 'COVERAGE=1' in config.makefile and 'make rebuild'
169
170   - prints out uncovered code to compile log
171   - leaves info for partly covered code files in yourcode.cxx.cov
172
173   Note:
174        You can silence lines which are never reached while testing
175        with a comment containing 'NEED_NO_COV'.
176
177   call 'make clean_coverage_results' to get rid of .cov files
178
179   (see also showCoverageForAll in gcov2msg.pl)
180
1815. Define which tests shall run
182
183   [look into Makefile.setup.local for examples]
184
185   a. Restrict tests to one or several libraries:
186
187      Set 'RESTRICT_LIB=lib1[:libN]*' to run only tests defined in the
188      specified libraries.
189
190      Set 'RESTRICT_LIB=' to test all libraries.
191
192   b. Restrict tests to specific modules:
193
194      Set "RESTRICT_MODULE='regexpr'" to run only tests located in
195      matching modules. Use '.' for all modules.
196
197   c. Restrict tests by name:
198
199      Set "RESTRICT_FUN='regexpr'" to run only tests whose function-name
200      matches the regexpr. Use '.' for all tests.
201
202   d. Restrict test frequency:
203
204      Set 'SKIP_SLOW=min' to restrict frequency of "slow tests". Slow tests
205      are all tests named 'TEST_SLOW_.*'. These tests will not run more often
206      than the specified amount of minutes.
207      Set 'SKIP_SLOW=0' to always run all tests (default).
208
209   e. Run unchanged tests:
210
211      By default only changed unit-tests will run. Set 'ONLY_CHANGED_TESTS=0'
212      to force testing unchanged code.
213
214      Tests will re-run if ..
215      - the test code was changed,
216      - any needed library was changed or if
217      - Makefile.setup.local or anything in the UnitTester itself was changed.
218
219      Calling 'make clean' in $ARBHOME will also force all tests to re-run.
220
221
2226. Global test environments
223
224   Environments provide a test situation which takes some reasonable time
225   for startup.
226
227   Multiple tests will be permitted to use one environment, but they will
228   be serialized (even if tests are running in multiple processes).
229
230   These environments have to be coded directly in TestEnvironment.cxx
231   (for existing environments see TestEnvironment.cxx@ExistingEnvironments)
232
233   If you write a test that wishes to use one of the environments simply write
234      TEST_SETUP_GLOBAL_ENVIRONMENT("name"); // starts environment 'name'
235
236
2377. Auto-patch generation
238
239   Whenever unit tests complete successfully, a patch (svn diff) is generated and
240   stored inside $ARBHOME/patches.arb/
241
242   After PATCHES_KEEP_HOURS seconds the patch will be deleted automatically
243   (see Makefile.setup.local), considering at least PATCHES_MIN_KEPT patches remain.
244
2458. Building tests
246
247   To build and execute a test-executable for a previously untested subdir
248   the following steps are necessary:
249
250   - add the test to the list of performed tests in ../Makefile@UNITS_TESTED
251   - if the test subject is a shared library, be sure to mention it a Makefile.test@SHAREDLIBS
252   - if the test subject is an executable, be sure to mention it a Makefile.test@EXEOBJDIRS
Note: See TracBrowser for help on using the repository browser.