source: trunk/UNIT_TESTER/README.txt

Last change on this file was 17428, checked in by westram, 2 months ago
  • partial merge from 'svalues' into 'trunk'
    • root branches always need to have identical remarks (associated with the root-edge)
      • condition previously implicit, now hardened by tests/assertions/…
      • fixed several bugs that violated this condition
  • adds: log:branches/svalues@17420:17427
File size: 8.2 KB
Line 
1
2How to use ARB unit testing
3---------------------------
4
51. Invocation
6
7   Edit config.makefile. Set UNIT_TESTS to 1
8
9   cd $ARBHOME
10   make unit_tests
11
12   'make all' runs tests as well ('make build' doesn't)
13
142. Add unit test code
15
16   Go to the unit containing your_function() and add something like
17   the following to the end of the file.
18
19       #if (UNIT_TESTS == 1)
20
21       #include <test_unit.h>
22
23       void TEST_your_function() {
24       }
25
26       #endif
27
28   The names of the test functions are important:
29       - they have to start with 'TEST_' and
30       - need to be visible at global scope.
31
32   (see also 'Building tests' at end of file)
33
34
35   Test result:
36
37      If TEST_your_function() does nothing special, it is assumed the unit test passed "ok".
38
39      If your function fails an assertion (TEST_ASSERT) or raises SIGSEGV in any other manner,
40      it is assumed the unit test failed and an error message is printed to the console.
41
42      (Any other test function in the same unit will nevertheless get called.)
43
44
45   Post conditions:
46
47      If a unit defines one or more tests called
48         void TEST_POSTCOND_xxx()
49      each of these post condition tests will be run after EACH single unit test.
50
51      These post condition tests are intended to check for unwanted global side-effects of
52      test-functions. Such side effects often let other (unrelated) test-functions fail
53      unexpectedly. Use them to undo these side-effects.
54
55      If the post-condition fails, the test itself will be reported as failure.
56
57      If the test itself has failed for other reasons, post condition tests will nevertheless
58      be executed (and may do their cleanups), but their failures will not be shown in the log.
59
60
61   Failing tests:
62
63      If you have some broken behavior that you cannot fix now, please do NOT leave
64      the test as "failing". Instead change TEST_ASSERT into
65
66          TEST_ASSERT_BROKEN(failing_condition);
67
68      That will print a warning as reminder as long as the condition fails.
69      When the behavior was fixed (so that the condition is fulfilled now),
70      it will abort the test as failed!
71      Just change TEST_ASSERT_BROKEN back into TEST_ASSERT then.
72
73      Set WARN_LEVEL to 0 in Makefile.setup.local to disable warnings.
74
75      Random failures:
76
77             Some tests tend to fail randomly if started from inside jenkins (due to
78             parallel builds), whereas they behave proper and provide safety during
79             normal development. You should disable such tests for jenkins by enclosing
80             the whole test with:
81
82                #if !defined(DEVEL_JENKINS)
83                #endif
84
85             One example for such a test is ../CORE/arb_cs.cxx@TEST_open_socket
86
87             Another possibility to handle random failures is to allow multiple tries.
88             This can be done using the macro TEST_ALLOW_RETRY(count). If called from
89             testcode this will allow the test to run 'count' times, failing only if
90             all 'count' calls of the test function result in failure.
91             Note: TEST_ALLOW_RETRY has no effect if a test already gets retried!
92
93   Missing tests:
94
95      You may drop a reminder like
96
97          MISSING_TEST(describe what is missing);
98
99      which will appear as warning during test run.
100
101
102   Order of tests:
103
104      Tests are executed in the order they appear in the code.
105      Multiple Files are sorted alphabethically.
106
107      Exceptions:
108      - If test name starts with 'TEST_###' (where ### are digits), it declares a test with priority ###.
109        The default priority is 100. Lower numbers mean higher priority, i.e. start more early.
110      - Other predefined priorities are
111        - TEST_BASIC_       (=20)
112        - TEST_EARLY_       (=50)
113        - TEST_             (=100 default)
114        - TEST_LATE_        (=200)
115        - TEST_SLOW_        (=900)
116        - TEST_AFTER_SLOW_  (=910)
117
118        TEST_SLOW_... is meant to indicate "slow" tests (i.e. tests which need more than
119        a second to execute).
120
121   Order of test units:
122
123      Tests from each unit (aka library) run consecutive.
124      Tests of different units are executed asynchronously.
125
126   Code differences when compiled with UNIT_TESTS=1
127
128      Some unit tests require differences in production code, e.g.
129      - to inspect class internals
130      - to provoke special conditions (e.g. reduce some buffer size to simulate big
131        amount of processed data)
132
133      Such differences should be marked with a comment containing 'UT_DIFF'.
134
135
1363. Valgrinding test code
137
138   If you set
139
140      VALGRIND=A
141
142   in UNIT_TESTER/Makefile.setup.local, valgrind will be started on the test-binary after the normal
143   unit tests passed with success.
144   Valgrind errors/warnings will not raise an error or abort testing.
145
146
147   If you set
148
149      VALGRIND=B
150
151   valgrind will be started BEFORE running tests.
152   Useful if test fail unexpectedly and you suspect a memory bug.
153
154   See also ENABLE_CRASH_TESTS in test_unit.h (all crash tests are reported by valgrind)
155
156
157   If you set
158
159      VALGRIND=E
160
161   some external tools like arb_pt_server will be spawned valgrinded and the generated
162   reports will be printed on test termination.
163
164   Any combination of the above works as well.
165
1664. Check test-coverage
167
168   set 'COVERAGE=1' in config.makefile and 'make rebuild'
169
170   - prints out uncovered code to compile log
171   - leaves info for partly covered code files in yourcode.cxx.cov
172
173   Note:
174        You can silence lines which are never reached while testing
175        with a comment containing 'NEED_NO_COV'.
176
177   call 'make clean_coverage_results' to get rid of .cov files
178
179   (see also showCoverageForAll in gcov2msg.pl)
180
1815. Define which tests shall run
182
183   [look into Makefile.setup.local for examples]
184
185   a. Restrict tests to one or several libraries:
186
187      Set 'RESTRICT_LIB=lib1[:libN]*' to run only tests defined in the
188      specified libraries.
189
190      Set 'RESTRICT_LIB=' to test all libraries.
191
192   b. Restrict tests to specific modules:
193
194      Set "RESTRICT_MODULE='regexpr'" to run only tests located in
195      matching modules. Use '.' to match any modules.
196      Prefix whole regexpr with '!' to match all but listed modules.
197
198   c. Restrict tests by name:
199
200      Set "RESTRICT_FUN='regexpr'" to run only tests whose function-name
201      matches the regexpr. Use '.' to match any test.
202      Prefix whole regexpr with '!' to match all but listed tests.
203
204   d. Restrict test frequency:
205
206      Set 'SKIP_SLOW=min' to restrict frequency of "slow tests". Slow tests
207      are all tests named 'TEST_SLOW_.*'. These tests will not run more often
208      than the specified amount of minutes.
209      Set 'SKIP_SLOW=0' to always run all tests (default).
210
211   e. Run unchanged tests:
212
213      By default only changed unit-tests will run. Set 'ONLY_CHANGED_TESTS=0'
214      to force testing unchanged code.
215
216      Tests will re-run if ..
217      - the test code was changed,
218      - any needed library was changed or if
219      - Makefile.setup.local or anything in the UnitTester itself was changed.
220
221      Calling 'make clean' in $ARBHOME will also force all tests to re-run.
222
223
2246. Global test environments
225
226   Environments provide a test situation which takes some reasonable time
227   for startup.
228
229   Multiple tests will be permitted to use one environment, but they will
230   be serialized (even if tests are running in multiple processes).
231
232   These environments have to be coded directly in TestEnvironment.cxx
233   (for existing environments see TestEnvironment.cxx@ExistingEnvironments)
234
235   If you write a test that wishes to use one of the environments simply write
236      TEST_SETUP_GLOBAL_ENVIRONMENT("name"); // starts environment 'name'
237
238
2397. Auto-patch generation
240
241   Whenever unit tests complete successfully, a patch (svn diff) is generated and
242   stored inside $ARBHOME/patches.arb/
243
244   After PATCHES_KEEP_HOURS seconds the patch will be deleted automatically
245   (see Makefile.setup.local), considering at least PATCHES_MIN_KEPT patches remain.
246
2478. Building tests
248
249   To build and execute a test-executable for a previously untested subdir
250   the following steps are necessary:
251
252   - add the test to the list of performed tests in ../Makefile@UNITS_TESTED
253   - if the test subject is a shared library, be sure to mention it a Makefile.test@SHAREDLIBS
254   - if the test subject is an executable, be sure to mention it a Makefile.test@EXEOBJDIRS
Note: See TracBrowser for help on using the repository browser.