source: trunk/UNIT_TESTER/README.txt

Last change on this file was 17817, checked in by westram, 5 years ago
  • Unit tests:
    • Makefile.setup.template:
      • move config sections (rarely used to bottom).
      • fix trailing whitespace.
      • add fun restriction vs crashtests.
      • add xrefs.
      • force Conf mode.
    • rename 5 tests (which call crashtests) by adding suffix → allows to filter
    • count tests skipped by RESTRICT_FUN/RESTRICT_MODULE as "skipped" ⇒ list them in summary.
    • ignore directory UNIT_TESTER/tests.
File size: 8.4 KB
Line 
1
2How to use ARB unit testing
3---------------------------
4
51. Invocation
6
7   Edit config.makefile. Set UNIT_TESTS to 1
8
9   cd $ARBHOME
10   make unit_tests
11
12   'make all' runs tests as well ('make build' doesn't)
13
142. Add unit test code
15
16   Go to the unit containing your_function() and add something like
17   the following to the end of the file.
18
19       #if (UNIT_TESTS == 1)
20
21       #include <test_unit.h>
22
23       void TEST_your_function() {
24       }
25
26       #endif
27
28   The names of the test functions are important:
29       - they have to start with 'TEST_' and
30       - need to be visible at global scope.
31
32   (see also 'Building tests' at end of file)
33
34
35   Test result:
36
37      If TEST_your_function() does nothing special, it is assumed the unit test passed "ok".
38
39      If your function fails an assertion (TEST_ASSERT) or raises SIGSEGV in any other manner,
40      it is assumed the unit test failed and an error message is printed to the console.
41
42      (Any other test function in the same unit will nevertheless get called.)
43
44
45   Post conditions:
46
47      If a unit defines one or more tests called
48         void TEST_POSTCOND_xxx()
49      each of these post condition tests will be run after EACH single unit test.
50
51      These post condition tests are intended to check for unwanted global side-effects of
52      test-functions. Such side effects often let other (unrelated) test-functions fail
53      unexpectedly. Use them to undo these side-effects.
54
55      If the post-condition fails, the test itself will be reported as failure.
56
57      If the test itself has failed for other reasons, post condition tests will nevertheless
58      be executed (and may do their cleanups), but their failures will not be shown in the log.
59
60
61   Failing tests:
62
63      If you have some broken behavior that you cannot fix now, please do NOT leave
64      the test as "failing". Instead change TEST_ASSERT into
65
66          TEST_ASSERT_BROKEN(failing_condition);
67
68      That will print a warning as reminder as long as the condition fails.
69      When the behavior was fixed (so that the condition is fulfilled now),
70      it will abort the test as failed!
71      Just change TEST_ASSERT_BROKEN back into TEST_ASSERT then.
72
73      Set WARN_LEVEL to 0 in Makefile.setup.local to disable warnings.
74
75      Random failures:
76
77             Some tests tend to fail randomly if started from inside jenkins (due to
78             parallel builds), whereas they behave proper and provide safety during
79             normal development. You should disable such tests for jenkins by enclosing
80             the whole test with:
81
82                #if !defined(DEVEL_JENKINS)
83                #endif
84
85             One example for such a test is ../CORE/arb_cs.cxx@TEST_open_socket
86
87             Another possibility to handle random failures is to allow multiple tries.
88             This can be done using the macro TEST_ALLOW_RETRY(count). If called from
89             testcode this will allow the test to run 'count' times, failing only if
90             all 'count' calls of the test function result in failure.
91             Note: TEST_ALLOW_RETRY has no effect if a test already gets retried!
92
93   Missing tests:
94
95      You may drop a reminder like
96
97          MISSING_TEST(describe what is missing);
98
99      which will appear as warning during test run.
100
101
102   Order of tests:
103
104      Tests are executed in the order they appear in the code.
105      Multiple Files are sorted alphabethically.
106
107      Exceptions:
108      - If test name starts with 'TEST_###' (where ### are digits), it declares a test with priority ###.
109        The default priority is 100. Lower numbers mean higher priority, i.e. start more early.
110      - Other predefined priorities are
111        - TEST_BASIC_       (=20)
112        - TEST_EARLY_       (=50)
113        - TEST_             (=100 default)
114        - TEST_LATE_        (=200)
115        - TEST_SLOW_        (=900)
116        - TEST_AFTER_SLOW_  (=910)
117
118        TEST_SLOW_... is meant to indicate "slow" tests (i.e. tests which need more than
119        a second to execute).
120
121   Order of test units:
122
123      Tests from each unit (aka library) run consecutive.
124      Tests of different units are executed asynchronously.
125
126   Code differences when compiled with UNIT_TESTS=1
127
128      Some unit tests require differences in production code, e.g.
129      - to inspect class internals
130      - to provoke special conditions (e.g. reduce some buffer size to simulate big
131        amount of processed data)
132
133      Such differences should be marked with a comment containing 'UT_DIFF'.
134
135
1363. Valgrinding test code
137
138   Note: valgrind was not used for some time and might have rotten.
139         I've completely switched to using sanitizers -> see ../config.makefile
140
141
142   If you set
143
144      VALGRIND=A
145
146   in UNIT_TESTER/Makefile.setup.local, valgrind will be started on the test-binary after the normal
147   unit tests passed with success.
148   Valgrind errors/warnings will not raise an error or abort testing.
149
150
151   If you set
152
153      VALGRIND=B
154
155   valgrind will be started BEFORE running tests.
156   Useful if test fail unexpectedly and you suspect a memory bug.
157
158   See also ENABLE_CRASH_TESTS in test_unit.h (all crash tests are reported by valgrind)
159
160
161   If you set
162
163      VALGRIND=E
164
165   some external tools like arb_pt_server will be spawned valgrinded and the generated
166   reports will be printed on test termination.
167
168   Any combination of the above works as well.
169
1704. Check test-coverage
171
172   set 'COVERAGE=1' in config.makefile and 'make rebuild'
173
174   - prints out uncovered code to compile log
175   - leaves info for partly covered code files in yourcode.cxx.cov
176
177   Note:
178        You can silence lines which are never reached while testing
179        with a comment containing 'NEED_NO_COV'.
180
181   call 'make clean_coverage_results' to get rid of .cov files
182
183   (see also showCoverageForAll in gcov2msg.pl)
184
1855. Define which tests shall run
186
187   [look into Makefile.setup.local for examples]
188
189   a. Restrict tests to one or several libraries:
190
191      Set 'RESTRICT_LIB=lib1[:libN]*' to run only tests defined in the
192      specified libraries.
193
194      Set 'RESTRICT_LIB=' to test all libraries.
195
196   b. Restrict tests to specific modules:
197
198      Set "RESTRICT_MODULE='regexpr'" to run only tests located in
199      matching modules. Use '.' to match any modules.
200      Prefix whole regexpr with '!' to match all but listed modules.
201
202   c. Restrict tests by name:
203
204      Set "RESTRICT_FUN='regexpr'" to run only tests whose function-name
205      matches the regexpr. Use '.' to match any test.
206      Prefix whole regexpr with '!' to match all but listed tests.
207
208   d. Restrict test frequency:
209
210      Set 'SKIP_SLOW=min' to restrict frequency of "slow tests". Slow tests
211      are all tests named 'TEST_SLOW_.*'. These tests will not run more often
212      than the specified amount of minutes.
213      Set 'SKIP_SLOW=0' to always run all tests (default).
214
215   e. Run unchanged tests:
216
217      By default only changed unit-tests will run. Set 'ONLY_CHANGED_TESTS=0'
218      to force testing unchanged code.
219
220      Tests will re-run if ..
221      - the test code was changed,
222      - any needed library was changed or if
223      - Makefile.setup.local or anything in the UnitTester itself was changed.
224
225      Calling 'make clean' in $ARBHOME will also force all tests to re-run.
226
227
2286. Global test environments
229
230   Environments provide a test situation which takes some reasonable time
231   for startup.
232
233   Multiple tests will be permitted to use one environment, but they will
234   be serialized (even if tests are running in multiple processes).
235
236   These environments have to be coded directly in TestEnvironment.cxx
237   (for existing environments see TestEnvironment.cxx@ExistingEnvironments)
238
239   If you write a test that wishes to use one of the environments simply write
240      TEST_SETUP_GLOBAL_ENVIRONMENT("name"); // starts environment 'name'
241
242
2437. Auto-patch generation
244
245   Whenever unit tests complete successfully, a patch (svn diff) is generated and
246   stored inside $ARBHOME/patches.arb/
247
248   After PATCHES_KEEP_HOURS seconds the patch will be deleted automatically
249   (see Makefile.setup.local), considering at least PATCHES_MIN_KEPT patches remain.
250
2518. Building tests
252
253   To build and execute a test-executable for a previously untested subdir
254   the following steps are necessary:
255
256   - add the test to the list of performed tests in ../Makefile@UNITS_TESTED
257   - if the test subject is a shared library, be sure to mention it a Makefile.test@SHAREDLIBS
258   - if the test subject is an executable, be sure to mention it a Makefile.test@EXEOBJDIRS
Note: See TracBrowser for help on using the repository browser.