source: tags/ms_r16q2/UNIT_TESTER/README.txt

Last change on this file was 12885, checked in by westram, 8 years ago
  • get rid of TEST_open_socket failures under jenkins
File size: 7.7 KB
Line 
1
2How to use ARB unit testing
3---------------------------
4
51. Invocation
6
7   Edit config.makefile. Set UNIT_TESTS to 1
8
9   cd $ARBHOME
10   make unit_tests
11
12   'make all' runs tests as well ('make build' doesn't)
13
142. Add unit test code
15
16   Go to the unit containing your_function() and add something like
17   the following to the end of the file.
18
19       #if (UNIT_TESTS == 1)
20
21       #include <test_unit.h>
22
23       void TEST_your_function() {
24       }
25
26       #endif
27
28   The names of the test functions are important:
29       - they have to start with 'TEST_' and
30       - need to be visible at global scope.
31
32   (see also 'Building tests' at end of file)
33
34
35   Test result:
36
37      If TEST_your_function() does nothing special, it is assumed the unit test passed "ok".
38
39      If your function fails an assertion (TEST_ASSERT) or raises SIGSEGV in any other manner,
40      it is assumed the unit test failed and an error message is printed to the console.
41
42      (Any other test function in the same unit will nevertheless get called.)
43
44
45   Post conditions:
46
47      If a unit defines one or more tests called
48         void TEST_POSTCOND_xxx()
49      each of these post condition tests will be run after EACH single unit test.
50
51      These post condition tests are intended to check for unwanted global side-effects of
52      test-functions. Such side effects often let other (unrelated) test-functions fail
53      unexpectedly. Use them to undo these side-effects.
54
55      If the post-condition fails, the test itself will be reported as failure.
56
57      If the test itself has failed for other reasons, post condition tests will nevertheless
58      be executed (and may do their cleanups), but their failures will not be shown in the log.
59
60
61   Failing tests:
62
63      If you have some broken behavior that you cannot fix now, please do NOT leave
64      the test as "failing". Instead change TEST_ASSERT into
65
66          TEST_ASSERT_BROKEN(failing_condition);
67
68      That will print a warning as reminder as long as the condition fails.
69      When the behavior was fixed (so that the condition is fulfilled now),
70      it will abort the test as failed!
71      Just change TEST_ASSERT_BROKEN back into TEST_ASSERT then.
72
73      Set WARN_LEVEL to 0 in Makefile.setup.local to disable warnings.
74
75      Random failures:
76
77             Some tests tend to fail randomly if started from inside jenkins (due to
78             parallel builds), whereas they behave proper and provide safety during
79             normal development. You should disable such tests for jenkins by enclosing
80             the whole test with:
81
82                #if !defined(DEVEL_JENKINS)
83                #endif
84
85             One example for such a test is ../CORE/arb_cs.cxx@TEST_open_socket
86
87
88   Missing tests:
89
90      You may drop a reminder like
91
92          MISSING_TEST(describe what is missing);
93
94      which will appear as warning during test run.
95
96
97   Order of tests:
98
99      Tests are executed in the order they appear in the code.
100      Multiple Files are sorted alphabethically.
101
102      Exceptions:
103      - If test name starts with 'TEST_###' (where ### are digits), it declares a test with priority ###.
104        The default priority is 100. Lower numbers mean higher priority, i.e. start more early.
105      - Other predefined priorities are
106        - TEST_BASIC_       (=20)
107        - TEST_EARLY_       (=50)
108        - TEST_             (=100 default)
109        - TEST_LATE_        (=200)
110        - TEST_SLOW_        (=900)
111        - TEST_AFTER_SLOW_  (=910)
112
113        TEST_SLOW_... is meant to indicate "slow" tests (i.e. tests which need more than
114        a second to execute).
115
116   Order of test units:
117
118      Tests from each unit (aka library) run consecutive.
119      Tests of different units are executed asynchronously.
120
121   Code differences when compiled with UNIT_TESTS=1
122
123      Some unit tests require differences in production code, e.g.
124      - to inspect class internals
125      - to provoke special conditions (e.g. reduce some buffer size to simulate big
126        amount of processed data)
127
128      Such differences should be marked with a comment containing 'UT_DIFF'.
129
130
1313. Valgrinding test code
132
133   If you set
134
135      VALGRIND=A
136
137   in UNIT_TESTER/Makefile.setup.local, valgrind will be started on the test-binary after the normal
138   unit tests passed with success.
139   Valgrind errors/warnings will not raise an error or abort testing.
140
141
142   If you set
143
144      VALGRIND=B
145
146   valgrind will be started BEFORE running tests.
147   Useful if test fail unexpectedly and you suspect a memory bug.
148
149   See also ENABLE_CRASH_TESTS in test_unit.h (all crash tests are reported by valgrind)
150
151
152   If you set
153
154      VALGRIND=E
155
156   some external tools like arb_pt_server will be spawned valgrinded and the generated
157   reports will be printed on test termination.
158
159   Any combination of the above works as well.
160
1614. Check test-coverage
162
163   set 'COVERAGE=1' in config.makefile and 'make rebuild'
164
165   - prints out uncovered code to compile log
166   - leaves info for partly covered code files in yourcode.cxx.cov
167
168   Note:
169        You can silence lines which are never reached while testing
170        with a comment containing 'NEED_NO_COV'.
171
172   call 'make clean_coverage_results' to get rid of .cov files
173
174   (see also showCoverageForAll in gcov2msg.pl)
175
1765. Define which tests shall run
177
178   [look into Makefile.setup.local for examples]
179
180   a. Restrict tests to one or several libraries:
181
182      Set 'RESTRICT_LIB=lib1[:libN]*' to run only tests defined in the
183      specified libraries.
184
185      Set 'RESTRICT_LIB=' to test all libraries.
186
187   b. Restrict tests to specific modules:
188
189      Set "RESTRICT_MODULE='regexpr'" to run only tests located in
190      matching modules. Use '.' for all modules.
191
192   c. Restrict tests by name:
193
194      Set "RESTRICT_FUN='regexpr'" to run only tests whose function-name
195      matches the regexpr. Use '.' for all tests.
196
197   d. Restrict test frequency:
198
199      Set 'SKIP_SLOW=min' to restrict frequency of "slow tests". Slow tests
200      are all tests named 'TEST_SLOW_.*'. These tests will not run more often
201      than the specified amount of minutes.
202      Set 'SKIP_SLOW=0' to always run all tests (default).
203
204   e. Run unchanged tests:
205
206      By default only changed unit-tests will run. Set 'ONLY_CHANGED_TESTS=0'
207      to force testing unchanged code.
208
209      Tests will re-run if ..
210      - the test code was changed,
211      - any needed library was changed or if
212      - Makefile.setup.local or anything in the UnitTester itself was changed.
213
214      Calling 'make clean' in $ARBHOME will also force all tests to re-run.
215
216
2176. Global test environments
218
219   Environments provide a test situation which takes some reasonable time
220   for startup.
221
222   Multiple tests will be permitted to use one environment, but they will
223   be serialized (even if tests are running in multiple processes).
224
225   These environments have to be coded directly in TestEnvironment.cxx
226   (for existing environments see TestEnvironment.cxx@ExistingEnvironments)
227
228   If you write a test that wishes to use one of the environments simply write
229      TEST_SETUP_GLOBAL_ENVIRONMENT("name"); // starts environment 'name'
230
231
2327. Auto-patch generation
233
234   Whenever unit tests complete successfully, a patch (svn diff) is generated and
235   stored inside $ARBHOME/patches.arb/
236
237   After PATCHES_KEEP_HOURS seconds the patch will be deleted automatically
238   (see Makefile.setup.local), considering at least PATCHES_MIN_KEPT patches remain.
239
2408. Building tests
241
242   To build and execute a test-executable for a previously untested subdir
243   the following steps are necessary:
244
245   - add the test to the list of performed tests in ../Makefile@UNITS_TESTED
246   - if the test subject is a shared library, be sure to mention it a Makefile.test@SHAREDLIBS
247   - if the test subject is an executable, be sure to mention it a Makefile.test@EXEOBJDIRS
Note: See TracBrowser for help on using the repository browser.