1 | |
---|
2 | How to use ARB unit testing |
---|
3 | --------------------------- |
---|
4 | |
---|
5 | 1. Invocation |
---|
6 | |
---|
7 | Edit config.makefile. Set UNIT_TESTS to 1 |
---|
8 | |
---|
9 | cd $ARBHOME |
---|
10 | make unit_tests |
---|
11 | |
---|
12 | 'make all' runs tests as well ('make build' doesn't) |
---|
13 | |
---|
14 | 2. Add unit test code |
---|
15 | |
---|
16 | Go to the unit containing your_function() and add something like |
---|
17 | the following to the end of the file. |
---|
18 | |
---|
19 | #if (UNIT_TESTS == 1) |
---|
20 | |
---|
21 | #include <test_unit.h> |
---|
22 | |
---|
23 | void TEST_your_function() { |
---|
24 | } |
---|
25 | |
---|
26 | #endif |
---|
27 | |
---|
28 | The names of the test functions are important: |
---|
29 | - they have to start with 'TEST_' and |
---|
30 | - need to be visible at global scope. |
---|
31 | |
---|
32 | (see also 'Building tests' at end of file) |
---|
33 | |
---|
34 | |
---|
35 | Test result: |
---|
36 | |
---|
37 | If TEST_your_function() does nothing special, it is assumed the unit test passed "ok". |
---|
38 | |
---|
39 | If your function fails an assertion (TEST_ASSERT) or raises SIGSEGV in any other manner, |
---|
40 | it is assumed the unit test failed and an error message is printed to the console. |
---|
41 | |
---|
42 | (Any other test function in the same unit will nevertheless get called.) |
---|
43 | |
---|
44 | |
---|
45 | Failing tests: |
---|
46 | |
---|
47 | If you have some broken behavior that you cannot fix now, please do NOT leave |
---|
48 | the test as "failing". Instead change TEST_ASSERT into |
---|
49 | |
---|
50 | TEST_ASSERT_BROKEN(failing_condition); |
---|
51 | |
---|
52 | That will print a warning as reminder as long as the condition fails. |
---|
53 | When the behavior was fixed (so that the condition is fulfilled now), |
---|
54 | it will abort the test as failed! |
---|
55 | Just change TEST_ASSERT_BROKEN back into TEST_ASSERT then. |
---|
56 | |
---|
57 | Set WARN_LEVEL to 0 in Makefile.setup.local to disable warnings. |
---|
58 | |
---|
59 | |
---|
60 | Missing tests: |
---|
61 | |
---|
62 | You may drop a reminder like |
---|
63 | |
---|
64 | MISSING_TEST(describe what is missing); |
---|
65 | |
---|
66 | which will appear as warning during test run. |
---|
67 | |
---|
68 | |
---|
69 | Order of tests: |
---|
70 | |
---|
71 | Tests are executed in the order they appear in the code. |
---|
72 | Multiple Files are sorted alphabethically. |
---|
73 | |
---|
74 | Exceptions: |
---|
75 | - If test name starts with 'TEST_###' (where ### are digits), it declares a test with priority ###. |
---|
76 | The default priority is 100. Lower numbers mean higher priority, i.e. start more early. |
---|
77 | - Other predefined priorities are |
---|
78 | - TEST_BASIC_ (=20) |
---|
79 | - TEST_EARLY_ (=50) |
---|
80 | - TEST_ (=100 default) |
---|
81 | - TEST_LATE_ (=200) |
---|
82 | - TEST_SLOW_ (=900) |
---|
83 | - TEST_AFTER_SLOW_ (=910) |
---|
84 | |
---|
85 | TEST_SLOW_... is meant to indicate "slow" tests (i.e. tests which need more than |
---|
86 | a second to execute). |
---|
87 | |
---|
88 | Order of test units: |
---|
89 | |
---|
90 | Tests from each unit (aka library) run consecutive. |
---|
91 | The order of the units is defined in ARB Makefile. |
---|
92 | |
---|
93 | |
---|
94 | 3. Valgrinding test code |
---|
95 | |
---|
96 | If you set |
---|
97 | |
---|
98 | VALGRIND=A |
---|
99 | |
---|
100 | in UNIT_TESTER/Makefile.setup.local, valgrind will be started on the test-binary after the normal |
---|
101 | unit tests passed with success. |
---|
102 | Valgrind errors/warnings will not raise an error or abort testing. |
---|
103 | |
---|
104 | |
---|
105 | If you set |
---|
106 | |
---|
107 | VALGRIND=B |
---|
108 | |
---|
109 | valgrind will be started BEFORE running tests. |
---|
110 | Useful if test fail unexpectedly and you suspect a memory bug. |
---|
111 | |
---|
112 | See also ENABLE_CRASH_TESTS in test_unit.h (all crash tests are reported by valgrind) |
---|
113 | |
---|
114 | |
---|
115 | If you set |
---|
116 | |
---|
117 | VALGRIND=E |
---|
118 | |
---|
119 | some external tools like arb_pt_server will be spawned valgrinded and the generated |
---|
120 | reports will be printed on test termination. |
---|
121 | |
---|
122 | Any combination of the above works as well. |
---|
123 | |
---|
124 | 4. Check test-coverage |
---|
125 | |
---|
126 | set 'COVERAGE=1' in config.makefile and 'make rebuild' |
---|
127 | |
---|
128 | - prints out uncovered code to compile log |
---|
129 | - leaves info for partly covered code files in yourcode.cxx.cov |
---|
130 | |
---|
131 | Note: |
---|
132 | You can silence lines which are never reached while testing |
---|
133 | with a comment containing 'NEED_NO_COV'. |
---|
134 | |
---|
135 | call 'make clean_coverage_results' to get rid of .cov files |
---|
136 | |
---|
137 | (see also showCoverageForAll in gcov2msg.pl) |
---|
138 | |
---|
139 | 5. Running only some tests |
---|
140 | |
---|
141 | Set 'RESTRICT=expr' to '.' to run all tests. |
---|
142 | |
---|
143 | Set it to sth else to run only tests for matching source files. |
---|
144 | This is helpful if you'd like to check test-coverage. |
---|
145 | |
---|
146 | Set 'RESTRICT_LIB=lib1:lib2' to test only a few libs. |
---|
147 | |
---|
148 | |
---|
149 | Slow tests ('TEST_SLOW_...'; see above) will only run every 15 minutes. |
---|
150 | |
---|
151 | |
---|
152 | 6. Global test environment |
---|
153 | |
---|
154 | @@@ Not in SVN yet |
---|
155 | |
---|
156 | |
---|
157 | 7. Auto-patch generation |
---|
158 | |
---|
159 | Whenever unit tests complete successfully, a patch (svn diff) is generated and |
---|
160 | stored inside $ARBHOME/patches.arb/ |
---|
161 | |
---|
162 | After PATCHES_KEEP_HOURS seconds the patch will be deleted automatically |
---|
163 | (see Makefile.setup.local), considering at least PATCHES_MIN_KEPT patches remain. |
---|
164 | |
---|
165 | 8. Building tests |
---|
166 | |
---|
167 | To build and execute a test-executable for a previously untested subdir |
---|
168 | the following steps are necessary: |
---|
169 | |
---|
170 | - add the test to the list of performed tests in ../Makefile@UNITS_TESTED |
---|
171 | - if the test subject is a shared library, be sure to mention it a Makefile.test@SHAREDLIBS |
---|
172 | - if the test subject is an executable, be sure to mention it a Makefile.test@EXEOBJDIRS |
---|