Use AddressSanitizer to memory debug arb?

[westram]

I've tried to use ​AddressSanitizer (included in gcc4.8 and higher) to memory-debug ARB and encountered several problems:

• ARB perl module no longer works (tests disabled in patch)
• all tests fail (terminated by AddressSanitizer)
• starting arb demo.arb causes a dump (illegal read - see below), but debugging that with gdb reveals "impossible" behavior (see stacktrace below)

Something seems to be completely wrong. I guess this could really be a nice tool for tracking down problems (if it only would work).

[westram]

tested vs [12476] with gcc 4.81

[westram]

use to fix #613

[westram]

[moved from description]

stacktrace:

Breakpoint 1, __asan_report_error (pc=140737348135967, bp=140737488344592, sp=140737488344584, addr=140737348601727, is_write=false, access_size=1) at ../../../../gcc-4.8.1/libsanitizer/asan/asan_report.cc:628
(gdb) whe 4
Python Exception <type 'exceptions.ImportError'> No module named gdb.frames: 
#0  __asan_report_error (pc=140737348135967, bp=140737488344592, sp=140737488344584, addr=140737348601727, is_write=false, access_size=1) at ../../../../gcc-4.8.1/libsanitizer/asan/asan_report.cc:628
#1  0x00007ffff2d68794 in __asan::__asan_report_load1 (addr=<optimized out>) at ../../../../gcc-4.8.1/libsanitizer/asan/asan_rtl.cc:226
#2  0x00007ffff7a46c1f in GB_concat_path (anypath_left=0x0, anypath_right=0x7ffff7ab8780 "") at adsocket.cxx:1175
#3  0x00007ffff7a4710e in GB_path_in_ARBLIB (relative_path_left=0x7ffff7ab8780 "", anypath_right=0x7ffff7ab8740 "arb_tcp.dat") at adsocket.cxx:1256
(More stack frames follow...)

gdb shows anypath_left=0x0, but execution claims to be ​here which is impossible if anypath_left is zero.

AddressSanitizer dump:

> arb demo.arb
Using ARBHOME='/home/ralf/ARB-bilbo/ARB.trunk.481'
Using properties from /home/ralf/.arb_prop
Please wait while the program ARB is starting .....
Waiting for '/home/ralf/.arb_tmp/sockets/arb_launcher.29445'..
[arb_launcher[0]: Starting 'arb_ntree demo.arb'..]
Using properties from '/home/ralf/.arb_prop/ntree.arb'
Using properties from '/home/ralf/ARB-bilbo/ARB.trunk.481/lib/arb_default/status.arb'
=================================================================
==29521== ERROR: AddressSanitizer: global-buffer-overflow on address 0x7ffd54d5277f at pc 0x7ffd54ce0c1f bp 0x7fffe4871900 sp 0x7fffe48718f8
READ of size 1 at 0x7ffd54d5277f thread T0
    #0 0x7ffd54ce0c1e (/home/ralf/ARB-bilbo/ARB.trunk.481/lib/libARBDB.so+0x1e3c1e)
    #1 0x7ffd54ce110d (/home/ralf/ARB-bilbo/ARB.trunk.481/lib/libARBDB.so+0x1e410d)
    #2 0x7ffd54c2ab4f (/home/ralf/ARB-bilbo/ARB.trunk.481/lib/libARBDB.so+0x12db4f)
    #3 0x7ffd54c31f0c (/home/ralf/ARB-bilbo/ARB.trunk.481/lib/libARBDB.so+0x134f0c)
    #4 0x7ffd54c31fb0 (/home/ralf/ARB-bilbo/ARB.trunk.481/lib/libARBDB.so+0x134fb0)
    #5 0x7ffd54c328c7 (/home/ralf/ARB-bilbo/ARB.trunk.481/lib/libARBDB.so+0x1358c7)
    #6 0x7ffd54cdc09f (/home/ralf/ARB-bilbo/ARB.trunk.481/lib/libARBDB.so+0x1df09f)
    #7 0x7ffd54cdc398 (/home/ralf/ARB-bilbo/ARB.trunk.481/lib/libARBDB.so+0x1df398)
    #8 0x7ffd54cdcfb7 (/home/ralf/ARB-bilbo/ARB.trunk.481/lib/libARBDB.so+0x1dffb7)
    #9 0x7ffd54bd5afc (/home/ralf/ARB-bilbo/ARB.trunk.481/lib/libARBDB.so+0xd8afc)
    #10 0x7eb820 (/home/ralf/ARB-bilbo/ARB.trunk.481/bin/arb_ntree+0x7eb820)
    #11 0x7f320a (/home/ralf/ARB-bilbo/ARB.trunk.481/bin/arb_ntree+0x7f320a)
    #12 0x556a28 (/home/ralf/ARB-bilbo/ARB.trunk.481/bin/arb_ntree+0x556a28)
    #13 0x556ef2 (/home/ralf/ARB-bilbo/ARB.trunk.481/bin/arb_ntree+0x556ef2)
    #14 0x559b1a (/home/ralf/ARB-bilbo/ARB.trunk.481/bin/arb_ntree+0x559b1a)
    #15 0x55a1e9 (/home/ralf/ARB-bilbo/ARB.trunk.481/bin/arb_ntree+0x55a1e9)
    #16 0x529dfa (/home/ralf/ARB-bilbo/ARB.trunk.481/bin/arb_ntree+0x529dfa)
    #17 0x7ffd4f4e5c8c (/lib/libc-2.11.1.so+0x1ec8c)
    #18 0x529c78 (/home/ralf/ARB-bilbo/ARB.trunk.481/bin/arb_ntree+0x529c78)
0x7ffd54d5277f is located 1 bytes to the left of global variable '*.LC19 (adtcp.cxx)' (0x7ffd54d52780) of size 1
  '*.LC19 (adtcp.cxx)' is ascii string ''
0x7ffd54d5277f is located 51 bytes to the right of global variable '*.LC18 (adtcp.cxx)' (0x7ffd54d52740) of size 12
  '*.LC18 (adtcp.cxx)' is ascii string 'arb_tcp.dat'
Shadow bytes around the buggy address:
  0x10002a9a2490: 00 00 00 04 f9 f9 f9 f9 00 02 f9 f9 f9 f9 f9 f9
  0x10002a9a24a0: 00 04 f9 f9 f9 f9 f9 f9 00 00 00 00 00 00 00 00
  0x10002a9a24b0: 03 f9 f9 f9 f9 f9 f9 f9 00 00 f9 f9 f9 f9 f9 f9
  0x10002a9a24c0: 00 06 f9 f9 f9 f9 f9 f9 00 00 f9 f9 f9 f9 f9 f9
  0x10002a9a24d0: 04 f9 f9 f9 f9 f9 f9 f9 00 00 04 f9 f9 f9 f9 f9
=>0x10002a9a24e0: 00 00 00 f9 f9 f9 f9 f9 00 04 f9 f9 f9 f9 f9[f9]
  0x10002a9a24f0:01 f9 f9 f9 f9 f9 f9 f9 00 00 00 00 00 00 00 00
  0x10002a9a2500: 00 00 00 00 00 00 04 f9 f9 f9 f9 f9 00 00 00 00
  0x10002a9a2510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002a9a2520: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002a9a2530: 00 00 00 00 00 05 f9 f9 f9 f9 f9 f9 00 00 f9 f9
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:     fa
  Heap righ redzone:     fb
  Freed Heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe
==29521== ABORTING
[arb_launcher[0]: 'arb_ntree demo.arb' has terminated with error 1]
[arb_launcher[0]: Still have 1 arb processes..]
[arb_launcher[0]: All launched processes terminated]

Session log has been stored in /home/ralf/.arb_prop/logs/session.20140619_180633.29455.tgz
    and is also accessible via /home/ralf/ARB_last_session.tgz

ARB terminated abnormally
[press ENTER]

[westram]

Retried with gcc 4.9.0:

• ARB perl module no longer works (tests disabled in patch)
• starting arb demo.arb causes a dump (illegal read - see below), but debugging that with gdb reveals "impossible" behavior (see stacktrace below)

• still same

• all tests fail (terminated by AddressSanitizer)

• several tests succeed
• failing (terminated by AS):
  • libARBDB (no summary; crashed?)
  • arb_test (no summary; crashed?)
  • TREEDISP (no summary; crashed?)
  • PRONUC (no summary; crashed?)
  • CONVERTALN (no summary; crashed?)
• deadlocked (looks like ptserver test environment fails to startup)
  • AWTC (interrupted; deadlock?)
  • MULTI_PROBE (interrupted; deadlock?)
  • arb_probe (interrupted; deadlock?)

[westram]

first "working" version with [12998]

[westram]

Replying to westram:

• failing [tests] (terminated by AS):
  • libARBDB (no summary; crashed?)

fixed by [13002:13003]

• arb_test (no summary; crashed?)

fixed by

• [13002:13003] ( startup)
• [13012:13014] (TEST_SLOW_arb_dna_rates)
• [13015] (clean environment)

• TREEDISP (no summary; crashed?)

fixed by [13016]

• PRONUC (no summary; crashed?)

fixed by [13005]

• CONVERTALN (no summary; crashed?)

fixed by [13001]

• deadlocked (looks like ptserver test environment fails to startup)
  • AWTC (interrupted; deadlock?)
  • MULTI_PROBE (interrupted; deadlock?)
  • arb_probe (interrupted; deadlock?)

fixed by some of the mentioned patches

[westram]

tested ​InitializationOrderFiasco with [13075]:

Index: AWTC/AWTC_next_neighbours.cxx
===================================================================
--- AWTC/AWTC_next_neighbours.cxx	(revision 13075)
+++ AWTC/AWTC_next_neighbours.cxx	(working copy)
@@ -20,6 +20,21 @@
 
 #include <climits>
 
+extern int extern_global;
+int __attribute__((noinline)) read_extern_global() {
+    return extern_global;
+}
+
+int glob_x = read_extern_global() + 1;
+
+struct staticClass {
+    staticClass() {
+        printf("%d\n", glob_x);
+    }
+};
+
+static staticClass sc;
+
 struct PT_FF_comImpl {
     aisc_com  *link;
     T_PT_MAIN  com;
Index: AWTC/AWTC_submission.cxx
===================================================================
--- AWTC/AWTC_submission.cxx	(revision 13075)
+++ AWTC/AWTC_submission.cxx	(working copy)
@@ -18,6 +18,10 @@
 #include <arb_strbuf.h>
 #include <arb_strarray.h>
 
+extern int glob_x;
+int foo() { return glob_x; }
+int extern_global = foo();
+
 #define awtc_assert(bed) arb_assert(bed)
 
 #define AWAR_SUBMIT_PARSER "tmp/submission/parser"

reports:

==14386==ERROR: AddressSanitizer: initialization-order-fiasco on address 0x0000006f64a0 at pc 0x438332 bp 0x7fffdd18e850 sp 0x7fffdd18e848
READ of size 4 at 0x0000006f64a0 thread T0
AWTC/AWTC_submission.cxx:22: #0 0x438331 in foo()
AWTC/AWTC_submission.cxx:23: #1 0x438364 in __static_initialization_and_destruction_0
AWTC/AWTC_submission.cxx:307: #2 0x446c9e in _GLOBAL__sub_I_AWTC_submission.cxx
    #3 0x467375 (/home/ralf/ARB-bilbo/ARB.sanitize.491.DEBUG/UNIT_TESTER/tests.slow/test_AWTC_AWTC_a+0x467375)

0x0000006f64a0 is located 0 bytes inside of global variable 'glob_x' from 'AWTC_next_neighbours.cxx' (0x6f64a0) of size 4
SUMMARY: AddressSanitizer: initialization-order-fiasco /home/ralf/ARB-bilbo/ARB.sanitize.491.DEBUG/AWTC/AWTC_submission.cxx:22 foo()
...

[westram]

mark changes that got fixed after arb 6.0.x

[westram]

Milestone renamed