While debugging issues involving binaries on a system running Linux, having a debugger such as GDB available is quite helpful.
However while working on a certain project we recently experienced quite some issues debugging applications involving threads.
Debugging the application on my local workstation worked quite fine, however on OpenWrt-targets – ARM as well as MIPS – it behaved rather strange: stack corruptions, missing traces, weird signals got issued…
After quite some time of debugging the debug issue, we found out the issue is caused by a stripped version of libpthread.so.
Stripped – not in the sense of a more lightweight but compatible version of the pthread library – but stripped by the utility “strip”, which purges all debug- and “other unneeded” symbols out of binaries to reduce their size, which usually is applied on all binaries by the OpenWrt framework automatically.
Usually binaries stripped by “strip” are still fully-fledged binaries, still usable with GDB (however without debugging symbols available of course). Applying strip on libpthread.so* however, it seems to strip out also stuff needed by GDB following and tracing threads. Without these symbols / meta-information not needed for running the actual application, but for tracking its threads, GDB results in mentioned issues above.
One might ask why someone is debugging binaries without debug symbols compiled in – reasons are obvious:
- there’s not enough storage available
- there’s not enough RAM available
- using gdbserver and having the binaries with debug symbols compiled in on another machine communicating with gdbserver
To check whether an object got stripped or not is quite easy using the “file” util:
$ file build_dir/target-arm_v5te_uClibc-0.9.30.1_eabi/root-foo/lib/libpthread-0.9.30.1.so
build_dir/target-arm_v5te_uClibc-0.9.30.1_eabi/root-foo/lib/libpthread-0.9.30.1.so: ELF 32-bit LSB shared object, ARM, version 1 (SYSV), dynamically linked (uses shared libs), stripped
$ file staging_dir/target-arm_v5te_uClibc-0.9.30.1_eabi/root-foo/lib/libpthread-0.9.30.1.so
staging_dir/target-arm_v5te_uClibc-0.9.30.1_eabi/root-foo/lib/libpthread-0.9.30.1.so: ELF 32-bit LSB shared object, ARM, version 1 (SYSV), dynamically linked (uses shared libs), not stripped
Long story short: When debugging applications involving threads, always use a non-stripped version of libpthread.so, even if debug symbols are not needed!