GDB behaves strange while debugging threads

While debugging issues involving binaries on a system running Linux, having a debugger such as GDB available is quite helpful.

However while working on a certain project we recently experienced quite some issues debugging applications involving threads.

Debugging the application on my local workstation worked quite fine, however on OpenWrt-targets – ARM as well as MIPS – it behaved rather strange: stack corruptions, missing traces, weird signals got issued…

After quite some time of debugging the debug issue, we found out the issue is caused by a stripped version of

Stripped – not in the sense of a more lightweight but compatible version of the pthread library – but stripped by the utility “strip”, which purges all debug- and “other unneeded” symbols out of binaries to reduce their size, which usually is applied on all binaries by the OpenWrt framework automatically.

Usually binaries stripped by “strip” are still fully-fledged binaries, still usable with GDB (however without debugging symbols available of course). Applying strip on* however, it seems to strip out also stuff needed by GDB following and tracing threads. Without these symbols / meta-information not needed for running the actual application, but for tracking its threads, GDB results in mentioned issues above.

One might ask why someone is debugging binaries without debug symbols compiled in – reasons are obvious:

To check whether an object got stripped or not is quite easy using the “file” util:

$ file build_dir/target-arm_v5te_uClibc-
build_dir/target-arm_v5te_uClibc- ELF 32-bit LSB shared object, ARM, version 1 (SYSV), dynamically linked (uses shared libs), stripped

$ file staging_dir/target-arm_v5te_uClibc-
staging_dir/target-arm_v5te_uClibc- ELF 32-bit LSB shared object, ARM, version 1 (SYSV), dynamically linked (uses shared libs), not stripped

Long story short: When debugging applications involving threads, always use a non-stripped version of, even if debug symbols are not needed!

Ben NanoNote able to control radio power sockets

The Ben NanoNote is now able to switch radio controlled power sockets, too!

The rfm12 433MHz module, produced by HopeRF,  is attached to the microSD port of the Ben NanoNote, which pins are exposed via an microSD card dummy adapter.

rfm12 module attached to a microSD dummy

The System-on-a-chip used inside the Ben NanoNote (ingenic JZ47XX) allows us to put the microSD pins into GPIO mode, so we can (as already done for the first device, the Netgear WGT634u router, I connected an rfm12-module to) create and export an SPI bus on top of them to be able communicating with the module.

That way there’s no need of opening the device and/or soldering anything anywhere – the module is attached directly to the microSD dummy which gets simply inserted into microSD slot of the NanoNote.

rfm12 module attached to NanoNote running ncurses UI for switching power sockets

I wrote a basic but fully working ncurses frontend using the rfm12 library listing configured devices waiting for getting switched.

There’s also a ready-to-use XMLRPC daemon available which exposes all configured devices and provides functions for controlling them, so UIs for controlling devices are not limited to run on the same system the rfm12 module is installed on.

ncurses UI for switching configured radio controlled power sockets on the Ben NanoNote

I’ve also written a little GUI in python using the qt4 toolkit, connecting to the master via XML-RPC:

switching radio controlled power via a GUI using pyqt4

Several other frontends are work-in-progress as e.g. a GUI for the Android platform, as well as one based on qt4/QML being able to run e.g. on phones running Meego/Maemo as operating system – both using the XML-RPC interface.

So all major parts of the project are mostly finished now and the API is more or less fix.

The the whole project now consists of (a rough overview):

  1. the kernel module which communicates with the actual hardware (so the rfm12 module) and exposes a character device to userspace
  2. the rfm12 library which
    • connects to the kernel module
    • contains the device type specific data and code to modulate signals which control the actual power sockets
    • provides functions for reading / writing configuration files and controlling / switching devices
  3. applications using the library and its functions for actual controlling of devices, which could be / already are:
    • UI applications linked directly against librfm12 (an ncurses frontend (shown above) is available yet)
    • daemons providing network interfaces (a daemon exporting functionality via XMLRPC is available yet , one doing the same but using JSON-RPC as underlying rpc method is going to be implemented soon)
  4. UIs running on different machines, using these RPC services  (e.g. above pyqt4  frontend; android- and qml-frontends are work-in-progress)

Devices are getting configured via configuration files, describing the product type of the device, a name and the actual code which is used to identify devices of a certain product type group.

The config used by the applications shown on the photos / screenshots above just looks this:

label = “one”
product = “P801B”
code = “1111110000”
label = “two”
product = “2272”
code = “1111110000”
label = “three”
product = “2272”
code = “1111101000”

One of the tasks I’m working on right now, is state sharing. While the list of configured devices and it’s states (on/off) is already shared among all XML-RPC clients (so having switched a unit in one client, others will fetch the changed state next time they poll/refresh), the state is not yet shared between several processes invoking the librfm12.

Issue is: Every process linked against librfm12 creates its own list of devices, including states – so every process has its own copy. Changes done in one process are not shared among others. This could be solved using IPC (System V shared memory, sockets every instance connects to, etc.).

That’s it – feedback and/or participation is highly appreciated!

Ben NanoNote having an rfm12 module attached via microSD-port and several types of radio controlled power sockets

Source Code is still available on github: 🙂