RSS Add a new post titled:

Wrapping libudev using LD_PRELOAD

Peter Hutterer and I were chasing down an X server bug which was exposed when running the libinput test suite against the X server with a separate thread for input. This was crashing deep inside libudev, which led us to suspect that libudev was getting run from multiple threads at the same time.

I figured I'd be able to tell by wrapping all of the libudev calls from the server and checking to make sure we weren't ever calling it from both threads at the same time. My first attempt was a simple set of cpp macros, but that failed when I discovered that libwacom was calling libgudev, which was calling libudev.

Instead of recompiling the world with my magic macros, I created a new library which exposes all of the (public) symbols in libudev. Each of these functions does a bit of checking and then simply calls down to the 'real' function.

Finding the real symbols

Here's the snippet which finds the real symbols:

static void *udev_symbol(const char *symbol)
    static void *libudev;
    static pthread_mutex_t  find_lock = PTHREAD_MUTEX_INITIALIZER;

    void *sym;
    if (!libudev) {
        libudev = dlopen("", RTLD_LOCAL | RTLD_NOW);
    sym = dlsym(libudev, symbol);
    return sym;

Yeah, the libudev version is hard-coded into the source; I didn't want to accidentally load the wrong one. This could probably be improved...

Checking for re-entrancy

As mentioned above, we suspected that the bug was caused when libudev got called from two threads at the same time. So, our checks are pretty simple; we just count the number of calls into any udev function (to handle udev calling itself). If there are other calls in process, we make sure the thread ID for those is the same as the current thread.

static void udev_enter(const char *func) {
    assert (udev_running == 0 || udev_thread == pthread_self());
    udev_thread = pthread_self();
    udev_func[udev_running] = func;

static void udev_exit(void) {
    if (udev_running == 0)
    udev_thread = 0;
    udev_func[udev_running] = 0;

Wrapping functions

Now, the ugly part -- libudev exposes 93 different functions, with a wide variety of parameters and return types. I constructed a hacky macro, calls for which could be constructed pretty easily from the prototypes found in libudev.h, and which would construct our stub function:

#define make_func(type, name, formals, actuals)         \
    type name formals {                     \
    type ret;                       \
    static void *f;                     \
    if (!f)                         \
        f = udev_symbol(__func__);              \
    udev_enter(__func__);                   \
    ret = ((typeof (&name)) f) actuals;         \
    udev_exit();                        \
    return ret;                     \

There are 93 invocations of this macro (or a variant for void functions) which look much like:

make_func(struct udev *,
      (struct udev *udev),

Using udevwrap

To use udevwrap, simply stick the filename of the .so in LD_PRELOAD and run your program normally:

# LD_PRELOAD=/usr/local/lib/ Xorg 

Source code

I stuck udevwrap in my git repository:;a=summary

You can clone it using

$ git git://
Posted Mon Aug 15 23:32:30 2016 Tags:

ChaosKey v1.0 Released — USB Attached True Random Number Generator

ChaosKey, our random number generator that attaches via USB, is now available for sale from the altusmetrum store.

We talked about this device at Debconf 16 last month

Support for this device is included in Linux starting with version 4.1. Plug ChaosKey into your system and the driver will automatically add entropy into the kernel pool, providing a constant supply of true random numbers to help keep the system secure.

ChaosKey is free hardware running free software, built with free software on a free operating system.

Posted Wed Aug 3 14:16:25 2016 Tags: ?tags/debian

AltOS 1.6.3 —

Bdale and I are pleased to announce the release of AltOS version 1.6.3.

AltOS is the core of the software for all of the Altus Metrum products. It consists of firmware for our cc1111, STM32L151, STMF042, LPC11U14 and ATtiny85 based electronics and Java-based ground station software.

Version 1.6.3 adds idle mode to AltosDroid and has bug fixes for our host software on desktops, laptops an android devices along with BlueTooth support for Windows.

1.6.3 is in Beta test for Android; if you want to use the beta version, join the AltosDroid beta program


AltOS fixes:

  • Fix hardware flow control on TeleBT v3.0. RTS/CTS is wired backwards on this board, switch from using the hardware to driving these pins with software.

AltosUI and TeleGPS Applications

AltosUI and TeleGPS New Features:

  • Add BlueTooth support for Windows operating system. This supports connections to TeleBT over BlueTooth rather than just USB.

AltosUI and TeleGPS Fixes:

  • Change Java detection and install on Windows. Detection is now done by looking for the 'javaw.exe' program, and installation by opening a browser on the web site.

  • Delay polling while the Fire Igniters is visible to allow for TeleMega to report back complete status over the radio.

  • Disallow changing RF calibration numbers in the configuration UI. There's no good reason to change this from the field, and recovering is really hard if you haven't written down the right number.

  • Fix USB device discovery on Mac OS X El Capitan. This makes the connected Altus Metrum USB devices appear again.

  • Fix acceleration data presented in MonitorIdle mode for TeleMetrum v2.0 flight computers.


AltosDroid new features:

  • Monitor Idle mode. Check state of flight computer while in idle mode over the radio link

  • Fire Igniters. Remotely fire ignires for recovery system ground tests.

  • Remote reboot. Cause the flight computer to reboot over the radio link. This provides a method for switching the flight computer from idle to flight mode without needing to reach the power switch.

  • Configurable frequency menu. Change the set of available frequencies and provide more descriptive names.

AltosDroid bug fixes:

  • Don't set target location if GPS hasn't locked yet.

  • Fix saving target states so they can be reloaded when the application restarts. When the application is shut down and restarted, all previous target state information will be restored (including GPS position if available).

  • Fix crash on some Android devices for offline maps when changing the map scale or location.

  • Don't require USB OTG support. This kept the latest AltosDroid from being offered on devices without USB device support, although it can work without that just fine using BlueTooth.

  • Don't require bluetooth to be enabled. This allows the application to operate with USB devices or just show old data without turning on the bluetooth radio.

  • Recover old tracker positions when restarting application. This finally allows you to safely stop and restart the application without losing the last known location of any tracker.


  • Document TeleMega and EasyMega additional pyro channel continuity audio alert pattern.
Posted Fri May 6 18:18:07 2016 Tags: Election Time — Vote Now

It's more important than usual to actually get your vote in — we're asking the membership to vote on changes the the bylaws that are necessary for to become a SPI affiliate project, instead of continuing on as a separate organization. While I'm in favor of this transition as I think it will provide much needed legal and financial help, the real reason we need everyone to vote is that we need ⅔ of the membership to cast ballots for the vote to be valid. Last time, we didn't reach that value, so even though we had a majority voting in favor of the change, it didn't take effect. If you aren't in favor of this change, I'd still encourage you to vote as I'd like to get a valid result, no matter the outcome.

Of course, we're also electing four members to the board. I'm happy to note that there are five candidates running for the four available seats, which shows that there are enough people willing to help serve the community in this fashion.

Posted Tue Apr 12 23:51:37 2016 Tags:

Automatic Calendar Management — Notmuch + Calypso

One of the big “features” of outlook/exchange in my world is automatically merging of incoming calendar updates from email. This makes my calendar actually useful in knowing what meetings people have asked me to attend. As I'm not willing to otherwise tolerate outlook, I decided to try and provide that in my preferred environment; notmuch and calypso.

Identifying calendar updates

The first trick is how to identify incoming messages with calendar updates. I'd love to be able to search for specific mime content types, but I haven't been able to construct such a search. Failing that, I'm just looking for messages containing the string 'text/calendar':

notmuch search --output=messages tag:inbox AND text/calendar

Next, I want to skip over previously scanned calendar updates, so I'll plan on tagging messages that have been scanned with the 'calendar' tag and skip those:

notmuch search --output=messages tag:inbox AND text/calendar AND not tag:calendar

jq — sed for json

With the messages containing likely calendar entries identified, the remaining task is to extract the MIME section containing the actual calendar data. Notmuch can generate json for the message, leaving us only needing to parse the json and extract the relevant section. I found the 'jq' tool in the archive, which looks like a rather complicated parsing and reformatting tool for json data. It took a while to understand, but I managed to generate a command string that takes a notmuch message and pulls out the content for all text/calendar elements:

jq -r '..| select(."content-type"? == "text/calendar") | .content'

This is a recursive walk over the data structure. It looks for structures with "content-type": "text/calendar" and dumps their "content" elements in raw text form.

Putting it all together

Here's the complete script:


SEARCH="tag:inbox AND not tag:calendar AND text/calendar"


trap "rm -r $TMP" 0 1 15

notmuch search --output=messages $SEARCH | while read message; do
    notmuch show --format=json $message | 
        jq -r '.. | select(."content-type"? == "text/calendar") | .content' > $TMP
    if [ -s $TMP ]; then
        calypso --import private/calendar $TMP && notmuch tag +calendar $message
        notmuch tag +calendar $message

I'd love to fix calypso's --import operation to talk to the local calypso daemon with the database loaded; the current mechanism actually loads the entire database into a new process and then applies the new data to that. With my calendar often containing hundreds of entries, that takes a while.

Posted Wed Jan 13 14:01:42 2016

AltOS 1.6.2 — TeleMega v2.0 support, bug fixes and documentation updates

Bdale and I are pleased to announce the release of AltOS version 1.6.2.

AltOS is the core of the software for all of the Altus Metrum products. It consists of firmware for our cc1111, STM32L151, STMF042, LPC11U14 and ATtiny85 based electronics and Java-based ground station software.

This is a minor release of AltOS, including support for our new TeleMega v2.0 board, a small selection of bug fixes and a major update of the documentation

AltOS Firmware — TeleMega v2.0 added

The updated six-channel flight computer, TeleMega v2.0, has a few changes from the v1.0 design:

  • CC1200 radio chip instead of the CC1120. Better receive performance for packet mode, same transmit performance.

  • Serial external connector replaced with four PWM channels for external servos.

  • Companion pins rewired to match EasyMega functionality.

None of these change the basic functionality of the device, but they do change the firmware a bit so there's a new package.

AltOS Bug Fixes

We also worked around a ground station limitation in the firmware:

  • Slow down telemetry packets so receivers can keep up. With TeleMega v2 offering a fast CPU and faster radio chip, it was overrunning our receivers so a small gap was introduced between packets.

AltosUI and TeleGPS applications

A few minor new features are in this release

  • Post-flight TeleMega and EasyMega orientation computations were off by a factor of two

  • Downloading eeprom data from flight hardware would bail if there was an error in a data record. Now it keeps going.


I spent a good number of hours completely reformatting and restructuring the Altus Metrum documentation.

  • I've changed the source format from raw docbook to asciidoc, which has made it much easier to edit and to use docbook features like links.

  • The css moves the table of contents out to a sidebar so you can navigate the html format easily.

  • There's a separate EasyMini manual now, constructed by taking sections from the larger manual.

Posted Sun Jan 10 21:03:08 2016 Tags:

TeleLaunchTwo — A Smaller Wireless Launch Controller

I've built a wireless launch control system for NAR and OROC. Those are both complex systems with a single controller capable of running hundreds of pads. And, it's also complicated to build, with each board hand-made by elves in our Portland facility (aka, my office).

A bunch of people have asked for something simpler, but using the same AES-secured two-way wireless communications link, so I decided to just build something and see if we couldn't eventually come up with something useful. I think if there's enough interest, I can get some boards built for reasonable money.

Here's a picture of the system; you can see the LCO end in a box behind the pad end sitting on the bench.

Radio Link

Each end has a 35mW 70cm digital transceiver (so, they run in the 440MHz amateur band). These run at 19200 baud with fancy forward error correction and AES security to keep the link from accidentally (or maliciously) firing a rocket at the wrong time. Using a bi-directional link, we also get igniter continuity and remote arming information at the LCO end.

The LCO Box

In the LCO box, there's a lipo battery to run the device, so it can be completely stand-alone. It has three switches and a button -- an arming switch for each of two channels, a power switch and a firing button. The lipo can be charged by opening up the box and plugging it into a USB port.

The Pad Box

The pad box will have some cable glands for the battery and each firing circuit. On top, it will have two switches, a power switch and an arming switch. The board has two high-power FETs to drive the igniters. That should be more reliable than using a relay, while also allowing the board to tolerate a wider range of voltages -- the pad box can run on anything from 12V to 24V.

The Box

Unlike the OROC and NAR systems, these boards are both designed to fit inside a specific box, the Hammond 1554E, and use the mounting standoffs provided. This box is rated at NEMA 4X, which means it's fairly weather proof. Of course, I have to cut holes in the box, but I found some NEMA 4X switches, will use cable glands for the pad box wiring and can use silicone around the BNC connector. The result should be pretty robust. I also found a pretty solid-seeming BNC connector, which hooks around the edge of the board and also clips on to the board.

Safety Features.

There's an arming switch on both ends of the link, and you can't fire a rocket without having both ends armed. That provides an extra measure of safety while working near the pad. The pad switch is a physical interlock between the power supply and the igniters, so even if the software is hacked or broken, disarming the box means the igniters won't fire.

The LCO box beeps constantly when either arming switch is selected, giving you feedback that the system is ready to fire. And you can see on any LED whether the pad box is also armed.

Posted Sun Dec 20 19:51:38 2015 Tags:

The Machine Architecture

Here's a brief introduction to some of the hardware concepts within The Machine.

As the team at HP that I'm working with are busy working on Linux kernel changes motivated by the hardware, I'm hoping that providing this kind of documentation will help Linux kernel developers outside of HP evaluate that work, and work by others in related areas.

Joining HP and learning about The Machine

In January, I joined HP to work on Linux for The Machine.

I'd watched Martin Fink's video and read other articles on the new hardware coming out of HP labs. I had hints of what they were up to, and the possibilities seemed exciting enough to entice me to go back to HP.

When I arrived at HP, one of the first things I got to read was the external reference specification for The Machine. 170 pages detailing a more significant shift in computer architecture than I had been given any hints of, both in my interviews at HP and from what I could see in the press.

Since then, I've been eager to tell people about what we're doing, and I'm happy to say that we're finally ready to start the conversation with this brief description.

A Short Outline of Storage within The Machine

The basic unit of The Machine is a collection of hardware grouped in a Load Store Domain. A Load Store Domain consists of:

  • Multiple independent Compute Nodes

    • Independent operating systems
    • Local memory
    • Load/store access to shared, in-memory storage
  • Shared byte-addressable persistent memory:

    • Non-volatile wrt operating system life cycle
    • Global address space
    • Hardware access control
    • Accessed with standard CPU load/store instructions

Here's a diagram of how the various bits of the hardware are hooked together:

And, a brief description of the elements within the picture:

  • Compute Node. A set of processing cores, caches and various ancillary peripherals.
  • Local Store. Memory directly connected to the processing cores.
  • Firewall. Hardware access control between the compute node and shared memory.
  • Shared Byte-addressable Persistent Memory. This is the storage within The Machine. It is accessed directly via normal CPU load/store instructions in units as small as one byte.

I've intentionally drawn the shared memory in a large box to emphasize the notion that this machine is more "memory-centric" and less "processor-centric".

The shared byte-addressable persistent memory forms the sole persistent storage within The Machine.

More to Come

I'll continue to publish information about The Machine and our related Linux work as we work on the hardware and software.

Posted Thu Apr 30 22:44:42 2015 Tags:

AltOS 1.6 — TeleDongle v3.0 support and bug fixes

Bdale and I are pleased to announce the release of AltOS version 1.6.

AltOS is the core of the software for all of the Altus Metrum products. It consists of firmware for our cc1111, STM32L151, STMF042, LPC11U14 and ATtiny85 based electronics and Java-based ground station software.

This is a major release of AltOS, including support for our new TeleDongle v3.0 board and a selection of bug fixes

AltOS Firmware — TeleDongle v3.0 added along with some fixes

Our updated ground station, TeleDongle v3.0, works just like the original TeleDongle, but is an all-new design:

  • CC1200 radio chip is about 5dB more sensitive than TeleDongle's CC1111.

  • LPC11U14 CPU can be reprogrammed over the USB link.

AltOS Bug Fixes

We also fixed a few bugs in the firmware:

  • Make sure the startup flight computer beeps are consistent. Sometimes, it was taking long enough to beep out the battery voltage that the flight computer state was changing in the middle, causing a bit of confusion.

  • Change TeleDongle's LED indicators during telemetry reception. The green LED blinks on successful packet reception, and the red LED blinks when a packet with an invalid checksum is received.

  • The SPI driver used in both TeleDongle v3 and TeleGPS has been rewritten to avoid locking up under heavy CPU load. If you've got a TeleGPS board, you'll want to reflash with new firmware.

AltosUI and TeleGPS applications

A few minor new features are in this release

  • AltosUI can now compute and display tilt angle when graphing eeprom log data from TeleMega and EasyMega.

  • The AltosUI tool window is shown when starting with a data file. This way, when you double-click on a file in the file manager, you'll get the whole AltosUI interface, rather than just the graphing window.

  • At the end of replaying an old log file, stick 'done' in the Age field so you can tell the recording is over.

Bug Fixes

There are a bunch of minor bug fixes, including the usual collection of attempts to make stuff behave better on Windows platforms.

  • Use a different Windows API to discover USB device ids. This works better on my new HP Windows 7 machine. Maybe it will work better for other people too?

  • Look in more places in the Windows registry to try and find the installed Java version. It appears that the default Java download from Oracle is a 32-bit version? In any case, that version sticks its install information in a different spot in the registry.

  • Fix file associations on Windows when Java isn't installed in the system root.

  • Make 'Scan Channels' work better with new AltOS firmware which only reports device configuration information once ever five seconds.

Posted Sat Feb 7 23:57:15 2015 Tags:

Re-joining HP

Thursday was my first day back with HP. I've joined Steve Geary's group to work on Linux support for “the machine”

I had a great time at Intel and wish my old team all the best.

Posted Fri Jan 9 18:17:25 2015 Tags: ?tags/debian

All Entries