Provisioning handheld GPSs with Ansible

A friend of mine rents out hand held GPS units for out door activities. It started out as a small scale business, with just a few units per order. However, business is going well, and not only the number of orders grew, also the number of units per order grew an order of magnitude. Recently he was contacted by organizers of large charity sporting events, requesting rental of more than a hundred units, all fitted with custom GPX tracks and specific maps.

Initially, provisioning the GPS units was a manual task. This process involved connecting the GPS unit to his MacBook, waiting until the unit is booted, opening the device’s mass storage device, removing all traces of the previous user, copying the GPX track, profiles and tracks, and “ejecting” the device. This process took about 3 minutes per unit, making orders of 150 units was close to unfeasible. As these large events tend to be just a day or a weekend, the provisioning overhead became quite significant.

Automating

When we discussed the problem, the need for automation really became obvious. We discussed several options, but in the end we settled for Ansible. Both of us had some experience with Ansible, and I already had some significant experience with dynamic inventories.

I started out familiarizing myself with the /sys namespace. It isn’t very well documented, but trial and error will get you quite far. I started out by following the symlinks in /sys/class/block, which lists all storage devices. This list also contains ramdisks, loopback devices, your boot disk, and other non-USB devices, so it needs some filtering. With the remaining devices, I just hoarded all the information I could find. Each device is exported as an Ansible host using the local protocol. Most of the information is added as a hostvar. After that, I tried interpreting this information to distinguish between different types of GPS. The idProduct property turned out to be the most useful tool with this respect. The unit type is exported as a group, containing the applicable devices.

In the play book, the first step is to unmount and remount the device. All devices are mounted on a mount point in /tmp, but sometimes the device is mounted elsewhere, due to manual actions. Once the device is mounted, it is cleaned configured. These GPS units are configured through their file systems. Configuring them constitutes copying over firmware binaries, removing tracks and way points, and copying over maps and tracks. When all is done, the device is unmounted again.

The setup

For development and testing, we used my laptop, a 7-port USB hub and a few GPS units. For production, we chose a RaspberryPi, an EyeBoot 49-port USB hub and (of course) a whole lot of GPS units. The EyeBoot is vastly overpowered for this application, but we were able to get one for cheap. One of the 49 ports is used to power the raspberry, leaving 48 ports available for actual work.

Going through a full cycle, we managed to provision 48 GPS units in 15 minutes. This is an order of magnitude faster than the manual process, and best of all, most of it is unattended, allowing this process to scale with purchase of hardware.

Lessons learned

It turned out the Raspberry is really not powerful enough to efficiently run Ansible. Ansible spawns a lot of short lived python processes, which all have to go through VM startup. PyPy won’t help here, as Ansible executes each task as a separate process, and JIT compilers don’t perform well here. So either we’ll need to replace Ansible with something less span-heavy, or replace the CPU with a more powerful one.

The EyeBoot is overpowered for this job. We already were aware of that when we bought it, and is is unlikely we’ll buy a second one, unless we can find a good bargain.

Plans for the future

We plan to rig the raspberry to some switches and LEDs, so we can start the provisioning using a button, as the SSH-approach doesn’t work well with less tech savvy personnel.

Alternatively, we’d like the RaspberryPi to connect to a server, and send events concerning device attachment (udev), and start order-driven provisioning automatically, using some practical web-based interface. This allows the provisioning unit to be shipped to clients who want to update their rented units with new tracks and maps, during the rental.