Experience Report on Updating Keyboard Firmware Written in Rust

Posted on October 30, 2023 by Richard Goulter
Tags: , , ,

Here are some notes on the experiences I had updating some keyboard firmware I had written in the Rust programming language.

Context: My Dive into the Mechanical Keyboard Rabbit Hole

This was for keyboard firmware I’d written in Rust.

In mid-2020, I took my first steps into the mechanical keyboard hobby when I bought the BM40 keyboard. I thought it looked pretty neat, since the keys were arranged in a symmetrical grid, and the keyboard was very small (so would require some skill to use). It was a decent keyboard to start with.

One of the nice things about the mechanical keyboard hobby is there are all sorts ways you can be creative with fancy keyboards. (e.g. from purchasing a different set of keycaps so that your keyboard looks nicer, trying out different styles of keyboard switches, etc.).

By the start of 2021, I was trying to design keyboard PCBs. (You can find the results of my playing around at github, rgoulter/keyboard-labs). – The idea of trying to pack bells & whistles onto a keyboard all within a pair of 100x100mm PCBs sounded like such a fun/interesting challenge.

These PCBs would need firmware written for them. Fortunately, almost all of the custom keyboards that enthusiasts buy allow customising the keyboard layout. And, the most popular by far was QMK, which is relatively easy to work with. I had enough experience to work with QMK. (I was the one who added HSRGB functionality to the BM40 rev1, which took about 3 days to merge; I also made a PR for BM40 rev2, which took just over 550 days to get merged…). I was able to use QMK to write firmware for my custom designs.

But, I also came across the keyberon, a library which supported writing keyboard firmware in Rust. Of course, I had to try it.

Mechanical Keyboard Firmware Options

For writing keyboard firmware, there are several popular options. There are probably others, but to my mind:

My experience has been that QMK is by far the most practical one of these to use.

e.g. Although it’s much easier to iterate on firmware using CircuitPython (since “flash firmware” is as simple as “copy file to thumbdrive”), my experience with KMK was that its tap-hold functionality didn’t work well enough for home-row modifiers; which essentially makes the firmware impractical for small keyboards.

However. Keyberon is written in Rust, so I was curious to give it a try.

General Impressions of Rust Keyboard Firmware

It’s to QMK’s credit that it does such a great job at making writing keyboard firmware so accessible. As QMK is a framework, you just bolt on your own parts to it (e.g. a keyboard pinout, if the keyboard isn’t already supported by QMK; and a keymap). You to can make use of a wide variety of software features (like layering, or tap-dance key behaviour), as well as hardware features (such as RGB LEDs, OLED screens, etc.). – Essentially, almost no embedded system knowledge is required to write a keymap, and very little is required to add on a keyboard definition.

As I understand it, Keyberon’s goal isn’t to compete with QMK. Its goal is to be a library for helping write keyboard firmware. (I think limiting the project scope is a good thing).

I reckon a downside to this is: it ends up that the keyboard firmware written in Rust that I’ve seen all have the feeling of “copied and pasted” to them. – The potential upside to this is all the keyboard firmware programs are small and easy to understand; but I can’t help but feel that everyone doing this copying-and-pasting is having to pay the cost of a learning curve in each instance.

On Rust vs C

I really like that Rust has features like Sum types (tagged unions), and destructuring.

I got the impression that in order to write Rust code, you still need to have a decent mental model for how C works. – e.g. when I’m writing TypeScript or Python, I don’t need to care about whether the values I’m using are concrete or abstract; however, in Rust, I’d frequently make the compiler point out that with the code I had written, the size of the value couldn’t be determined. To me, that kind of restriction is less confusing if I can think “oh, right, in C, you’d have to …”.

In terms of “reading others’ code”: I would rather read code written in Rust than code written in C. – Frequently, C’s CPP #ifdefs make its code complex to read, since it’s that much more you need to keep in mind when reading the code. It’s also harder to setup tooling for C such that you can use development-environment features like “jump to definition”.

It’s really useful to be able to rely on code others have written. I wouldn’t want to set up a C project to depend on others code. (It’s more straightforward to add a dependency to a Rust project than to a C project). But, especially for this use case, building on QMK allows firmware which does more, with a wider variety of peripherals, than writing Rust code would. To an extent, I’d think the much of the code that gets into QMK also benefitted from there being more embedded firmware code in C than in Rust.

It’s difficult to write Rust code.
It can be difficult to correctly express lifetimes, or generic trait bounds. A strict compiler which adds friction is great for maintaining code, but it makes just shitting out code much harder. (And sometimes “shitting out code” is what you want to be doing).

In terms of “risk of writing code that’s too clever”: If someone’s writing code in C that’s too clever, one mechanism they have to use is the C Pre-Processor. Rust’s macros can be very similar; but Rust also has other ways of writing code that’s too clever.

Rust is also a complex language. I don’t want to point to any one thing and say “this is too hard to understand”; but with so many sophisticated features, it can be difficult to have a good intuition for how Rust’s features interact with each other. – So if you’re coming back to a codebase after some months of not having used Rust, it can take time to recall a good mental model.

That QMK is much more practical to use than the other keyboard firmware alternatives is a testament to just how much work has gone into making the project accessible.

Earlier Iterations of my Rust Firmware

My Rust keyboard firmware has been a project which I return to maintain once or twice a year. I initially wrote it, largely by copy-pasting from similar codebases. As a sideproject / hobby repo, each iteration was an opportunity for me to have fun, and make the code more sophisticated. e.g.:

I mean, each iteration was a chance to take something that worked, and add onto it a bit so the code was less messy. – “The first time, as a scientist; the second time, as an engineer”.

Updating the Code

I had tried (and failed) to update the code’s dependencies back in 2023 May.

My expectation when updating dependencies is: if I can bump the dependencies and there are no compilation errors, I expect the code to work as it did before.
If I run into compilation errors (e.g. function signatures or types changed), then I’ll have to look to see what changed, figure out the smallest reasonable change to get the code to compile, and hope that works.

I found difficulty in updating 3 things: the version of the realtime framework (RTIC), the version of the HAL (API for driving the hardware), and the toolchain version. – When I updated just that, then the code might compile but the firmware wouldn’t work as it did before.

When I updated the toolchain, the code compiled but didn’t run the same way it had before. – This was frustrating and unfortunate.
But, given that I was able to update the toolchain after resolving the other two problems, my best understanding is: my code was wrong, and happened to work with the old compiler version.

Similarly, when I tried updating the HAL dependency, the code would compile but wouldn’t result in a working keyboard; ditto for the RTIC.
This was frustrating and unfortunate.
But in this case, I think the problem was more clearly my bad code.
– I was using the wrong kind of Timer in the firmware. A closer read of the documentation indicated “if the system runs above such and such frequency, then don’t use this Timer”.
If there’s any fault of the dependencies, it’s apparently that they happened to previously be lenient enough to run bad code.

I’d put this frustrating experience up to the difficulty of embedded development.

That said, I will say I also ended up replacing the dependency of the USB HID keyboard from Keyberon’s implementation to usbd-human-interface-device’s.
For one, usbd-human-interface-device implemented support for media keys (play/pause, next track, etc.) which Keyberon didn’t.
But I’d ran into a bug related to sending USB HID reports when using Keyberon’s implementation, which went away when using usbd-human-interface-device’s.

Things I Found Difficult about Embedded Development

Deploying the Program is Harder

The easiest kind of deployment action is to run software on some Linux machine that you control. e.g. To deploy a web server, copy the program over, and then run the program there.

Running a program on an embedded device is somewhat more involved.

For a newbie, there are many things about this which are quite confusing.

Fortunately, it has become much easier in the time since I started. With UF2 bootloaders, flashing the firmware onto a device is as simple as copying a file to a thumbdrive. – Albeit, compiling the UF2 file is not totally trivial; and you still have to flash a UF2 bootloader onto the device.

Printf Debugging Harder to Set Up

A staple of debugging is “printf debugging”. You put a bunch of print statements in your code, and use those as evidence to lead you to a better understanding of how a system works.

With embedded devices (even very powerful ones), it’s … not quite that simple.

Ideally, with USB devices, you can somehow set up some kind of CDC serial console and output messages over that. But, if you’re at the level of copy-paste coding, this isn’t so simple to set up. (… and doesn’t help if you mess up something in the firmware before it gets to that point).

I saw probe-rs puts in a bunch of effort to make debugging embedded software much easier, to the extent where you can just run cargo embed and you can have this printf debugging. – But, you still have to figure out how to setup a probe device, and connect that to your device, etc.

The Small Details Might Really Matter

In one sense, programming is about structuring complexity so that you only need to pay attention to the things that are important to you (and can ignore the things that aren’t).

But I’d say with embedded development, it’s more likely that small details will be significant, compared to writing an HTTP API server.

One mistake I made was constructing a USB class after I’d constructed a USB device; this caused the program to crash, and it wasn’t obvious to me as to why. (Well, once I got the SWD connected and printf debugging working, then it was obvious what was happening).

Getting a Good Mental Model of Rust

I think it’s widely accepted that when learning Rust, you’ll be struggling to get code to compile. (Even code which would be acceptable in memory-managed languages like TypeScript or Python). Especially with concepts related to ownership and lifetimes.

More than once, I’d encounter difficulty trying to convince the compiler that my code was fine; or I might have some intuition that what I wanted to try was okay, but would have difficulty expressing that in how Rust actually worked. – I do believe many experienced Rust developers will try and avoid having to fight the compiler when they get into these situations.

Fancy Tech that Helped

LLMs

I used LLMs both for code-assistance in the IDE, as well as asking questions to a chat bot.

Code Suggestions in the Editor

Because the code for keyboard firmware so naturally reflected the structure of the keyboard’s schematic, the kind of code I was writing was very easy for GitHub’s CoPilot to predict. – e.g. you might leave a comment saying “the column pins are PA3, PB7, PB8, PA13, …”, and then GitHub CoPilot is able to predict what you’ll want let col_pins = to match against. This effectively removes a lot of drudge; rather than having to type 30-50 chars of what you know, you only have to type about 5 or so.

CoPilot didn’t get it right 100% of the time; but, with a nudge in the right direction, it was able to suggest exactly the code I wanted to write. – Even with having to check the output & adjust, overall, it was beneficial to use CoPilot.

CoPilot was also very useful for writing rote bits of code which I’d describe as “the kind of stuff you’d seach ‘how do I do this’ and copy-paste” (if you’re new to the language/library) or perhaps “the kind of thing you’d write without thought” (if you’re familiar with the language/library).
– Perhaps a downside being, if you’re not familiar with best practices, nor idioms of what the code should be, then maybe you’re just writing bad code.

(I also ran into some tooling problems: there were some contexts where CoPilot didn’t make suggestions, and some contexts where “use CoPilot’s completion” conflicted with other bindings).

Asking LLMs for Help

I liked the idea that Go-lang made for a good language to use for side-projects, since the language strives to be simple to write. This reduces the cost of coming back to the project after months of not having touched the technology; whereas with a more complicated language, you have more to recall.

Rust is a complex language. There are going to be things that won’t be obvious.

This is the kind of problem StackOverflow is good for. You’re likely to have the same questions as other people, and ask “how can I…?” or “why can’t I combine this and that…?”.

I tried asking ChatGPT for help whenever I was stuck with something.

The results were hit and miss.

When I got a good result, it would provide code snippets tailored closely to what I’d asked for.

When I got bad results, the LLM would confidently tell me “oh, this is how this works”, and it felt the Abbott and Costello joke about 7x13 = 28. – The joke is funny because I know basic multiplication; but I wonder what the joke is like if you have no understanding of the domain.

Ultimately, at the moment, I think this leaves only a few places where I’m comfortable asking LLMs stuff: domains where inaccuracy leaves me no-worse-off than the average person; and domains where I know enough to verify the LLM output to make use of it.

Nix

Probably rustup with a toolchain file and Cargo.lock are sufficient to reproduce the same binaries/behaviour that had worked before. I used Nix to take care of this, instead.

The additional benefits I got from Nix:

I’m able to trust that a Nix package isn’t affected by a dirty environment in a way that’s difficult to achieve with a Makefile.

Just (Task Runner)

One thing I don’t see eye to eye on with QMK is it has its own bespoke qmk CLI (which both depends on, and is intended for use within the qmk_firmware repository; you can’t use make without having qmk installed).

One DX improvement I value from the qmk cli is its qmk flash, which has a “do what I mean” feel to it, in that it waits for the expected device to become available, and then will flash the firmware appropriately.

While Makefiles are suitable for describing how files are to be built, justfiles are suitable for describing tasks to be run. So, I added a justfile with a convenience command to help me flash the firmware. It’s a small thing, but it improved the DX.

LSP

https://grugbrain.dev/#grug-on-type-systems

grug very like type systems make programming easier. for grug, type systems most value when grug hit dot on keyboard and list of things grug can do pop up magic. this 90% of value of type system or more to grug

LSP means it’s easier for “hit dot on keyboard and list of things grug can do pop up magic”.

Also useful is: being able to show the type, and being able to show the documentation for the type.

(Albeit, for some reason, the LSP in Helix was better able to find information in the RTIC framework’s tasks than the LSP I used with Emacs could. Perhaps this has to do with proc macros or some other thing.).

Newer post Older post