Experience with the Tweag Configuration Language as an End User

I recently had the chance at writing some keyboard firmware with semickolon’s fak. (e.g. keyboard definition, keymap definition).

Fak uses tweag’s Nickel for declarative definitions of keyboards and keymaps. – Nickel is sophisticated type-safe configuration language.

I’d first heard of Tweag in discussions around the Nix package manager, and how people found the Nix expression language’s lack of types to add friction to writing packages. I’d put Nickel in the same family as dhall, or cue, or perhaps jsonnet. (Although of those, I have minor experience with jsonnet, and briefly tried Dhall. I don’t have enough experience to do a good comparison). – In a sense, these could all be described as “JSON plus functions plus modules (plus types)”.

Here are some notes on that.

Context: What Kind of Configuration Does a Keyboard Use

Fancy keyboards allow the user to customize the keyboard’s behaviour.

One example is for users can change the keyboard’s keymap to some alternative layout (such as Dvorak, or Colemak) on the keyboard itself; this reduces friction, since then the computer can assume it’s using a typical QWERTY keyboard, and the user doesn’t have to worry about changing the OS configuration to use a keymap that might not be installed on the OS.

Anyway, the point is, in order to be able to change the keyboard’s behaviour, you have to some way of the user describing that behaviour.

With QMK firmware, the approach previously had been that you’d write out your keyboard definition and keymap definition in the C programming language, and compile the firmware and flash this onto the keyboard.

C is a simple language; but with the C PreProcessor, it’s also unrestrained and formless.
The formless nature can be a benefit for the creative expression of the end user (in terms of how the keymap is expressed),
but I guess it’s a significant burden for the framework maintainers, presumably since it makes changing interfaces of the framework difficult.

Hence, there’s been a move towards data-driven configuration.

QMK’s info.json adds a layer of abstraction so that the relevant information can be declared from a single source of truth; this data can then be used to generate the code which QMK had been using before. The keyboard definitions describe the keyboard’s pinout, and what features it supports / how it supports them; and the keymap definitions describe the layout of keys. (e.g. my info.json for PyKey40, or for my “X-2” keyboard).

And, as I described in my previous post that I’d rather read Rust than read C; it’s easier to read someone else’s JSON than to read someone else’s C.

So. Custom keyboard firmware makes use of configurations for things like keyboard definitions and keymap definitions.

Nickel

Roughly, my feeling is that Nickel is to JSON and YAML what Rust is to C and C++.

C is ubiquitous, and bare bones. C++ has some features which make it seem nicer for some use cases, but it’s got many rough edges. Rust is ‘complex’, but avoids many of the previous rough edges, and has some really nice features.

JSON is ubiquitous, and bare bones. etc.

Technically, these “JSON plus functions plus modules” languages operate at a different level than JSON.. but I think that since the goal is to declare a particular configuration, I’d put them alongside each other when writing them.

Cosmetically, much of Nickel’s syntax does seem similar to Rust. (Whereas, say, Dhall’s syntax seems similar to Haskell).

Since I was using Nickel to write configuration (as opposed to writing the application which uses the configuration, or library modules for the configuration), I didn’t poke around with Nickel’s typing or contracts.

“JSON plus Functions” Allows for Clearer Configuration

With JSON, there’s no choice but to be explicit, unfactored, and undocumented.

Consider the value each of the additions to JSON in “JSON plus functions plus modules (plus types)”.

Types can be useful for avoiding ‘illegal’ configurations, or otherwise navigating code, even though technically types provide no value at runtime.

“Plus functions” implies “plus variables”. – I recall one way of describing programming was: “means of naming, means of combining, and means of abstraction”. That’s really the functionality “JSON plus functions” is getting at.

“Plus modules” on top of that is (more/less) simply being able to break up one big file into many smaller ones.

Without a means of naming/combining/abstracting, JSON values must be explicit and can’t be factored. “list of numbers from 4 to 30” would be tedious to describe in JSON, but may be useful to describe in USB keyboard configurations (e.g. “values A-Z in the keyboard usage page”).

And for anything non-trivial, there’s always going to be a benefit in being able to add documentation.

Configuration Languages Hamstrung by JSON

These “JSON plus functions” languages do allow for a nicer experience, but I think they’re also inherently limited by JSON’s success.

I like sum types, pattern matching and value destructuring. (And apparently alongside pattern matching, I like tuples).

But since JSON values are either primitive (null, boolean, number, string) or an array or an dictionary… there’s not really a natural way to express sum types (or tuples) as JSON values. – Or languages like Go-lang also lack sum types; so while you can parse JSON into Go values without too much difficulty, there’s more friction trying to do the same with sum types.

I guess with sum types, you need to know all the variations of the type before you can do meaningfully useful things with the value (check for invalid variants, exhaustively check variants have been considered, etc.); the other JSON values don’t have this constraint. Still, it’d be nice to somehow have sum types in more places.

It Lacks Tuples

I’m not sure if it was a deliberate design decision motivated by a preference for the explicit, or a distaste of implementing a tuple using heterogeneous values in an array (or perhaps a limitation that lists can’t be heterogeneous?), but I noticed that Nickel doesn’t support tuple types (in contrast to its other fancy features).

e.g. the signature for Nickel’s std.array.partition is:

partition : forall a. (a -> Bool) -> Array a -> { right : Array a, wrong : Array a }

whereas e.g. Haskell’s Data.List.partition uses tuples:

partition :: (a -> Bool) -> [a] -> ([a], [a]) 

Similarly, Rust’s Iterator’s partition returns a tuple, JavaScript’s lodash returns an array, elm’s List.partition returns a tuple.

I don’t think the other “JSON plus functions” languages have tuples, either; so it’s not Nickel’s so much as JSON’s limitation.

Smelly Syntax: Lists vs Arrays and Recursive vs Iterative

In my use case, use of nickel was done at compile time on a powerful machine, so I’m not concerned with memory consumption, nor too concerned about runtime.

My taste was tickled by that Nickel’s std.array has functions which are geared towards list manipulation (e.g. fold_left), whereas the structure is an array.

I found this at times awkward since AFAICT in Nickel, arrays don’t have syntax to destructure them.

Rather, it’s natural to interact with lists recursively. A list is recursively defined as either an empty list, or some item followed by a list. (e.g. the (list 1 2 3) is 1 followed by (list 2 3); and ultimately (list 3) is 3 followed by the empty list). In Haskell, a list can be pattern-matched against x:xs (the head of the list, followed by its tail).

Whereas, it’s natural to interact with arrays using loops and indices.

It’s natural for an expression-oriented language like Nickel to work with lists (and it would be unusual for an expression-based language to have loop constructs, I think); and JSON has arrays.

Configuration Complexity is About How You Use It

A blogpost I think about a lot is the “Configuration Complexity Clock”. Its picture tells 1,000 words; but roughly the point is that at each step, the trade-offs look desirable for going from hard-coded values to a configuration; and from a configuration to a DSL; and from a DSL to hard-coded values. – Hence, don’t just accept “hard coded values bad” as a dogma, and take care when adopting a configuration or a DSL.

What’s “too hard to read” is also dependent on the person.. Steve Yegge’s “Portrait of a n00b” discusses that what’s considered acceptable to someone unfamiliar with the code may be unacceptable to someone who is very familiar with the code. (Code that’s too sparse or too verbose means changing the code takes more effort than it needs to; whereas code that’s too dense or too terse also means changing the code takes more effort than it needs to).

Adding functions to JSON provides a more powerful tool for configuration. But, sometimes a “more powerful tool” is like a rocket-powered chainsaw: if you’re not careful about how you’re using it, you might end up causing more damage than it’s worth.

Types Aren’t Sufficient to Describe Interfaces

Something John Ousterhout’s A Philosophy of Software Design talks about is about the distinction between a software module’s interface and its implementation. Its interface is “what someone needs to know to use the module”. – He gives the example of a bad interface: on Windows, you can’t delete a file if it’s open by some process; this is ‘bad’ because in order to delete a file, you need to also know that the file is not opened by any process (or which processes to close!).

The broader point is that a module is “complex” if its interface requires superfluous details (the interface is over-specified), or if using the interface also requires knowing implementation details not exposed as part of the interface (the interface is under-specified).

PoSD makes the point that, similar to how Test-Driven Development emphasises the benefits of writing unit tests before writing the implementation, it’s similarly beneficial to write out a module’s documentation before writing the implementation.

While types show what an implementation is capable of, documentation describes what a module user needs to know in order to use the module (especially since it’s often not practical or possible to describe with types).

To illustrate the point, looking at a rather gnarly Nickel excerpt in fak:

let HoldTapKeyInterrupt = {
  decision | [| 'none, 'hold, 'tap |] | default = 'none,
  trigger_on | [| 'press, 'release |] | default = 'press
} in

let HoldTapKeyInterrupts = (
  # ...
  std.contract.Sequence [ Array HoldTapKeyInterrupt, ValidLength ]
) in

let HoldTapBehavior = let
  default_key_interrupts = std.array.replicate key_count {}
in {
  timeout_decision | [| 'hold, 'tap |] | default = 'hold,
  timeout_ms | Uint16 | default = 200,
  key_interrupts | HoldTapKeyInterrupts | default = default_key_interrupts,
  # ...
} in

It’s not too hard to discern what values can be constructed for these types (e.g. a HoldTapKeyInterrupt could be { "decision": "none", "trigger_on": "press" }), but the types themselves aren’t sufficient to describe what the key interrupt logic is, or how the values should be used. (For this code, there’s documentation elsewhere).

A HoldTapBehavior value could be expressed in JSON. having types and documentation guide understanding.

Getting back to Nickel: it does support adding documentation by way of adding a doc metadata decoration to record fields. – But I guess I wish I saw people say “types and documentation”. (Although perhaps by the time you add enough features to the DSL, you’ve essentially just got hard-coded values in a general purpose programming language).

Tooling/LSP

I expect it will improve, but I found that the Nickel LSP server wasn’t quite clever enough to go-to-definition for all the situations I tried it in.

Overall

Overall, I was pleased with my experience.

“JSON plus functions” does let me apply creativity to describe a configuration in a way that I find to be a good balance of cleverness and clarity.

There are more costs/risks to using Nickel compared to using plain JSON, but there are also many benefits.

---

---