Experience Updating the C Worksheet Project

Posted on May 18, 2024 by Richard Goulter
Tags: , , ,

Around a decade ago, I made a pretty neat project: a “worksheet” user interface for simple C programs. – By executing the worksheet, you get to see the program’s output annotated alongside the source code.

An illustrative example:

int main(int argc, char* argv) {
  int x = 5;                                      //> x is int
  int *px;                                        //> px is pointer to int
  px = &x;                                        //> px = 0x7ffc365efd5c = 5
  *px = 3;                                        //> (*px) = 3
  x;                                              //> 3
  px = 0;                                         //> px = (nil)
  *px = 5;                                        //> SEGFAULT
  x;
  printf("done.\n");
}

The //> comments are automatically generated by the worksheet program.

More examples on the c-worksheet-instrumentor readme.

The project consists of a “c-worksheet” program which does all the heavy lifting of computing the C worksheet annotations, and a vim plugin which provides a nice UI for using the worksheet functionality.

I recently performed some maintenance on it to bring it up to scratch, & to check that it’s in working condition.

You’d hope untouched code would stay working. – Nothing changed, so nothing should have broken.
In reality, since the project dependencies constantly get updated (e.g. library dependencies, and system tools), some code needed to change in order to build the code from source.

Logging some thoughts here:

Updating the C Worksheet Instrumentor (Scala Project)

The “C Worksheet Instrumentor” is the Scala part of the project which does the heavy work.

Overall, the changes I made involved:

Most of the project was written in 2015 as part of my undergraduate dissertation.
Since then, I did add CI/CD jobs to the project in 2018.

Implementation-wise:
This “C worksheet instrumentor” project was implemented using Scala, & a parser written with Antlr (which generates Java code from a grammar definition file). The tests were written using ScalaTest.

This tech stack is different-enough from “only Scala files” so as to make setting up the build files annoying.

For building this, I used gradle.

For Scala, I think the SBT (Scala Build Tool) is more common. I wrote about my frustrations trying to set up the project with either back then. (Since then, I moved to use gradle; I think because I was able to add a ProGuard configuration to gradle, but didn’t for SBT).
– I wish I could say my conclusion of “they both suck” was sophomoric. Perhaps “they’re both painful to set up” would be a more precise and tactful way of expressing it.

Updating Gradle

In this case, where the build.gradle file was written in mid-2015, the up to date release of gradle would have been v2.3; whereas at the time of writing, it’s Gradle 8.7.
– I wasn’t able to use gradle 8.7 to run this old gradle file.

I’ve since learned Gradle tooling itself does have solutions for “different projects depend on different gradle versions”: gradle supports generating a “gradle wrapper” script. This wrapper script will download & invoke the appropriate gradle version.
However, for my codebase, using a gradle wrapper generated by gradle 2.3 still ran into some problems:
I couldn’t successfully run the gradle 2.3 wrapper using jdk 21. Not a big deal.
Using jdk 8, I managed to at least build the project; but the tests failed due to a classpath error. (Presumably a transitive dependency updated with a requirement for a more recent Gradle version. – There wasn’t a lockfile of versions used; although Gradle apparently introduced this feature in Gradle 6.8).

Anyway.
Since the code didn’t build, I stripped the build.gradle file down to its basics in order to get something to build.
Previously, I had added code to the build.gradle related to shadowjar and proguard. (shadowjar would help to bundle dependency JARs all into one JAR file, and proguard could assemble a small JAR with stripped down dependencies). I recall adding these partly because I was frustrated with how slow the internet took to download a 20MiB JAR from GitHub Releases. And partly because I liked the idea of the distributed lib/ dir having a single JAR.
– Distributing software is hard. I knew less about that back then than I do now, I feel.

I was then able to get an LLM chatbot to help advise me on the changes I’d need to make to the build.gradle file for a more recent gradle.
e.g. it was able to explain to me to change from jcenter() to mavenCentral(), and syntax changes to the dependencies {} and jar {} blocks. This saves several minutes of time & a search or two.

And since Scala 2.11 didn’t run with a recent Java Runtime Environment, I updated the code (and matching dependencies) to Scala 2.12.

In the end: diff of changes to get the code to build.

Updating CI

One thing I did do well with this project was to have a fairly good test suite.

I’d left the project in a state where it had CI jobs running on Travis CI, and on Appveyor.

I do appreciate that these CI services often will let you run jobs for free, if your repository is public.

The CI jobs I have for this project are almost trivial: install dependencies & check that gradle build runs.

In the years since, Travis CI stopped running my jobs. So, I moved the builds for Linux/macOS to GitHub Actions.

The builds failed AppVeyor due to Java version issues.
I got stuck for a moment since AppVeyor wasn’t reading the appveyor.yml file for build details. Deleting/recreating the AppVeyor sorted things out. (It had been something to do with AppVeyor’s behaviour for arbitrary git projects vs for GitHub-linked projects).

Adding Nix

The nix package manager has been a valuable tool to learn. It’s got a steep learning curve, but it’s very good at dealing with problems related to packaging. (Which, for developers, turns out to be a lot).

One of the ways that writing CI/CD can be frustrating is for cases where something works locally, but figuring out how to run it on a fresh host can be frustrating.
With nix, this isn’t an issue. – Indeed, the CI/CD job I added which used nix really is trivial: it installs nix, and then runs nix build ..

Another advantage with Nix is easy setup for isolated development environments. I remember reading blogposts like “my development setup for ”. These days, it’s pretty much “a formatter, a linter, a language server”. Using a shell.nix means it’s easy to make these tools available (i.e. installed) when working on the project.

The difficulty I ran into adding Nix support for the project was due to differences between how gradle wants to manage dependencies, vs how nix wants to. Searching through the nixpkgs codebase, a common approach is to use some kind of Fixed Output Derivation deps pattern in order to deal with the idiosyncracies of each. (Also, gradle uses ~/.gradle, and nix builds its packages with HOME set to a non-existant directory, so the gradle.user.home had to be explicitly set to avoid issues).

Minor Polishing

With the code building from source, and CI/CD in working state so as to run the build/tests for each commit, changes can be made with the confidence that if anything breaks, CI/CD will be able to catch problems.

I spent time to add an “End to End” integration test: a test which called the command line program, and checked the results of the output. This was an improvement over the existing test suite, which ran the Scala code in-process.

I also noticed that some tests failed when running on macOS where cc defaults to clang.
Turns out, some of the Worksheet functionality works with GCC but not Clang.
So, I refactored the tests so that the relevant parts of the test suite could be run with either GCC or with Clang.

Another polish to add to the project was using scalafmt to ensure consistent code formatting across the project.

Chatbot LLMs Often Useful

As mentioned, I found a chatbot LLM to be useful for helping to update the code.

To be honest, tasks like “update CI/CD” or “update the build file” are not glamorous: they might require knowing a fair amount about the tools, but the tools provide limited value.
(Whereas e.g. Nix is a powerful tool which lends itself to use cases outside of just “build this project”, and so is worth taking the time to learn).
– For these tasks, using chatbot LLMs helps make progress without having to spend a long time learning the tool.

e.g. Since gradle files are written with the Groovy language (a Ruby-ish JVM language), and this is the only use of Groovy I have, it’s quicker to ask a chatbot “please write the Groovy code to…” than to spend time looking up Groovy syntax, and what methods Groovy lists have.

Where I found it limited, with the LLM I was chatting with, was when I was asking a dumb question (trying something that goes against the grain of a tool’s approach), the chatbot was more inclined to give me an answer that sounded helpful, than to push back and say “no, it doesn’t work that way”.

Lost Context and Knowledge

Working on a project so intermittently means you suffer from losing context/knowledge of various parts of it.

Phrasing this in terms of risk and change: using something more unusual means there’s a higher risk that it’s more difficult to change if you need to.

The more implicit and magical something is, the harder it is to identify.

Scala is a complex language with a wide variety of sophisticated features.
I haven’t had to change the Scala codebase all that much. However, I do appreciate its ‘sophisticated’ features, such as the pattern matching, and named arguments.

Updating the Vim Plugin

The C Worksheet is a neat idea, but it only really sense for the UI to be part of a text editor (rather than just listing annotated output on the command line).

I had written c-worksheet.vim back in 2015.

Implementation-wise:
The meat of the code was fairly straightforward: rather than leaning on vimscript, I used vim’s python commands in order to invoke Python code; and that Python code managed the interface between the Vim source and the c-worksheet output.

The upside to using vim’s python integration was not having to wrangle with vimscript to figure out how to interface it with the Scala program.
The downside was that it was affected by Python’s less-than-smooth transition from Python 2 to Python 3.

In 2015, it was fine to use the python2 pyeval. In 2024, I needed to use the python3 py3eval.

Other than updating the Python code, the rest of the vimscript fortunately still worked. +1 to vim’s support backwards compatability.

I saw that Vim has a new native package management functionality, with commands like :packadd.
However, I was using the autoload feature to defer loading much of my plugin’s loading, and using ftplugin/c/ in order to provide my plugin functionality to C files.. – Using :packadd is also a way of deferring when code gets loaded; but by loading the package when opening a C file, it wouldn’t then also load my plugin.
– The intricacies of how vim loads its vimscript is something I didn’t entirely recall. (Albeit, I’ve moved on to using Doom Emacs as my primary editor).

After updating the code, I took the time to write a few tests for the code.
In this case, integration tests that checked the plugin could run c-worksheet against a “Hello World” source file. (And some non-functional tests to check that the plugin behaves if the c-worksheet program isn’t available).

The main trouble I had with this was trying to setup CI/CD with Windows.
The computer I use for day to day things is Linux. I happen to have Windows installed on another partition; but, I’m not very familiar with the Windows command line.
(And also, Vim’s batch mode didn’t behave well on Windows; I was able to set up the tests using Neovim, but not Vim).

With hindsight, there are things I wish I’d done better. The code could have been written more defensively: e.g. pyeval is gated behind vim’s python2 option. My code doesn’t guard against this. Nor does it guard against whether the c-worksheet command can be run. And its ‘service management’ is naive/sloppy; the interface with the C Worksheet Instrumentor is also naive/sloppy. – The code was clearly thrown together to the level of “it’s at least useful in some cases”, rather than “it’s cleanly and robustly useful in as many cases as it can be”.

Newer post Older post