Example of Cannot Execute Required File Not Found on NixOS

One of the problems many NixOS users encounter is difficulty running binaries which have been compiled on other Linux systems.

The error message from trying is typically:

cannot execute: required file not found

which is confusing because the file is right there.

In this post, I’ll walk through a “hello world” example, which hopefully illustrates what’s going on.

Quickly Entering non-NixOS from NixOS

A simple example of this is a “hello world” C program compiled on a non-NixOS system.

From NixOS, we can still enter a non-NixOS system easily using distrobox. (Distrobox is similar to Fedora’s toolbox; it lets you easily work with your HOME directory from within a container, which can be useful for OSs like NixOS or Fedora Silverblue).

Distrobox requires docker or podman; so, ensure that’s enabled:

virtualisation.podman.enable = true;

e.g. should be able to run:

podman version

Then we can run distrobox & create an ubuntu distrobox with:

$ nix shell nixpkgs#distrobox

$ distrobox create ubuntu --image ubuntu:latest

And enter this with distrobox enter ubuntu.

Building Hello World in the Ubuntu Distrobox

Within the ubuntu distrobox, we install build-essential (since we want to compile a simple C program):

$ sudo apt update
$ sudo apt install build-essential

then e.g. hello.c (unintuitively, you can use the text editor from the host system, such as helix or neovim, even within the ubuntu distrobox):

#include <stdio.h>

int main(int argc, char **argv) {
  printf("hello world\n");
}

And then compile this (gcc hello.c). And run it, if you like: ./a.out.

Examining a.out

Checking what ldd says about the compiled hello.c, from within the ubuntu distrobox:

$ ldd ./a.out
        linux-vdso.so.1 (0x00007ffe04ff9000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcda5200000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fcda561d000)

and then leaving the ubuntu distrobox (e.g. ^D), from the NixOS host we see a different result:

$ ldd ./a.out
        linux-vdso.so.1 (0x00007ffe06e36000)
        libc.so.6 => /nix/store/anlf335xlh41yjhm114swi87406mq5pw-glibc-2.38-44/lib/libc.so.6 (0x00007f6dd47d1000)
        /lib64/ld-linux-x86-64.so.2 => /nix/store/anlf335xlh41yjhm114swi87406mq5pw-glibc-2.38-44/lib64/ld-linux-x86-64.so.2 (0x00007f6dd49c1000)

Trying to run this on the NixOS host results in an error:

$ ./a.out
bash: ./a.out: cannot execute: required file not found

Using fish shell, the error is more useful:

$ ./a.out
exec: Failed to execute process './a.out': The file exists and is executable. Check the interpreter or linker?

A Bash Script Analogy

ld-linux is the dynamic-linker.

What’s going on is comparable to executing a shell script with a shebang:

#!/usr/bin/env bash
echo moo | cowsay

If we have an executable script with a bad interpreter, e.g. badinterp:

#!/nonexist
echo "hello"

Then trying to run this with bash yields the same “required file not found” error:

$ ./badinterp
bash: ./badinterp: cannot execute: required file not found

although fish shell once again is more useful:

$ ./badinterp
exec: Failed to execute process './badinterp': The file specified the interpreter '/nonexist', which is not an executable command.

If running the above executable script (e.g. cowsay-moo) without cowsay on PATH, we get a different error:

$ ./cowsay-moo
./cowsay-moo: line 2: cowsay: command not found

That the executable isn’t on the PATH is a much easier to understand.

The other most common kind of linker error is analogous to that:

error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory

whereas a shell uses PATH to find exectuables, the dynamic linker uses LD_LIBRARY_PATH to find dynamically linked libraries.

This is also why setting LD_LIBRARY_PATH often results in problems related to “version GLIBCXX_ not found”. – By setting the LD_LIBRARY_PATH, the dynamic linker loads a glibc version that’s different than what the program expects, and so runs into problems.

Working Around the Problem: FHS Environments

One workaround provided by nixpkgs is FHS environments. (These are implemented using bubblewrap, which flatpak uses).

e.g. in this case, we can write a simple fhs.nix:

{ pkgs ? import <nixpkgs> {} }:

pkgs.buildFHSEnv {
  name = "hello";
  runScript = ./a.out;
}

and then build/run this with nix-build ./fhs.nix, and running ./result/bin/hello.

(FWIW, this FHS env would need to be re-built every time the binary changes).

Automating this Workaround: nix-alien

This approach of “write an FHS env for the unpatched binary, and run that” has been automated by the tool nix-alien.

We can add nix-alien to the shell with:

$ nix shell "github:thiagokokada/nix-alien#nix-alien"

and then run this with:

$ nix-alien ./a.out
[nix-alien] File '/home/rgoulter/.cache/nix-alien/f5127167-26a2-5c7d-89d0-fb363e223e43/fhs-env/default.nix' created successfuly!
....
hello world

The thing to note is that it creates this .cache/nix-alien/UUID/fhs-env/default.nix file.

NOTE: if you change the arguments to nix-alien, you may need to call nix-alien with --recreate to ensure it recreates the default.nix.

Examining the default.nix it creates:

{ pkgs ? import
    (builtins.fetchTarball {
      name = "nixpkgs-unstable-20240509145238";
      url = "https://github.com/NixOS/nixpkgs/archive/f1010e0469db743d14519a1efd37e23f8513d714.tar.gz";
      sha256 = "sha256-doPgfj+7FFe9rfzWo1siAV2mVCasW+Bh8I1cToAXEE4=";
    })
    { }
}:

let
  inherit (pkgs) buildFHSUserEnv;
in
buildFHSUserEnv {
  name = "a.out-fhs";
  targetPkgs = p: with p; [

  ];
  runScript = "/home/rgoulter/playground/distrobox-build/a.out";
}

The builtins.fetchTarball is essentially performing the same role as <nixpkgs>. (The latter uses nixpkgs on NIX_PATH, the former uses a particular revision of the nixpkgs repository).

buildFHSUserEnv is an alias for buildFHSEnv.

targetPkgs is empty (so doesn’t have any effect on this example), but it’s useful for additional dynamically linked libraries the precompiled binary may need.

A Universal NixOS Workaround:

Another solution, which works in some cases where buildFHSEnv does not, is nix-ld.

This can be enabled by updating your NixOS config with:

  programs.nix-ld.enable = true;

Directly running the same binary again leads to a different error message:

$ ./a.out
cannot execute ./a.out: You are trying to run an unpatched binary on nixos, but you have not configured NIX_LD or NIX_LD_x86_64-linux. See https://github.com/Mic92/nix-ld for more details

As the nix-ld readme suggests, we can set NIX_LD with a shell.nix:

{ pkgs ? import <nixpkgs> {} }:
pkgs.mkShell {
  NIX_LD_LIBRARY_PATH = pkgs.lib.makeLibraryPath [];
  NIX_LD = pkgs.lib.fileContents "${pkgs.stdenv.cc}/nix-support/dynamic-linker";
}

entering this with nix-shell, and running this:

$ ./a.out
hello world

Another Workaround: patchelf

Another way to work around the program is to just patch the binary directly. (This is typically how precompiled binaries are handled in nixpkgs packages).

This can be done with patchelf. Again, adding it to PATH with:

nix shell nixpkgs#patchelf

then e.g. copying the interpreter from the output of ldd above:

$ patchelf --set-interpreter /nix/store/anlf335xlh41yjhm114swi87406mq5pw-glibc-2.38-44/lib64/ld-linux-x86-64.so.2 ./a.out

and running the binary:

$ ./a.out
hello world

Automatic patchelf in Nix Packaging: autoPatchelfHook

When writing Nix packages, nixpkgs has an autoPatchelfHook which can take care of automatically performing tasks like this.

This makes more sense when packaging a prebuilt release of some program, but we can still apply it to this prebuilt ./a.out:

e.g. a hello.nix file with:

{ pkgs ? import <nixpkgs> {} }:

pkgs.stdenv.mkDerivation {
  pname = "hello-world";
  version = "1.0.0";
  buildInputs = [
    pkgs.autoPatchelfHook
  ];
  dontUnpack=true;
  installPhase=''
    mkdir -p $out/bin
    cp ${./a.out} $out/bin/hello
  '';
}

this can be built with nix-build hello.nix and the resulting patched binary run with ./result/bin/hello.

The autoPatchelfHook is invoked as part of the ‘fixup’ phase.