Hraban’s Nix Notes

Nix the language has many esoteric features. Some are rare but not novel:

• lazy evaluation (Haskell)
• currying (Haskell) (n.b.: you can emulate currying in most languages, but “true” elegant currying support requires all functions to be single arg)
• with statement (JS)

Others are truly unique:

A short intro to Nix the language. This is invaluable to understanding everything else because you actually understand what the code snippets do. Reading this doc was a turning point in my understanding of all things Nix.

After working with Nix for a while now (summer-fall 2022) I’m not sold on the present didactic value of dealing with flakes, today. It’s yet another thing to learn, yet another layer of abstraction, and the gains are quite minimal.

“Channel Nix” (a common name for pre-flake Nix) is so pervasive that you must learn it either way. You can’t avoid it. And it works fine. So… why learn flakes, again? Oh because they’re the future. I agree. But not the present.

That being said:

• Xe Iaso’s series
• https://nixos.wiki/wiki/Flakes

(I’m not yet 100% on what a channel is but I know enough to make use of them)

Channels are collections of packages that you can refer to by just their name on your local computer. The most famous one is nixpkgs, which is a mega derivation that contains all possible software you could want on a normal computer or server.

Channels are used to “bootstrap” your own derivations, if you want to build further. In particular nixpkgs which contains not just packages, but also utility nix functions, comparable to a stdlib. You manage them locally and update them manually (or through cron I guess).

Here’s the catch: you can get by without really knowing the distinction. This can make learning Nix confusing, because sometimes you do need to know, and it’s one more confusingly named thing making an appearance which you could previously ignore.

(I strongly advise against using any channels but nixpkgs until you are so comfortable with it all that you will ignore any advice regardless. If you need a custom bundle of packages somehow, just put that bundle in its own file and import that, or host it on github and import it from there, or … etc. Using channels locally is a dangerous game.)

Here’s what you need to know about channels on a Mac (with “multi-user install”):

# The main user doesn’t have any channels:
user$ nix-channel --list

# But the root does:
user$ sudo -i

root# nix-channel --list
nixpkgs https://nixos.org/channels/nixpkgs-unstable

# So you update your channels as root:
root# nix-channel --update
unpacking channels...

root# logout

# And now you update all packages you installed as a user:
user$ nix-env -u
upgrading 'ffmpeg-full-4.4.2' to 'ffmpeg-full-5.1.2'
upgrading 'gcc-wrapper-11.3.0' to 'gcc-wrapper-12.2.0'
these 8 paths will be fetched (76.26 MiB download, 309.64 MiB unpacked):
  /nix/store/3ai4nmvmxiyw7l82nfz1czf2zzxml2p6-gcc-wrapper-12.2.0
  ...

The concept here is: nix-env installs from the nixpkgs channel—it doesn’t really know about updates. It just has this big derivation collection where every piece of software has only one version, one way it’s built (its derivation). It gets its nixpkgs channel from the global copy of that channel. Who manages that? Root. So to update your packages, you first must update that nixpkgs channel (as root of course). But root doesn’t know what packages you installed so it can’t run nix-env -u (meaningfully)–that’s a nix-env thing for your user only. Once nixpkgs is updated globally, you go back to your user and say, given this new nixpkgs, which of my globally installed packages need updating? That’s nix-env -u.

Contrast this to a regular package manager which combines versions, package repositories and installations all in one. Example: Homebrew knows what version of a package you installed, what versions are available, and where it can get those packages. Same for apt, etc. Yes they all have a separate update and upgrade step, but because it’s the same tool that manages everything, it can be all done by the same user. In Nix, because of the big separation between the user managing the channel and the user managing your local profile installations, that’s not possible.

N.B.: Again, this is on a Mac with “multi user installation” (the default). I don’t know how other systems do it (although I’m pretty sure any sane NixOS configuration will be the same).

Note also channels can be used anywhere in any nix derivation by doing <channel-name>. The reason to avoid this is that it makes your derivation files non-portable and implicitly dependent on state. This is not the same as files in your ~/.nix-defexpr which are only picked up by nix-env (that’s fine).

Flakes (as far as I understand) pin all their inputs, including any potential channels. So your flake.lock will contain a hash specifying exactly which nixpkgs version you used to build that flake. That makes flakes more repeatable than channel-based nix, which is still somewhat dependent on global state (the particular version of nixpkgs installed on your system that day). Although nixpkgs does its best to stay backwards compatible and it works most of the time. Versioning in Nix is an unsolved can of worms, tbh, and now you have the tools to understand where the pain starts.

I am working on my own module system combining Common Lisp and Nix: cl-nix-lite.

• https://nixos.wiki/wiki/Overlays
• https://flyingcircus.io/blog/nixos-the-dos-and-donts-of-nixpkgs-overlays/

This is getting hard-core but it’s useful to pin a package to a version.

I’m not entirely clear on what they are, yet, but every collection of packages (Haskell, Ruby, Qt, …) seems to be part of a so-called “scope”.

Cf overrideScope', pkgs.lib.makeScope and pkgs.newScope.

• https://nixos.wiki/wiki/Overlays#Overriding_a_package_inside_a_scope

Definitely.

{
  allowUnfree = true;
}

Presentation: Nix + Docker, a match made in heaven.

https://nix.dev/tutorials/building-and-running-docker-images

This is a lot of work:

https://iohk.zendesk.com/hc/en-us/articles/4415830650265-Uninstall-nix-on-MacOS

and

https://github.com/NixOS/nix/issues/458#issuecomment-1264906595

Nix stores all files in the “nix store”. This is the plumbing, like how git’s underlying data model is its object database.

The Nix store is an object store with an important feature: one of the valid types of elements it can store is a directory of files. In practice, the key ends almost always up being the hash of the derivation (I think?). This is used for memoization of builds. It is used to store the source, which is built into a result, and that result itself is also stored as a separate entry in the store.

Nix builds are heavily cached. Locally using the nix store, remotely using some remote caching system. Excellent demonstration here:

https://scrive.github.io/nix-workshop/06-infrastructure/01-caching-nix.html

In particular, the linked fibonacci nix explains local caching:

# Run with nix-build -v -E 'import ./fib.nix "foo" 4'
let
  nixpkgs = import <nixpkgs> { };

  inherit (nixpkgs) stdenv;

  prefixed-fib = prefix:
    let fib = n:
      assert builtins.isInt n;
      assert n >= 0;
      let
        n-str = builtins.toString n;
      in
        if n == 0 || n == 1
        then
          stdenv.mkDerivation {
            name = "${prefix}-fib-${n-str}";
            unpackPhase = "true";

            buildPhase = ''
              echo "Producing base case fib(${n-str})..."
              sleep 3
              echo "The answer to fib(${n-str}) is ${n-str}"
            '';

            installPhase = ''
              mkdir -p $out
              echo "${n-str}" > $out/answer
            '';
          }
        else
          let
            fib-1 = fib (n - 1);
            fib-2 = fib (n - 2);

            n-1-str = builtins.toString (n - 1);
            n-2-str = builtins.toString (n - 2);
          in
          stdenv.mkDerivation {
            name = "${prefix}-fib-${n-str}";
            unpackPhase = "true";

            buildPhase = ''
              fib_1=$(cat ${fib-1}/answer)
              fib_2=$(cat ${fib-2}/answer)

              echo "Calculating the answer of fib(${n-str}).."
              echo "Given fib(${n-1-str}) = $fib_1,"
              echo "and given fib(${n-2-str}) = $fib_2.."

              sleep 3

              answer=$(( $fib_1 + $fib_2 ))
              echo "The answer to fib(${n-str}) is $answer"
            '';

            installPhase = ''
              mkdir -p $out
              echo "$answer" > $out/answer
            '';
          }
    ;
  in fib;
in
prefixed-fib

UPDATE: Actually, when I run that with 50 instead of 5, it does explode! It doesn’t get to caching quickly enough and somewhere in the planning of jobs it creates a 50! large graph.

https://discourse.nixos.org/t/why-is-fetchtarball-not-mentioned-in-chapter-11-fetchers-of-the-nixpkgs-manual/15319

There are two families of fetchers: builtins (implemented in Nix source in C++) and fetchers, which are Nix fetchers implemented in Nix, in nixpkgs.stdenv. You want to use the latter. Crucially:

The builtins (e.g. builtins.fetchurl) are defined in Nix source code. One difference is that the built-in functions cause download during evaluation, which is bad for all sorts of reasons.

Very important difference!

This is where the utils live

https://teu5us.github.io/nix-lib.html

A unique feature of the Nix language that I haven’t seen anywhere else.

Strings have a hidden property, “context”, which is a set of derivations. Meaning a string in Nix can have hidden references to other nix packages. These sets grow on concatenation, so if you have one string referencing { A }, another referencing { B }, and you concat them (or any of their substrings), you get a new string with context { A, B }.

This neat feature allows Nix to automatically infer the dependencies for a derivation by looking at the union of all derivations in all of the strings in its own derivation. This is why when you have for example:

buildPhase = ''
  ${pkgs.gcc}/bin/gcc -o foo foo.c
''

You don’t need to explicitly specify GCC as a build dependency: it becomes a member of the context of the buildPhase, and through that is found as a dependency of the entire derivation.

Neat!

This is also why they discourage using:

buildPhase = ''
  gcc -o foo foo.c
''

Even though that technically works, Nix now doesn’t know that you depend on the GCC derivation.

Example:

nix-repl> :p builtins.getContext "${p.sbcl} and ${p.scala}"
{
  "/nix/store/3pw34cakm1vx6mc8zhik6vrmgg3rs2ni-sbcl-2.2.9.drv" = { outputs = [ "out" ]; };
  "/nix/store/qsahyw0h47f9nkcby97ganq7w59aw1i3-scala-2.13.10.drv" = { outputs = [ "out" ]; };
}

This is the first time I learn something about Nix that answers more questions than it raises.

This is a hard one to pin down. I have not found a satisfying explanation. And I’m not sure if it’s actually a Nix-the-language feature, or a stdlib thing.

Simple demo of difference between evaluating a derivation, and building it:

Let’s evaluate a derivation:

nix-repl> p.writeText "test" "hey"
«derivation /nix/store/y5mprkgdmiqjzwly9qwjhz9n196jy93x-test.drv»

nix-repl> (p.writeText "test" "hey").outPath
"/nix/store/66i0ggdxdk73in27dsmyqn0n696041nd-test"

The derivation has been built:

cat /nix/store/y5mprkgdmiqjzwly9qwjhz9n196jy93x-test.drv

Derive([("out","/nix/store/66i0ggdxdk73in27dsmyqn0n696041nd-test","","")],[("/nix/store/8b6s8cgrp0m25fnfqq8qrx54d91v8vgg-stdenv-darwin.drv",["out"]),("/nix/store/nb5r5f4lj8cy0di6894zn483976mxni4-bash-5.1-p16.drv",["out"])],["/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"],"x86_64-darwin","/nix/store/dx09hl4rkg0smsk216qnh033rzx2y04y-bash-5.1-p16/bin/bash",["-e","/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"],[("__darwinAllowLocalNetworking",""),("__impureHostDeps","/bin/sh /usr/lib/libSystem.B.dylib /usr/lib/system/libunc.dylib /dev/zero /dev/random /dev/urandom /bin/sh"),("__propagatedImpureHostDeps",""),("__propagatedSandboxProfile",""),("__sandboxProfile",""),("allowSubstitutes",""),("buildCommand","target=$out''\nmkdir -p \"$(dirname \"$target\")\"\n\nif [ -e \"$textPath\" ]; then\n  mv \"$textPath\" \"$target\"\nelse\n  echo -n \"$text\" > \"$target\"\nfi\n\neval \"$checkPhase\"\n\n(test -n \"$executable\" && chmod +x \"$target\") || true\n"),("buildInputs",""),("builder","/nix/store/dx09hl4rkg0smsk216qnh033rzx2y04y-bash-5.1-p16/bin/bash"),("checkPhase",""),("cmakeFlags",""),("configureFlags",""),("depsBuildBuild",""),("depsBuildBuildPropagated",""),("depsBuildTarget",""),("depsBuildTargetPropagated",""),("depsHostHost",""),("depsHostHostPropagated",""),("depsTargetTarget",""),("depsTargetTargetPropagated",""),("doCheck",""),("doInstallCheck",""),("enableParallelBuilding","1"),("enableParallelChecking","1"),("executable",""),("mesonFlags",""),("name","test"),("nativeBuildInputs",""),("out","/nix/store/66i0ggdxdk73in27dsmyqn0n696041nd-test"),("outputs","out"),("passAsFile","buildCommand text"),("patches",""),("preferLocalBuild","1"),("propagatedBuildInputs",""),("propagatedNativeBuildInputs",""),("stdenv","/nix/store/phj9vavfvy45gadbc5d9fnirpb50mqw3-stdenv-darwin"),("strictDeps",""),("system","x86_64-darwin"),("text","hey")])

(Use nix show-derivation for saner output)

So, the derivation has been built, that means the package was built, and we should have the file, right?

cat /nix/store/66i0ggdxdk73in27dsmyqn0n696041nd-test

cat: /nix/store/66i0ggdxdk73in27dsmyqn0n696041nd-test: No such file or directory

Oh, we just evaluated the derivation, but we didn’t build it. A derivation is the final “build script”. If you see Nix as “yaml with functions”, then a derivation is the final ansible file you’d be left with after all functions have been executed. But that’s just the beginning of actually building the software.

Let’s use the REPL to do that:

nix-repl> :b (p.writeText "test" "hey")

This derivation produced the following outputs:
  out -> /nix/store/66i0ggdxdk73in27dsmyqn0n696041nd-te

And try again:

cat /nix/store/66i0ggdxdk73in27dsmyqn0n696041nd-test

hey

Sick.

buildInputs:

programs and libraries used by the new derivation at run-time

nativeBuildInputs:

programs and libraries used at build-time that, if they are a compiler or similar tool, produce code to run at run-time—i.e. tools used to build the new derivation

buildInputs should persist at runtime. To take an example from discourse: if you build a wrapper tool around jq that automatically handles compression, using zlib, you’ll need both those tools available at runtime (and the zlib derivation at build time, at least for its headers). Nix does set up the envvars at build time for you, but because the output is just a binary, you can’t rely on Nix to set the PATH or LD_LIBRARYPATH at runtime: binaries are binaries. You must ensure that, at buildtime, somehow, you hard-code the path to those tools. This is done behind the scenes using patchelf.

nativeBuildInputs is only for tools you use to do the building, e.g. GNU Make. These are “native” to the platform doing the building.

Nix could theoretically consider those nativeBuildInputs garbage collectable? I don’t think it does that, but it could. I think.

• Nixpkgs manual: Variables specifying dependencies
• BuildInputs not propagating to the derivation
• Use `buildInputs` or nativeBuildInputs` for `nix-shell`?

Derivations expressed using stdenv have a build, host, and target platform. These attributes are, respectively, the platform on which a derivation is built, the platform on which it runs, and, for certain sorts of packages (e.g. a compilers), the platform which it targets.

When not cross-compiling, every derivation’s build, host, and target platforms are the same. However, when cross-compiling, they differ. For example, a package which is cross-compiled from x86₆₄-linux to aarch64-linux, has (build, host, target) platforms at (x86₆₄-linux, aarch64-linux, aarch64-linux).

https://github.com/NixOS/nixpkgs/issues/68967#issuecomment-532432617

Fixpoints are a way to solve “self recursion”, which allows you to implement functions which refer to their own output from within themselves. This is mind bending if you think of it in an imperative context, but it makes total sense if you see functions’ return values as “aspirational”, aka lazy. What made it click for me was seeing a function returning an attrset, as a function returning a big bag of getter-attributes.

Example: given this function:

myfunc = { bla }: {
  a = 5 + bla;
  b = do-a-thing bla;
}

Instead of seeing it as a function returning an attrset like that, I mentally see it like this:

myfunc = { bla }: {
  get a = () => 5 + bla;
  get b = () => do-a-thing bla;
}

This mental model helps me wrap my head around lazy evaluation, fixpoints, etc.

Now suddenly, it makes more sense you could get your own return value as an argument to yourself:

myfunc = { self, bla }: {
  get a = () => self.b + bla;
  get b = () => do-a-thing bla;
}

This makes sense because things are only evaluated on-demand. Even that self is only evaluated on-demand. So:

let res = myfunc { self = res; bla = 1234; }

This works because at this point, nothing is actually evaluated at all. It’s just a bunch of callbacks setup, in case anyone actually gets called. And even when you call self.b, it still doesn’t call self.a or self. Just that one callback.

I only really grokked this once I implemented an entire scope myself (lisp-packages-lite), ran into a problem (“how do I override a dependency and make all other packages in my scope automatically use that updated dependency?”) and needed a way to fix that problem. Overrides are the only way.

https://r6.ca/blog/20140422T142911Z.html

This is very down & dirty, advanced, dirty stuff, and you don’t often need it or run into it at all.

Setup hooks are a magic handle for a dependency (e.g. libgz) to automatically run things in dependent derivations (e.g. tar, assuming it uses libgz). E.g.: setting its PATH or LD_LIBRARYPATH or something (this doesn’t actually happen but it could).

I’m not intimately familiar with them yet but they seem a missing piece of a confusing puzzle for me, already.

• https://nixos.org/manual/nixpkgs/stable/#ssec-setup-hooks
• https://nixos.org/guides/nix-pills/basic-dependencies-and-hooks.html

On MacOS (at least in multi-user mode but maybe also in single-user mode) nix builds are managed by a separate daemon process. This process gets its configuration from nix.conf.

But the nix user facing CLI tool itself also gets its configuration from nix.conf. This can be confusing: sometimes you want to change an option, so you change it in nix.conf, re-run your nix command, but it doesn’t seem to get picked up! E.g. enabling or disabling the sandbox. Why?

What happened is that your option was a nix daemon option (e.g. sandbox = true/false), but you didn’t restart the daemon, so the option hasn’t taken effect yet.

On Mac, you can do:

sudo launchctl stop org.nixos.nix-daemon
sudo launchctl start org.nixos.nix-daemon

Now the daemon will have picked up your new config.

UPDATE: Bollocks, just use the empty string.

Use this if you don’t know what the real hash is so at least the derivation compiles, and nix-build will fail and tell you the right hash.

This is only necessary because the length of a checksum string is checked at evaluation time, so you can’t just say e.g. sha256 = "foo"; that will not evaluate.

Equivalent of Python’s {}.get("foo", "bar")

{}.foo or 3

3

{ foo = 15; }.foo or 3

15

Test the presence of a key in an attrset:

{} ? a

false

{a = false;} ? a

true

Using a name contained in a variable:

let x = "foo"; in { foo = 3; } ? ${x}

true

Every key of the attrset you pass to mkDerivation will be stored in the final derivation. Derivations only support primitive types, no attrsets, functions, etc. They’re also strictly evaluated.

In order to store a “regular” object in a derivation you can use the passthru key:

mkDerivation {
  name = "yada";
  otherargs = "blabla";

  passthru = {
    here = i: can: "store anything, ${i} ${can}";
  };
}

The passthru key won’t be available to anyone importing the derivation from the store (e.g. using import).

let .. in is syntactic sugar for with rec { } :, with one subtle difference around function argument shadowing (which I can’t recall right now).

Useful debugging trick:

let
  x = "hidden stuff you want to inspect";
  y = "more hidden vars you want to access and evaluate for debugging";
in
  mkDerivation {
    name = "how-do-I-get-access-to-those-hidden-vars";
  }

Try this:

let root = rec {
  x = "hidden stuff you want to inspect";
  y = "more hidden vars you want to access and evaluate for debugging";
}; in
  with root;
  mkDerivation {
    name = "how-do-I-get-access-to-those-hidden-vars";
    passthru = { inherit root; };
  }

Now you can access the entire let binding (because really it’s just a single attrset) from the REPL.

You can filter files from a directory, essentially by creating a new derivation.

{ ${null} = 3; }

{ }

Useful in conditionals:

{ ${if 1 == 2 then "foo" else null} = 3; }

{ }

Similar:

let pkgs = (import <nixpkgs> {});
in
  pkgs.lib.attrsets.optionalAttrs (1 == 2) { foo = 3; }

{ }

Set parameter dontUnpack = true; on pkgs.stdenv.mkDerivation:

pkgs.stdenv.mkDerivation {
  name = "foo";
  dontUnpack = true;
  installPhase = ''
    mkdir -p $out/bin
    echo echo foo > $out/bin/foo
    chmod +x $out/bin/foo
  '';
}

Put this in front of any code you want to debug by seeing its value:

Before:

let
  my-func = x: (x * x);
in
  ...

After:

let
  my-func = x: builtins.trace "Got x: ${x}" (x * x);
in
  ...

{pkgs, ...}:

{
  imports = [
    # On Gandi:
    # <nixpkgs/nixos/modules/virtualisation/openstack-config.nix>
    # ./gandicloud.nix

    # On a VM from a nixos iso:
    ./hardware-configuration.nix
  ];

  nix.settings.experimental-features = [ "nix-command" "flakes" ];

  environment = {
    systemPackages = with pkgs; [
      emacs
      file
      htop
      iotop
      lsof
      moreutils
      mosh
      netcat
      pstree
      pv
      rsync
      tmux
      traceroute
      tree
      wget
    ];
    variables.EDITOR = "emacs";
  };
  system = {
    autoUpgrade = {
      enable = true;
      allowReboot = true;
    };
  };
  programs.mosh.enable = true;

  users.users.asdf = {
    isNormalUser = true;
    description = "Main user";
    extraGroups = ["networkmanager" "wheel"];
    openssh.authorizedKeys.keys = [ "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF0+BDB4kr/I+kAyoWTVRtzGqpRzBTTjVAPD9cYCujUy" ];
    packages = with pkgs; [ ];
  };

  # Bootloader.
  boot.loader.systemd-boot.enable = true; boot.loader.efi.canTouchEfiVariables = true;
  boot.loader.efi.efiSysMountPoint = "/boot/efi";

  systemd.services.foo = {
    script = ''
    echo "Doing some stuff"
    (date ; echo hello bye bye) >> /tmp/hello-bye
  '';
    wantedBy = [ "multi-user.target" ];
  };


  # Set your time zone.
  time.timeZone = "America/New_York";

  # Select internationalisation properties.
  i18n.defaultLocale = "en_US.UTF-8";

  i18n.extraLocaleSettings = {
    LC_ADDRESS = "en_US.UTF-8";
    LC_IDENTIFICATION = "en_US.UTF-8";
    LC_MEASUREMENT = "en_US.UTF-8";
    LC_MONETARY = "en_US.UTF-8";
    LC_NAME = "en_US.UTF-8";
    LC_NUMERIC = "en_US.UTF-8";
    LC_PAPER = "en_US.UTF-8";
    LC_TELEPHONE = "en_US.UTF-8";
    LC_TIME = "en_US.UTF-8";
  };

  services.openssh.enable = true;
  # # Enable the X11 windowing system.
  # services.xserver.enable = true;

  # # Enable the KDE Plasma Desktop Environment.
  # services.xserver.displayManager.sddm.enable = true;
  # services.xserver.desktopManager.plasma5.enable = true;

  # # Configure keymap in X11
  # services.xserver = {
  #   layout = "us";
  #   xkbVariant = "alt-intl";
  # };

  # Configure console keymap
  console.keyMap = "us";

}

After saving, run nixos-rebuild switch (or use test first, check it works, then switch).

(setq-local org-confirm-babel-evaluate nil)
(setq explicit-shell-file-name "/run/current-system/sw/bin/bash")

date
echo hi
hostname

A 1GB VPS doesn’t have enough memory for nixos-rebuild switch. Nothing will appear to happen.

You can detect this by adding ; echo $? after the command: if it prints 137 you’re OOM. It should output 0.

Add a swap file, e.g. in /root/swap, then run the command again.

nixos-rebuild -v --upgrade switch; echo $?

I’ve added ; echo $? here because the failure mode is otherwise silent, and it happened to me on a minimal size VM from gandi. See Configuration.

NixOS is pinned to a release by setting the nixos channel. See which one:

nix-channel --list

If you want to upgrade to a newer version, or in this case the “always bleeding edge latest”:

nix-channel --add https://nixos.org/channels/nixos-unstable nixos

And run this to upgrade your system:

nixos-rebuild switch --upgrade