Hraban’s Nix Notes

This is a living document, it’s where I keep all my research notes while learning Nix. I try to keep this up-to-date, but it’s entirely possible you find old (or just plain bad) advice. Please send any suggestions / ideas / etc to hraban at 0brg dot net.

N.B.: This entire document is © Hraban Luyat.

What is Nix?

  • JSON with functions
  • Haskell + SH + YAML (or is it ∩ ?)
  • A package manager
  • An operating system

Nix at its core is a programming language. Many parts of the ecosystem are optional knowledge and you can get by without. The programming language that is Nix, though, is not optional: learn it, or be perennially frustrated.

Everything else can be reduced to the language.

Features unique to the Nix Language

Nix the language has many esoteric features. Some are rare but not novel:

  • lazy evaluation (Haskell)
  • currying (Haskell) (n.b.: you can emulate currying in most languages, but “true” elegant currying support requires all functions to be single arg)
  • with statement (JS)

Others are truly unique:

path literals

first class file and directory values

(I actually don’t know exactly how this works!)

Concepts and explanation

Nix - A One Pager

A short intro to Nix the language. This is invaluable to understanding everything else because you actually understand what the code snippets do. Reading this doc was a turning point in my understanding of all things Nix.

Flakes

After working with Nix for a while now (summer-fall 2022) I’m not sold on the present didactic value of dealing with flakes, today. It’s yet another thing to learn, yet another layer of abstraction, and the gains are quite minimal.

“Channel Nix” (a common name for pre-flake Nix) is so pervasive that you must learn it either way. You can’t avoid it. And it works fine. So… why learn flakes, again? Oh because they’re the future. I agree. But not the present.

That being said:

Flake utils: reuse a default.nix

{ pkgs ? import <nixpkgs> {} }:

with pkgs;

stdenv.mkDerivation {
  pname = "foobar";
  version = "0.0.1";
  src = ./.;
  buildPhase = ''
    printf '#!/usr/bin/env bash\n\necho Foo blargh: "$@"\n' > foobar
    chmod +x foobar
  '';
  installPhase = ''
    mkdir -p "$out/bin"
    mv foobar "$out/bin/"
  '';
  dontStrip = true;
}
{
  description = "A flake for building foobar";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem
      (system:
        let
          pkgs = nixpkgs.legacyPackages.${system};
        in
          {
            defaultPackage = import ./default.nix { inherit pkgs; };
            devShell = import ./default.nix { inherit pkgs; };
          });
}
nix run . -- foo bar
Foo blargh: foo bar

Build a flake project

nix build .

And to remove it (so you can rebuild it):

l="$(readlink result)"
rm -f result
nix store delete "$l"

This also helps to clear the cache of a flake run: build it first so you can get the deriv path as a symlink in your pwd (named result), then do the above to remove it.

Log build output

For build:

nix -L build -vv .

For run:

nix -L --print-build-logs run -vv . -- arg1 arg2 ...

N.B.: This only works if the flake build isn’t cached. See Build a flake project.

Channels

(I’m not yet 100% on what a channel is but I know enough to make use of them)

Channels are collections of packages that you can refer to by just their name on your local computer. The most famous one is nixpkgs, which is a mega derivation that contains all possible software you could want on a normal computer or server.

Channels are used to “bootstrap” your own derivations, if you want to build further. In particular nixpkgs which contains not just packages, but also utility nix functions, comparable to a stdlib. You manage them locally and update them manually (or through cron I guess).

Here’s the catch: you can get by without really knowing the distinction. This can make learning Nix confusing, because sometimes you do need to know, and it’s one more confusingly named thing making an appearance which you could previously ignore.

(I strongly advise against using any channels but nixpkgs until you are so comfortable with it all that you will ignore any advice regardless. If you need a custom bundle of packages somehow, just put that bundle in its own file and import that, or host it on github and import it from there, or … etc. Using channels locally is a dangerous game.)

Here’s what you need to know about channels on a Mac (with “multi-user install”):

# The main user doesn’t have any channels:
user$ nix-channel --list

# But the root does:
user$ sudo -i

root# nix-channel --list
nixpkgs https://nixos.org/channels/nixpkgs-unstable

# So you update your channels as root:
root# nix-channel --update
unpacking channels...

root# logout

# And now you update all packages you installed as a user:
user$ nix-env -u
upgrading 'ffmpeg-full-4.4.2' to 'ffmpeg-full-5.1.2'
upgrading 'gcc-wrapper-11.3.0' to 'gcc-wrapper-12.2.0'
these 8 paths will be fetched (76.26 MiB download, 309.64 MiB unpacked):
  /nix/store/3ai4nmvmxiyw7l82nfz1czf2zzxml2p6-gcc-wrapper-12.2.0
  ...

The concept here is: nix-env installs from the nixpkgs channel—it doesn’t really know about updates. It just has this big derivation collection where every piece of software has only one version, one way it’s built (its derivation). It gets its nixpkgs channel from the global copy of that channel. Who manages that? Root. So to update your packages, you first must update that nixpkgs channel (as root of course). But root doesn’t know what packages you installed so it can’t run nix-env -u (meaningfully)–that’s a nix-env thing for your user only. Once nixpkgs is updated globally, you go back to your user and say, given this new nixpkgs, which of my globally installed packages need updating? That’s nix-env -u.

Contrast this to a regular package manager which combines versions, package repositories and installations all in one. Example: Homebrew knows what version of a package you installed, what versions are available, and where it can get those packages. Same for apt, etc. Yes they all have a separate update and upgrade step, but because it’s the same tool that manages everything, it can be all done by the same user. In Nix, because of the big separation between the user managing the channel and the user managing your local profile installations, that’s not possible.

N.B.: Again, this is on a Mac with “multi user installation” (the default). I don’t know how other systems do it (although I’m pretty sure any sane NixOS configuration will be the same).

Note also channels can be used anywhere in any nix derivation by doing <channel-name>. The reason to avoid this is that it makes your derivation files non-portable and implicitly dependent on state. This is not the same as files in your ~/.nix-defexpr which are only picked up by nix-env (that’s fine).

Flakes (as far as I understand) pin all their inputs, including any potential channels. So your flake.lock will contain a hash specifying exactly which nixpkgs version you used to build that flake. That makes flakes more repeatable than channel-based nix, which is still somewhat dependent on global state (the particular version of nixpkgs installed on your system that day). Although nixpkgs does its best to stay backwards compatible and it works most of the time. Versioning in Nix is an unsolved can of worms, tbh, and now you have the tools to understand where the pain starts.

Channel subdirectories

You can import a subdirectory from a channel by specifying it directly in the name:

<nixpkgs/some/file>

Lisp

I am working on my own module system combining Common Lisp and Nix: cl-nix-lite.

Overlays

This is getting hard-core but it’s useful to pin a package to a version.

Pin system wide

If you’re lucky the older version of the package is also in nixpkgs. For example:

[
  # sbcl 2.2.6 constantly rebuilds from source and there's a problem on mac
  (self: super: { sbcl = super.sbcl_2_2_4 ; })
]

But this is the exception rather than the rule. See the overlay wiki if you need to rebuild from scratch.

Deep override system wide

In the above case, the overlay is not enough, because it’s an overlay which means that existing packages don’t get to see it. So anything in nixpkgs that depends on sbcl will use the broken sbcl 2.2.6.

We need overrides:

{
  packageOverrides = pkgs: {
    sbcl = pkgs.sbcl_2_2_4;
    mytest = pkgs.sbcl_2_2_4;
  };
}

Note that this only works for old style nix-* tools, not for flakes, i.e. not for nix * (not the space versus the dash).

Update: someone fixed this, it’s not necessary anymore.

Deep override system dependency input using flakes

Pin in a single package

Example:

# I don’t want to use an overlays = here, because that won’t take effect if I
# actually get passed a pkgs. The question is if that’s a good or a bad thing?
{ pkgs ? import <nixpkgs> {} }:

let overlay = (self: super: {
      mytest = super.sbcl_2_1_10;
    });
    # The docs advise against this. But it does work.
    x = pkgs.extend overlay;
in

# I don’t know how to make let do a same-level shadowing bind of a name, so I
# just nest it 🤷‍♀️ one day I’ll probably learn how to do this. Today is not that
# day.
let pkgs = x;
in

with pkgs;

stdenv.mkDerivation {
pname = "foobar";
version = "0.0.1";
src = ./.;
buildDependencies = [ mytest ];
buildPhase = ''
    printf '#!/usr/bin/env bash\n\necho Version "%s": "$@"\n' "$(${mytest}/bin/sbcl --version)" > foobar
    chmod +x foobar
'';
installPhase = ''
    mkdir -p "$out/bin"
    mv foobar "$out/bin/"
'';
dontStrip = true;
}

Scopes

I’m not entirely clear on what they are, yet, but every collection of packages (Haskell, Ruby, Qt, …) seems to be part of a so-called “scope”.

Cf overrideScope', pkgs.lib.makeScope and pkgs.newScope.

Non-free / Unfree packages

Definitely.

{
  allowUnfree = true;
}

The Store

Nix stores all files in the “nix store”. This is the plumbing, like how git’s underlying data model is its object database.

The Nix store is an object store with an important feature: one of the valid types of elements it can store is a directory of files. In practice, the key ends almost always up being the hash of the derivation (I think?). This is used for memoization of builds. It is used to store the source, which is built into a result, and that result itself is also stored as a separate entry in the store.

Hashes

The store is content addressable, meaning the key for lookup is the hash of the contents. The algorithm is described in detail here:

https://comono.id/posts/2020-03-20-how-nix-instantiation-works/

You can use Nix just fine without ever understanding this (just like the git object store) but it’s nice to peek under the hood.

Caching

Nix builds are heavily cached. Locally using the nix store, remotely using some remote caching system. Excellent demonstration here:

https://scrive.github.io/nix-workshop/06-infrastructure/01-caching-nix.html

In particular, the linked fibonacci nix explains local caching:

# Run with nix-build -v -E 'import ./fib.nix "foo" 4'
let
  nixpkgs = import <nixpkgs> { };

  inherit (nixpkgs) stdenv;

  prefixed-fib = prefix:
    let fib = n:
      assert builtins.isInt n;
      assert n >= 0;
      let
        n-str = builtins.toString n;
      in
        if n == 0 || n == 1
        then
          stdenv.mkDerivation {
            name = "${prefix}-fib-${n-str}";
            unpackPhase = "true";

            buildPhase = ''
              echo "Producing base case fib(${n-str})..."
              sleep 3
              echo "The answer to fib(${n-str}) is ${n-str}"
            '';

            installPhase = ''
              mkdir -p $out
              echo "${n-str}" > $out/answer
            '';
          }
        else
          let
            fib-1 = fib (n - 1);
            fib-2 = fib (n - 2);

            n-1-str = builtins.toString (n - 1);
            n-2-str = builtins.toString (n - 2);
          in
          stdenv.mkDerivation {
            name = "${prefix}-fib-${n-str}";
            unpackPhase = "true";

            buildPhase = ''
              fib_1=$(cat ${fib-1}/answer)
              fib_2=$(cat ${fib-2}/answer)

              echo "Calculating the answer of fib(${n-str}).."
              echo "Given fib(${n-1-str}) = $fib_1,"
              echo "and given fib(${n-2-str}) = $fib_2.."

              sleep 3

              answer=$(( $fib_1 + $fib_2 ))
              echo "The answer to fib(${n-str}) is $answer"
            '';

            installPhase = ''
              mkdir -p $out
              echo "$answer" > $out/answer
            '';
          }
    ;
  in fib;
in
prefixed-fib

UPDATE: Actually, when I run that with 50 instead of 5, it does explode! It doesn’t get to caching quickly enough and somewhere in the planning of jobs it creates a 50! large graph.

Fetching (builtins vs fetchers)

https://discourse.nixos.org/t/why-is-fetchtarball-not-mentioned-in-chapter-11-fetchers-of-the-nixpkgs-manual/15319

There are two families of fetchers: builtins (implemented in Nix source in C++) and fetchers, which are Nix fetchers implemented in Nix, in nixpkgs.stdenv. You want to use the latter. Crucially:

The builtins (e.g. builtins.fetchurl) are defined in Nix source code. One difference is that the built-in functions cause download during evaluation, which is bad for all sorts of reasons.

Very important difference!

Builtins & pkgs.lib

This is where the utils live

https://teu5us.github.io/nix-lib.html

String context

A unique feature of the Nix language that I haven’t seen anywhere else.

Strings have a hidden property, “context”, which is a set of derivations. Meaning a string in Nix can have hidden references to other nix packages. These sets grow on concatenation, so if you have one string referencing { A }, another referencing { B }, and you concat them (or any of their substrings), you get a new string with context { A, B }.

This neat feature allows Nix to automatically infer the dependencies for a derivation by looking at the union of all derivations in all of the strings in its own derivation. This is why when you have for example:

buildPhase = ''
  ${pkgs.gcc}/bin/gcc -o foo foo.c
''

You don’t need to explicitly specify GCC as a build dependency: it becomes a member of the context of the buildPhase, and through that is found as a dependency of the entire derivation.

Neat!

This is also why they discourage using:

buildPhase = ''
  gcc -o foo foo.c
''

Even though that technically works, Nix now doesn’t know that you depend on the GCC derivation.

Example:

nix-repl> :p builtins.getContext "${p.sbcl} and ${p.scala}"
{
  "/nix/store/3pw34cakm1vx6mc8zhik6vrmgg3rs2ni-sbcl-2.2.9.drv" = { outputs = [ "out" ]; };
  "/nix/store/qsahyw0h47f9nkcby97ganq7w59aw1i3-scala-2.13.10.drv" = { outputs = [ "out" ]; };
}

This is the first time I learn something about Nix that answers more questions than it raises.

Derivations

This is a hard one to pin down. I have not found a satisfying explanation. And I’m not sure if it’s actually a Nix-the-language feature, or a stdlib thing.

Simple demo of difference between evaluating a derivation, and building it:

Let’s evaluate a derivation:

nix-repl> p.writeText "test" "hey"
«derivation /nix/store/y5mprkgdmiqjzwly9qwjhz9n196jy93x-test.drv»

nix-repl> (p.writeText "test" "hey").outPath
"/nix/store/66i0ggdxdk73in27dsmyqn0n696041nd-test"

The derivation has been built:

cat /nix/store/y5mprkgdmiqjzwly9qwjhz9n196jy93x-test.drv
Derive([("out","/nix/store/66i0ggdxdk73in27dsmyqn0n696041nd-test","","")],[("/nix/store/8b6s8cgrp0m25fnfqq8qrx54d91v8vgg-stdenv-darwin.drv",["out"]),("/nix/store/nb5r5f4lj8cy0di6894zn483976mxni4-bash-5.1-p16.drv",["out"])],["/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"],"x86_64-darwin","/nix/store/dx09hl4rkg0smsk216qnh033rzx2y04y-bash-5.1-p16/bin/bash",["-e","/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"],[("__darwinAllowLocalNetworking",""),("__impureHostDeps","/bin/sh /usr/lib/libSystem.B.dylib /usr/lib/system/libunc.dylib /dev/zero /dev/random /dev/urandom /bin/sh"),("__propagatedImpureHostDeps",""),("__propagatedSandboxProfile",""),("__sandboxProfile",""),("allowSubstitutes",""),("buildCommand","target=$out''\nmkdir -p \"$(dirname \"$target\")\"\n\nif [ -e \"$textPath\" ]; then\n  mv \"$textPath\" \"$target\"\nelse\n  echo -n \"$text\" > \"$target\"\nfi\n\neval \"$checkPhase\"\n\n(test -n \"$executable\" && chmod +x \"$target\") || true\n"),("buildInputs",""),("builder","/nix/store/dx09hl4rkg0smsk216qnh033rzx2y04y-bash-5.1-p16/bin/bash"),("checkPhase",""),("cmakeFlags",""),("configureFlags",""),("depsBuildBuild",""),("depsBuildBuildPropagated",""),("depsBuildTarget",""),("depsBuildTargetPropagated",""),("depsHostHost",""),("depsHostHostPropagated",""),("depsTargetTarget",""),("depsTargetTargetPropagated",""),("doCheck",""),("doInstallCheck",""),("enableParallelBuilding","1"),("enableParallelChecking","1"),("executable",""),("mesonFlags",""),("name","test"),("nativeBuildInputs",""),("out","/nix/store/66i0ggdxdk73in27dsmyqn0n696041nd-test"),("outputs","out"),("passAsFile","buildCommand text"),("patches",""),("preferLocalBuild","1"),("propagatedBuildInputs",""),("propagatedNativeBuildInputs",""),("stdenv","/nix/store/phj9vavfvy45gadbc5d9fnirpb50mqw3-stdenv-darwin"),("strictDeps",""),("system","x86_64-darwin"),("text","hey")])

(Use nix show-derivation for saner output)

So, the derivation has been built, that means the package was built, and we should have the file, right?

cat /nix/store/66i0ggdxdk73in27dsmyqn0n696041nd-test
cat: /nix/store/66i0ggdxdk73in27dsmyqn0n696041nd-test: No such file or directory

Oh, we just evaluated the derivation, but we didn’t build it. A derivation is the final “build script”. If you see Nix as “yaml with functions”, then a derivation is the final ansible file you’d be left with after all functions have been executed. But that’s just the beginning of actually building the software.

Let’s use the REPL to do that:

nix-repl> :b (p.writeText "test" "hey")

This derivation produced the following outputs:
  out -> /nix/store/66i0ggdxdk73in27dsmyqn0n696041nd-te

And try again:

cat /nix/store/66i0ggdxdk73in27dsmyqn0n696041nd-test
hey

Sick.

buildInputs vs nativeBuildInputs

buildInputs:

programs and libraries used by the new derivation at run-time

nativeBuildInputs:

programs and libraries used at build-time that, if they are a compiler or similar tool, produce code to run at run-time—i.e. tools used to build the new derivation

buildInputs should persist at runtime. To take an example from discourse: if you build a wrapper tool around jq that automatically handles compression, using zlib, you’ll need both those tools available at runtime (and the zlib derivation at build time, at least for its headers). Nix does set up the envvars at build time for you, but because the output is just a binary, you can’t rely on Nix to set the PATH or LDLIBRARYPATH at runtime: binaries are binaries. You must ensure that, at buildtime, somehow, you hard-code the path to those tools. This is done behind the scenes using patchelf.

nativeBuildInputs is only for tools you use to do the building, e.g. GNU Make. These are “native” to the platform doing the building.

Nix could theoretically consider those nativeBuildInputs garbage collectable? I don’t think it does that, but it could. I think.

“build” vs “host” vs “platform”

Derivations expressed using stdenv have a build, host, and target platform. These attributes are, respectively, the platform on which a derivation is built, the platform on which it runs, and, for certain sorts of packages (e.g. a compilers), the platform which it targets.

When not cross-compiling, every derivation’s build, host, and target platforms are the same. However, when cross-compiling, they differ. For example, a package which is cross-compiled from x8664-linux to aarch64-linux, has (build, host, target) platforms at (x8664-linux, aarch64-linux, aarch64-linux).

https://github.com/NixOS/nixpkgs/issues/68967#issuecomment-532432617

Fixpoints aka self recursion aka overrides

Fixpoints are a way to solve “self recursion”, which allows you to implement functions which refer to their own output from within themselves. This is mind bending if you think of it in an imperative context, but it makes total sense if you see functions’ return values as “aspirational”, aka lazy. What made it click for me was seeing a function returning an attrset, as a function returning a big bag of getter-attributes.

Example: given this function:

myfunc = { bla }: {
  a = 5 + bla;
  b = do-a-thing bla;
}

Instead of seeing it as a function returning an attrset like that, I mentally see it like this:

myfunc = { bla }: {
  get a = () => 5 + bla;
  get b = () => do-a-thing bla;
}

This mental model helps me wrap my head around lazy evaluation, fixpoints, etc.

Now suddenly, it makes more sense you could get your own return value as an argument to yourself:

myfunc = { self, bla }: {
  get a = () => self.b + bla;
  get b = () => do-a-thing bla;
}

This makes sense because things are only evaluated on-demand. Even that self is only evaluated on-demand. So:

let res = myfunc { self = res; bla = 1234; }

This works because at this point, nothing is actually evaluated at all. It’s just a bunch of callbacks setup, in case anyone actually gets called. And even when you call self.b, it still doesn’t call self.a or self. Just that one callback.

I only really grokked this once I implemented an entire scope myself (lisp-packages-lite), ran into a problem (“how do I override a dependency and make all other packages in my scope automatically use that updated dependency?”) and needed a way to fix that problem. Overrides are the only way.

https://r6.ca/blog/20140422T142911Z.html

Setup hooks

This is very down & dirty, advanced, dirty stuff, and you don’t often need it or run into it at all.

Setup hooks are a magic handle for a dependency (e.g. libgz) to automatically run things in dependent derivations (e.g. tar, assuming it uses libgz). E.g.: setting its PATH or LDLIBRARYPATH or something (this doesn’t actually happen but it could).

I’m not intimately familiar with them yet but they seem a missing piece of a confusing puzzle for me, already.

Nix daemon and nix.conf

On MacOS (at least in multi-user mode but maybe also in single-user mode) nix builds are managed by a separate daemon process. This process gets its configuration from nix.conf.

But the nix user facing CLI tool itself also gets its configuration from nix.conf. This can be confusing: sometimes you want to change an option, so you change it in nix.conf, re-run your nix command, but it doesn’t seem to get picked up! E.g. enabling or disabling the sandbox. Why?

What happened is that your option was a nix daemon option (e.g. sandbox = true/false), but you didn’t restart the daemon, so the option hasn’t taken effect yet.

On Mac, you can do:

sudo launchctl stop org.nixos.nix-daemon
sudo launchctl start org.nixos.nix-daemon

Now the daemon will have picked up your new config.

Nixpkgs

A big master collection of common packages and utility code (‘nixpkgs.lib’). Flakes are an explicit move away from this nixpkgs system of defining everything in one big monorepo. Nixpkgs contains:

  • Many individually useful “leaf” packages like ‘tree’ or ‘ripgrep’
  • Package scopes, e.g. all Python packages, Node.JS, Common Lisp, etc
  • Nix utility functions (like lib.mergeAttrs etc)
  • Stdenv (in various forms): a “build environment” which is a big ball of bash mud to build derivations, its most famous entrypoint being ‘stdenv.mkDerivation’.

Don’t worry about the wisdom of putting this all in one giant monorepo. Do some breathing exercises instead.

CI/CD

Nixpkgs has powerful CI/CD tooling to help tame the beast.

  • nixpkgs-update aka @r-ryantm: a bot that creates PRs to keep packages in nixpkgs up to date
  • Hydra: a massive buildfarm that periodically builds all packages in master (and sometimes staging?) and uploads them to a binary cache, cache.nixos.org, after which the nixpkgs-unstable and nixos-unstable labels are updated.
  • borg: CI for PRs on the NixOS/nixpkgs repo which builds packages (and analyzes the contents for metadata, e.g. how many packages need rebuilding, in which arch, who are the maintainers, etc?)
  • https://github.com/NixOS/nixpkgs/pull/263065#issuecomment-1780397019

Commands

Nix does not mis any opportunity to confuse you. The ecosystem is split in two parts: channels and flakes.

This is reflected in the CLI (Command Line Interface; the stuff you type in the terminal):

  • Channels: Any command starting with nix- e.g. nix-build, nix-shell, nix-instantiate, nix-env.
  • Flakes: Any command starting with just nix e.g. nix build, nix develop

And of course why not immediately break that rule? Here are commands that applies to both:

  • nix-collect-garbage Because it operates on the nix store, which is used by both flakes and channel based Nix alike. They’re not that different, after all.
  • nix search You can use this to search in the <nixpkgs> channel, from which you can install using nix-env. [This isn’t the full story; see search].

Flakes

nix registry

I have no idea what this is but apparently I once configured this?

It’s used to.. set overlays using flakes or something? What the actual fuck.

https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-registry.html

You can use it to shadow <nixpkgs> to something else I think. In flakes.

UPDATE: I figured it out: these are the flake equivalents to channels. Run nix registry list to see your list. This is why you can do e.g. nix run nixpkgs#utm, and why you can use nixpkgs as an argument to your flake’s outputs function without specifying it as an explicit input.

If you don’t pin these registries they will be cached and stored for some time (1h). To avoid this, you can ‘pin’ a registry:

nix registry pin nixpkgs

This will pin your nixpkgs to the latest master. Benefit: no more unexpected waiting for nix commands to complete while it downloads a (probably unnecessary) update to nixpkgs. Drawback: you must remember to update your nixpkgs pin whenever you need it.

  • System registry

    By default, pins are local to a user. This is fine if you only have one user account on your system for yourself and you don’t want to affect others.

    If you have multiple user accounts for the same human being, e.g. one per project, you can use system pins to share a pin across all users. They are stored in /etc/nix/registry.json and can be updated the same way, by passing the registry location explicitly:

    sudo nix --extra-experimental-features "nix-command flakes" registry pin --registry /etc/nix/registry.json nixpkgs
    

    I’m not sure why I had to pass those feature flags to this command. Apparently my root user doesn’t share the same nix config as I do?

  • Sources

nix build --print-build-logs: full build log

The default of printing the last 10 lines on failure is never enough for me.

nix run ...

Run a one-off command from a flake.

  • nix run github:.. Run a flake directly from github
    nix run github:bennofs/nix-index#nix-locate -- -r 'bin/hello$'
    
    haskellPackages.hello.out                     1,028,832 x /nix/store/hdvjc2mv07q0cm92jl9kns1nw49qbqfm-hello-1.0.0.2/bin/hello
    mbedtls.out                                     101,584 x /nix/store/v9k27s13bqakldl6qim30vrqy224g2sh-mbedtls-3.3.0/bin/hello
    mbedtls_2.out                                    39,384 x /nix/store/cdrrcfrphz6kylmz0p8g0jkw3gxhm9pf-mbedtls-2.28.2/bin/hello
    hello.out                                        35,816 x /nix/store/nm8k29dzx8s3ym7nkhwwczgwp50lslsc-hello-2.12.1/bin/hello
    fltk.bin                                              0 s /nix/store/z6xifj7jl5g5szdpbzf9v2gdyh3mws49-fltk-1.3.8-bin/bin/hello
    fltk14.bin                                            0 s /nix/store/83f8yxn6rzmazi5ywn7q5xhb5kxcr908-fltk-1.4.x-2021-12-21-bin/bin/hello
    
    • Run a flake by its PR
      nix run github:NixOS/nixpkgs/pull/228826/head#sbcl
      

      Will run SBCL from PR #228826 on NixOS/nixpkgs.

      Like all nix commands that (indirectly) download tarballs, this has a default TTL of 1 hour. This means if someone changes the PR and you want to re-download it immediately, you’ll have to invalidate it:

      nix run --tarball-ttl 0 ...
      
  • nix run nixpkgs#.. Run a one-off command from the nixpkgs channel
    nix run nixpkgs#cowsay -- Holla
    
     _______ 
    < Holla >
     ------- 
            \   ^__^
             \  (oo)\_______
                (__)\       )
    

    \/\

    ||----w |
    ||     ||
    
  • nix run --expr ... Evaluate and run a Nix expression inline

    To evaluate and run a bespoke Nix expression all in one, you can use --expr. Imagine you’re on Mac, but the clisp package is marked as broken (its .meta.broken property is true). You want to test it anyway. You can do:

    nix run --impure --expr '(import <nixpkgs> {}).clisp.overrideAttrs (_: { meta.broken = false; })'
    

nix shell ... ephemeral interactive shell with a command in PATH

$ nix shell nixpkgs#hello
$ hello
Hello, world!

Useful if you want to play around with a command a bit, e.g. ffmpeg.

nix shell ... --command ... run a non-standard one-off command from a flake

With nix run only the main, blessed command for that flake can be run. If you want to run a different command, e.g. psql in the postgresql derivation, you can use nix shell:

nix shell nixpkgs#postgresql --command psql -V
psql (PostgreSQL) 14.6

Source: https://stackoverflow.com/a/73271062.

nix profile install ... Install a flake

Install a flake system wide.

E.g.:

nix profile install .

or

nix profile install github:hraban/emacs-nix

See also: installing a channel-based derivation.

Channel based

nix-env -i Install a package

Apparently you can use nix-env to install a package system-wide but I never do this. I don’t know if you pass -A or -p or why.

I always edit my ~/.nix-defexpr/desktop.nix and re-run nix-env -iA desktop. This is not ideal because it re-downloads tarballs if you’ve garbage collected them. I’m still looking for a better way.

Here’s how to install vim system wide:

$ nix-env -f '<nixpkgs>' -i -A vim

Update: I’ve started doing this because why not 🤷‍♀️ it’s easy.

See also: installing a flake.

nix-shell -p Temporary shell with a package on your PATH

Say you want to play around with ffmpeg without installing it system-wide:

outer $ nix-shell -p ffmpeg
inside nix-shell $ ffmpeg -whatever -these -insane -commands -are
inside nix-shell $ exit
outer $ # ffmpeg is now not available anymore

This even works for mac apps:

$ nix-shell -p iterm2
$ open $buildInputs/Applications/iTerm2.app
  • Behind the scenes

    Full disclosure: I’m not sure why exactly $buildInputs is set to the output path of the derivation you asked to have in the path of your shell. From what I understand, nix-shell is intended to give you the environment in which to build a derivation, not the actual output thereof. So what I think is happening here, with -p, is that you’re saying to nix-shell: “imagine a derivation with this package as its buildInput, now give me a shell where I can build it.” Of course it’s a trick because you’re not interested in building anything–you just want to run that app. Ha ha.

    Let’s test that hypothesis!

    $ nix-shell -p ripgrep iterm2 emacs
    [nix-shell:~]$ echo $buildInputs
    /nix/store/1h3ymrn6mai4y3z1gi13yl5alf15xixd-ripgrep-13.0.0 /nix/store/fdyzbibl32mgjn5dkq0cv87qy39zqw4w-iterm2-3.4.18 /nix/store/nmmagir82c5j52vmp12yl6h9319q3phi-emacs-28.2
    

    Ha!..? I was right. I think 😶.

    A telling peek behind the curtain appears in the error message when you pass it a fake package name:

    nix-shell -p ripgrep hallelujah plim plom
    
    error: undefined variable 'hallelujah'
    
           at «string»:1:117:
    
                1| {...}@args: with import <nixpkgs> args; (pkgs.runCommandCC or pkgs.runCommand) "shell" { buildInputs = [ (ripgrep) (hallelujah) (plim) (plom) ]; } ""
                 |                                                                                                                     ^
    (use '--show-trace' to show detailed location information)
    

    There you have it: nix-shell -p ... is a wrapper around a runCommand derivation, with the given package names taken from the <nixpkgs> channel.

nix-env --list-generations: List all global system snapshots

A “generation” is the state of all your globally installed system package versions at some point. If you run nix-env -u, it creates a new generation with the latest available version of all packages.

nix-env --uninstall

To uninstall a package

nix-shell -E ... build local project

To build the current directory and load it in a shell’s path:

nix-shell -E 'with import <nixpkgs> {} ; runCommand "dummy" {buildInputs = [( import ./. {}) ] ; } ""'

This goes against the spirit of nix-shell, which is intended for starting a shell with the build dependencies of your current project. You use it to debug the build of the project, not to use the project. To stay more in the spirit of Nix, you should register the project with nix-env (or something?) and load that using nix-shell -p.

But this is one hell of a useful little snippet if you just want to test something.

Corollary: this is how you load and run a github project:

nix-shell  -E 'with import <nixpkgs> {} ; runCommand "dummy" {buildInputs = [( import (fetchFromGitHub { owner = "hraban" ; repo = "git-tools" ; rev = "build/nix" ; sha256 = "sha256-yn7RPjAJjABxW1Wh05rcmN6zd4C6AAa6G0rDS3el37k";  }  ) {} ) ] ; } ""'

This is annoying because you must specify a valid SHA-256. Here’s a dirty way that uses an older, impure function:

nix-shell  -E 'with import <nixpkgs> {} ; runCommand "dummy" {buildInputs = [( import (builtins.fetchTarball "https://github.com/hraban/git-tools/archive/build/nix.tar.gz") {} ) ] ; } ""'

Or:

nix-shell  -E 'with import <nixpkgs> {} ; runCommand "dummy" {buildInputs = [( import (builtins.fetchGit { url = "https://github.com/hraban/git-tools.git"; ref="build/nix" ; rev="eb365fc3de42fea251a931c6d87754d355e81ec5";}) {} ) ] ; } ""'

nix-shell with project from local nixpkgs

Want to try out a package from the latest nixpkgs, e.g. staging, or your local modifications? Checkout nixpkgs locally and run this:

nix-shell -E 'with import ./. { } ; runCommand "dummy" { buildInputs = [  roswell ]  ; } "" '

(That example is for roswell)

nix-build --debug: Debug output from builds

Alternatively: nix-build -v or nix-build -vv

nix-build --arg pkgs ..: Use custom nixpkgs

Using --args you can set a custom value for an argument. Because nixpkgs is often an arg named pkgs with its default set to importing the official <nixpkgs>, you can override it using:

nix-build --arg pkgs "import /path/to/local/nixpkgs {}"

This allows testing a change to nixpkgs in your own app.

It relies on the convention that default.nix starts with the line:

{ pkgs ? import <nixpkgs> {} }:
...

Using nix-build --arg passes a value for that argument, which avoids the default <nixpkgs> import.

nix-build https://... build a nix derivation from a remote location

Fetch and build a Nix derivation:

nix-build https://github.com/bennofs/nix-index/archive/master.tar.gz
./result/bin/nix-index --help | head -3
/nix/store/q36m3zp2183fd2wb4a5rbrqqj3lf7k9g-nix-index-0.1.6
Builds an index for nix-locate

Usage: nix-index [OPTIONS]

default.nix and nix-build

The simplest way to specify a simple default build script for your project is to create a file called default.nix that evaluates to a single function which accepts an attrset as argument and returns either a derivation, or an attrset of derivations.

Convention is for the argument to contain at least one element called pkgs, which defaults to <nixpkgs>.

  • Single top-level derivation

    The most common form:

    { pkgs ? import <nixpkgs> {} }:
    
    pkgs.stdenv.mkDerivation {
      name = "test-1";
      version = "0.0.1";
      unpackPhase = "true";
      installPhase = ''
        mkdir -p "$out"
        echo hello > "$out"/world
      '';
    }
    

    Allows you to use a simple nix-build call without any args:

    $ nix-build
    /nix/store/axsq0mgv6ymnvgi20lwz1qn9a97757zl-test-1
    $ tree result/
    result/
    └── world
    
    0 directories, 1 file
    
  • Multiple top-level derivations
    { pkgs ? import <nixpkgs> {} }:
    
    rec {
      deriv1 = pkgs.stdenv.mkDerivation {
        name = "test-1";
        version = "0.0.1";
        unpackPhase = "true";
        installPhase = ''
          mkdir -p "$out"
          echo hello > "$out/world"
          echo bye > "$out/all"
        '';
      };
      deriv2 = pkgs.stdenv.mkDerivation {
        name = "test-2";
        version = "0.0.1";
        src = deriv1;
        installPhase = ''
          mkdir -p "$out"
          cp -r "$src/world" "$out"
          echo deriv2 was here > "$out/extra"
        '';
      };
    }
    

    Now you can specify a derivation to build:

    $ nix-build . -A deriv1
    /nix/store/vr9jaxpqwkghqa2p3w68w5qqx6b31c82-test-1
    $ tree result/
    result/
    ├── all
    └── world
    
    0 directories, 2 files
    $ nix-build . -A deriv2
    /nix/store/hwmm1mszhh3k288mkqs5b2lszcbxhcpz-test-2
    $ tree result/
    result/
    ├── extra
    └── world
    
    0 directories, 2 files
    

    A single nix-build call builds all derivations and puts the results in result, result-2, ..., result-N:

    mbp23-2303:foo user$ nix-build
    /nix/store/a6jmjsmyv6f4455ds27xfm8pdq4vv9x6-test-1
    /nix/store/7flh7xl5amabx40nsh5svypzvvdjny6s-test-2
    mbp23-2303:foo user$ ls -l result*
    lrwxr-xr-x@ 1 user  wheel  50 Aug  4 19:28 result -> /nix/store/a6jmjsmyv6f4455ds27xfm8pdq4vv9x6-test-1
    lrwxr-xr-x@ 1 user  wheel  50 Aug  4 19:28 result-2 -> /nix/store/7flh7xl5amabx40nsh5svypzvvdjny6s-test-2
    mbp23-2303:foo user$ tree result*
    result
    ├── all
    └── world
    result-2
    ├── extra
    └── world
    
    2 directories, 4 files
    

Misc

nixpkgs.lib.attrNames: Get attribute names of a set

Here’s how I used the REPL to figure out “given an attribute set, how do I get an array of its keys?”

> nixpkgs = import <nixpkgs> { }
> nixpkgs.<tab>
# nothing interesting
> nixpkgs.stdenv.<tab>
# nothing relevant
# Googled “nix get keys of set”, found https://github.com/NixOS/nixpkgs/blob/master/lib/attrsets.nix
> nixpkgs.lib<tab>
> nixpkgs.lib.attr<tab>
> nixpkgs.lib.attrNames { a = 3 ; }
[ "a" ]

nix show-derivation /nix/store/yadayada.drv

To pretty print the contents of a .drv file.

https://nixos.org/guides/nix-pills/our-first-derivation.html

nix-collect-garbage -d Clean up everything

The -d flag also deletes all your old profiles.

See https://nixos.org/manual/nix/stable/package-management/garbage-collection.html.

You can also list generations to see what would be deleted.

Get the out path of a derivation

  • nix-store -q --outpus /....drv

    When you have a raw derivation file available, you can use nix-store -r:

    tree $(nix-store -q --outputs $(nix-instantiate '<nixpkgs>' -A hello))
    
    /nix/store/7hrvp0zcg5kbrw9p98asbhii8kpc6kpb-hello-2.12.1
    ├── bin
    │   └── hello
    └── share
        ├── info
        │   └── hello.info
        └── man
            └── man1
                └── hello.1.gz
    
    5 directories, 3 files
    

    This is mostly useful for local projects without flakes, where you can use $(nix-instantiate .).

  • nix path-info ...

    The flake-aware version is useful when you have a flake-based project:

    tree $(nix path-info nixpkgs#hello)
    
    /nix/store/qh8s6yb93b1wl0jxg5i0pfq2rf7qm44j-hello-2.12.1
    ├── bin
    │   └── hello
    └── share
        ├── info
        │   └── hello.info
        └── man
            └── man1
                └── hello.1.gz
    
    5 directories, 3 files
    

    And while we’re on the topic, if you ever need a reference to the derivation itself (the flake equivalent of nix-instantiate), that’s:

    nix path-info --derivation nixpkgs#hello
    
    /nix/store/f040m7v5zbgy2z5h833p9iqxqcka0zs2-hello-2.12.1.drv
    

    From: https://github.com/NixOS/nix/issues/3908#issuecomment-1186633661

nix-store --gc --print-roots

List all “entrypoints” into the nix store.

nix-store --delete: Delete a single package

E.g. I want to force a rebuild of cl-async-base:

$ nix-build . -A lispPackages_new.sbclPackages.cl-async-base
$ path="$(readlink result)"
$ rm result
$ nix-store --delete "$path"

Optionally removing any other roots which still point to it.

(There’s a better way of getting the store path for that package rather than rebuilding it locally and reading the link, but I can’t think of it right now.)

nix-store --query --deriver /nix/store/...: Find the derivation for a file in store

Looking at a binary or package like

/nix/store/h20fjqggj7mlzjd9rnbq3rvnszqfd8a8-cl-async-base-20211020-git/src/util/package.lisp

And wondering what the original derivation was that yielded this output?

nix-store --query --deriver /nix/store/h20fjqggj7mlzjd9rnbq3rvnszqfd8a8-cl-async-base-20211020-git
/nix/store/kjffy7qd5s3a6x09v6379sd3yp1s1m61-cl-async-base-20211020-git.drv

Combine with nix show-derivation to view the contents:

nix show-derivation $drv
{
  "/nix/store/kjffy7qd5s3a6x09v6379sd3yp1s1m61-cl-async-base-20211020-git.drv": {
    "outputs": {
      "out": {
        "path": "/nix/store/h20fjqggj7mlzjd9rnbq3rvnszqfd8a8-cl-async-base-20211020-git"
      }
    },
    "inputSrcs": [
      "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
    ],
    "inputDrvs": {
      "/nix/store/8445q1qfjgppdm5i5s06f62h50ybwghg-cffi-0.24.1.drv": [
        "out"
      ],
      "/nix/store/8yr0m1vg7p9rlgiic32jsfp0glj8prn9-sbcl-2.2.9.drv": [
        "out"
      ],
      "/nix/store/aklq242pl7qmfabwdrwhds17mir8c8fh-trivial-features-20211209-git.drv": [
        "out"
      ],
      "/nix/store/ihcl9092hinqpzxi7kanyiqc8b6k9h9g-cffi-grovel-0.24.1.drv": [
        "out"
      ],
      "/nix/store/ixpa3jbfzlg51nwcywcdk0iaa1ig1ymy-build-cl-async-base.lisp.drv": [
        "out"
      ],
      "/nix/store/p5wmm0397cfyxmnd0fkl8f3kgl58vz66-libuv-1.44.2.drv": [
        "out"
      ],
      "/nix/store/qfdan6wif20jjyh1mg8049j0mla43crf-cl-libuv-20200610-git.drv": [
        "out"
      ],
      "/nix/store/qg1rh2g8i7aidwm37hx1wjg5zwinfc2j-source.drv": [
        "out"
      ],
      "/nix/store/rvqvbjw19rvl0paihmgwp1b1dzh9aapj-cffi-toolchain-0.24.1.drv": [
        "out"
      ],
      "/nix/store/v6z9qni64gf9khkw1im6zjr8k77fa0a8-babel-20200925-git.drv": [
        "out"
      ],
      "/nix/store/z3xd1ixq280bb08dqy422p3l1m797v5f-stdenv-darwin.drv": [
        "out"
      ],
      "/nix/store/z4s4n80dk5kj317iqdxhd6m2lxbzqr3d-bash-5.1-p16.drv": [
        "out"
      ],
      "/nix/store/z4w04y7y0a8vrd65jji60x8m611xid8d-bordeaux-threads-v0.8.8.drv": [
        "out"
      ],
      "/nix/store/ziyr9iz8zws4dc4ar5j4sqaib54w5iac-alexandria-20220707-git.drv": [
        "out"
      ]
    },
    "system": "x86_64-darwin",
    "builder": "/nix/store/n63wlghd6cczckb9ymkgxz2pahlfvspk-bash-5.1-p16/bin/bash",
    "args": [
      "-e",
      "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
    ],
    "env": {
      "CLASSPATH": "",
      "CL_SOURCE_REGISTRY": "/nix/store/r05yhrga1qqca57gxmy9p04i5nqw5nrs-alexandria-20220707-git//:/nix/store/jnhjgcrcafy0nra887kky44agz4ijfjk-babel-20200925-git//:/nix/store/b5d7hdnwyq40qr5rmwpblk1c1y8yzf9m-bordeaux-threads-v0.8.8//:/nix/store/n42cnxxs9vp46r9gnpnfy82by82hz7vm-cffi-0.24.1//:/nix/store/xhcg5nx60zfhdv1mygcm92r3vsvihv9j-cffi-grovel-0.24.1//:/nix/store/fdcyvs68vy97wr8asdg466mv9xbqn9cc-cffi-toolchain-0.24.1//:/nix/store/xx0faplsbqgwckss1y45smlwzqrkdpas-cl-libuv-20200610-git//:/nix/store/izv04sl4rf5xp46m771dbrb1r6lh4zkr-trivial-features-20211209-git//",
      "LD_LIBRARY_PATH": "/nix/store/m4zkynz61h1arp97yrwfyr0a1ip3l7h2-libuv-1.44.2/lib",
      "__darwinAllowLocalNetworking": "",
      "__impureHostDeps": "/bin/sh /usr/lib/libSystem.B.dylib /usr/lib/system/libunc.dylib /dev/zero /dev/random /dev/urandom /bin/sh",
      "__propagatedImpureHostDeps": "",
      "__propagatedSandboxProfile": "",
      "__sandboxProfile": "",
      "asds": "cl-async-base",
      "buildInputs": "",
      "buildPhase": "# In addition to lisp dependencies, make asdf see the .asd's\n# of the systems being built\n#\n# *Append* src since `lispLibs` can provide .asd's that are\n# also in `src` but are not in `systems` (that is, the .asd's\n# that will be deleted in installPhase). We don't want to\n# rebuild them, but to load them from lispLibs.\n#\n# NOTE: It's important to read files from `src` instead of\n# from pwd to get go-to-definition working with SLIME\nexport CL_SOURCE_REGISTRY=$CL_SOURCE_REGISTRY:/nix/store/39m5l7imwhvdjgvwk10mxqzmmdp2b35h-source//\n\n# Similiarily for native deps\nexport LD_LIBRARY_PATH=:$LD_LIBRARY_PATH\nexport CLASSPATH=:$CLASSPATH\n\n# Make asdf compile from `src` to pwd and load `lispLibs`\n# from storeDir. Otherwise it could try to recompile lisp deps.\nexport ASDF_OUTPUT_TRANSLATIONS=\"/nix/store/39m5l7imwhvdjgvwk10mxqzmmdp2b35h-source:$(pwd):/nix/store:/nix/store\"\n\n# track lisp dependencies for graph generation\n# TODO: Do the propagation like for lisp, native and java like this:\n# https://github.com/teu5us/nix-lisp-overlay/blob/e30dafafa5c1b9a5b0ccc9aaaef9d285d9f0c46b/pkgs/development/lisp-modules/setup-hook.sh\n# Then remove the \"echo >> nix-drvs\" from buildScript\necho $lispLibs >> __nix-drvs\n\n\n# Finally, compile the systems\n/nix/store/wyw4hrcam44zj8gblpd6c18r3y5i24g1-sbcl-2.2.9/bin/sbcl --script $buildScript\n",
      "buildScript": "/nix/store/4h8bv8v2xdgp914cqz7clyi14hj8f0dg-build-cl-async-base.lisp",
      "builder": "/nix/store/n63wlghd6cczckb9ymkgxz2pahlfvspk-bash-5.1-p16/bin/bash",
      "cmakeFlags": "",
      "configureFlags": "",
      "depsBuildBuild": "",
      "depsBuildBuildPropagated": "",
      "depsBuildTarget": "",
      "depsBuildTargetPropagated": "",
      "depsHostHost": "",
      "depsHostHostPropagated": "",
      "depsTargetTarget": "",
      "depsTargetTargetPropagated": "",
      "doCheck": "",
      "doInstallCheck": "",
      "dontFixup": "1",
      "dontStrip": "1",
      "dontUnpack": "",
      "installPhase": "mkdir -pv $out\ncp -r * $out\n\n# Remove all .asd files except for those in `systems`.\nfind $out -name \"*.asd\" \\\n| grep -v \"/\\(cl-async-base\\)\\.asd$\" \\\n| xargs rm -fv || true\n",
      "javaLibs": "",
      "lisp": "/nix/store/wyw4hrcam44zj8gblpd6c18r3y5i24g1-sbcl-2.2.9/bin/sbcl --script",
      "lispLibs": "/nix/store/b5d7hdnwyq40qr5rmwpblk1c1y8yzf9m-bordeaux-threads-v0.8.8 /nix/store/n42cnxxs9vp46r9gnpnfy82by82hz7vm-cffi-0.24.1 /nix/store/xx0faplsbqgwckss1y45smlwzqrkdpas-cl-libuv-20200610-git",
      "mesonFlags": "",
      "name": "cl-async-base-20211020-git",
      "nativeBuildInputs": "",
      "nativeLibs": "",
      "out": "/nix/store/h20fjqggj7mlzjd9rnbq3rvnszqfd8a8-cl-async-base-20211020-git",
      "outputs": "out",
      "patches": "",
      "pname": "cl-async-base",
      "propagatedBuildInputs": "",
      "propagatedNativeBuildInputs": "",
      "src": "/nix/store/39m5l7imwhvdjgvwk10mxqzmmdp2b35h-source",
      "stdenv": "/nix/store/90xbrkay3mil301qnbfkn2yg3df0y0gn-stdenv-darwin",
      "strictDeps": "",
      "system": "x86_64-darwin",
      "systems": "cl-async-base",
      "version": "20211020-git"
    }
  }
}

nix-store -q --references ...: List dependencies

nix-store -q --references /nix/store/px1r6r4abnyc6lq8a9q7x03pnkg2zsn9-hello-2.12.1
/nix/store/1xscydv49868c2vkv93q388b66ij7lcs-swift-corefoundation-unstable-2018-09-14

You can also run this in a nix project directory to get all its dependencies:

nix-store -q --references $(nix-instantiate default.nix)
/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh
/nix/store/nb5r5f4lj8cy0di6894zn483976mxni4-bash-5.1-p16.drv
/nix/store/8lc2p7j9fg2hwpyfv4a04csk92pyzb5n-stdenv-darwin.drv
/nix/store/3pw34cakm1vx6mc8zhik6vrmgg3rs2ni-sbcl-2.2.9.drv
/nix/store/8ay3sqrkpkzqb8sj4gyvqfdy4jjh8adl-nix-lisppackages-new-demo
/nix/store/q6r6wgdb5bs7vadvvwa0n0b92ai3hma6-source.drv

Evalute raw Nix expression

This is a crazy hack thank god I found this guy:

nix-instantiate --read-write-mode --strict --eval -E '1 + 2'
3

Or a file:

echo '4 * 3' | nix-instantiate --read-write-mode --strict --eval -
12

http://www.chriswarbo.net/projects/nixos/scripting_with_nix.html

Legend.

Overriding properties

  • Remote derivation, using the new flake UI

    I have a derivation in a remote location, and I want my friend to test it. It lives in a subdirectory of a main repo, it’s not even a flake, I want him to fetch it, change a property on the derivation, then build that, and run it, all in one go:

    nix run --impure --expr 'let x = builtins.fetchTarball { url="https://github.com/hraban/cl-nix-lite/archive/refs/heads/master.tar.gz"; } ; y = import "${x}/examples/hello-binary" {} ; in y.overrideAttrs (_: { dontFixup = true; } )'
    

    Note that what I’m building isn’t remotely a flake, at all. I’m just using nix run, but with --expr, which just takes a raw derivation, no need for flakes.

    This should be enough, except that this derivation is so stubborn that it’s not compatible with nix run, because the derivation name differs from the final binary name:

    error: unable to execute '/nix/store/ainnsjsjvl7pdq4kl6a2260jcg460n3z-system-hello-binary/bin/system-hello-binary': No such file or directory
    

    To fix that:

    nix run --impure --expr 'let x = builtins.fetchTarball { url="https://github.com/hraban/cl-nix-lite/archive/refs/heads/master.tar.gz"; } ; y = import "${x}/examples/hello-binary" {} ; in y.overrideAttrs (_: { dontFixup = true; name = "hello-binary"; } )'
    
    Hello, world
    

    Note: This uses .overrideAttrs, which works on derivations only! This is not the same as .override, which works on functions. Derivations as plain old serializable objects, which are the result of a nix expression, which then can finally be built. This is not the same as overriding a function argument (using .override), which would change the way the entire derivation object itself is created. In this case it’s lucky that I only want to change an attribute on the derivation because that is always possible, rather than overriding a function argument, which only works if the function declares the right parameters for you to override in the first place.

    C.f. the channels equivalent.

  • Remote derivation, using the channels UI

    I have a derivation in a subdir of github:hraban/cl-nix-lite, and it doesn’t work on Jesse’s computer. I want him to try again, with the dontFixup flag set to true. To achieve this, he can execute:

    cd /tmp
    nix-build -E 'let x = builtins.fetchTarball { url="https://github.com/hraban/cl-nix-lite/archive/refs/heads/master.tar.gz"; } ; y = import "${x}/examples/hello-binary" {} ; in y.overrideAttrs (_: { dontFixup = true; } )' && ./result/bin/hello-binary
    

    C.f. the flake equivalent.

  • Overriding a function argument from a nixpkgs derivation

    Situation: I’m removing all the old SBCL compilers from Nixpkgs, but one derivation (CLPM) has pinned a (very) old version because the latest version breaks it. It pinned 2.0.8, we’re on 2.3.0 now, coming onto 2.3.2. I want to know what the latest SBCL version is with which CLPM still works. Let’s try the last one to confirm that it’s broken:

    nix run --impure --expr "let p = import <nixpkgs> {}; in p.clpm.override {sbcl_2_0_8 = p.sbcl; }"
    
    error: builder for '/nix/store/zxjjb03d1ghw6zdxhhn4f9q68qfhxaix-clpm-0.4.1.drv' failed with exit code 1;
           last 10 log lines:
           > 71: (SB-IMPL::TOPLEVEL-INIT)
           > 72: ((FLET SB-UNIX::BODY :IN SB-IMPL::START-LISP))
           > 73: ((FLET "WITHOUT-INTERRUPTS-BODY-3" :IN SB-IMPL::START-LISP))
           > 74: (SB-IMPL::%START-LISP)
           >
           > unhandled condition in --disable-debugger mode, quitting
           > ;
           > ; compilation unit aborted
           > ;   caught 2 fatal ERROR conditions
           > ;   caught 18 WARNING conditions
           For full logs, run 'nix log /nix/store/zxjjb03d1ghw6zdxhhn4f9q68qfhxaix-clpm-0.4.1.drv'.
    

    Ok, that breaks. (To see the full log I can run it again with the --print-build-logs command.)

    Note: this uses nix run, but I don’t actually use any flakes here.

    I now search my way down1 to the breaking version by passing specific versions:

    nix run --impure --expr "let p = import <nixpkgs> {}; in p.clpm.override {sbcl_2_0_8 = p.sbcl_2_2_11; }"
    

    Etc.

    Note: this uses .override, which is for functions! The CLPM derivation accepts SBCL as a function argument, which is (usually) automatically passed by the calling environment using callPackage. .override, on any function defined to allow overrides, allows you to override these arguments and evaluate the function again (an advantage of lazy evaluation). This is not the same as .overrideAttrs.

Nix code / nixlang / Nix the language

pkgs.writeText: Write a text file

pkgs.lib.fakeSha256: Stub SHA-256

UPDATE: Bollocks, just use the empty string.

Use this if you don’t know what the real hash is so at least the derivation compiles, and nix-build will fail and tell you the right hash.

This is only necessary because the length of a checksum string is checked at evaluation time, so you can’t just say e.g. sha256 = "foo"; that will not evaluate.

{}.foo or "bar": get default attr

Equivalent of Python’s {}.get("foo", "bar")

{}.foo or 3
3
{ foo = 15; }.foo or 3
15

{} ? a: hasAttr

Test the presence of a key in an attrset:

{} ? a
false
{a = false;} ? a
true

Using a name contained in a variable:

let x = "foo"; in { foo = 3; } ? ${x}
true

mkDerivation { passthru = ...}

Every key of the attrset you pass to mkDerivation will be stored in the final derivation. Derivations only support primitive types, no attrsets, functions, etc. They’re also strictly evaluated.

In order to store a “regular” object in a derivation you can use the passthru key:

mkDerivation {
  name = "yada";
  otherargs = "blabla";

  passthru = {
    here = i: can: "store anything, ${i} ${can}";
  };
}

The passthru key won’t be available to anyone importing the derivation from the store (e.g. using import).

with rec { ... }: ; vs let ... in ...

let .. in is syntactic sugar for with rec { } :, with one subtle difference around function argument shadowing (which I can’t recall right now).

Useful debugging trick:

let
  x = "hidden stuff you want to inspect";
  y = "more hidden vars you want to access and evaluate for debugging";
in
  mkDerivation {
    name = "how-do-I-get-access-to-those-hidden-vars";
  }

Try this:

let root = rec {
  x = "hidden stuff you want to inspect";
  y = "more hidden vars you want to access and evaluate for debugging";
}; in
  with root;
  mkDerivation {
    name = "how-do-I-get-access-to-those-hidden-vars";
    passthru = { inherit root; };
  }

Now you can access the entire let binding (because really it’s just a single attrset) from the REPL.

pkgs.lib.cleanSource(With): like .nixignore

You can filter files from a directory, essentially by creating a new derivation.

pkgs.lib.cleanSource

A pre-fab straightforward “does mostly what you want” way to do this.

Always use this instead of src = ./.:

src = pkgs.lib.cleanSource ./.;

It removes common sources of noise unrelated to your app source code, including the result symlink created by nix-build itself.

While it does remove e.g. the .git folder, it does not remove files that are marked as ignore by your .gitignore file.

pkgs.lib.cleanSourceWith

A customizable version of pkgs.lib.cleanSource. It takes a filter function which is asked about every single file in your source, and when it returns true, the file is kept.

In fact, pkgs.lib.cleanSource is implemented using pkgs.lib.cleanSourceWith.

Example:

lib.cleanSourceWith {
  filter = path: type: type != "directory" || baseNameOf path != ".svn";
  src = ./.;
}

to filter out the .svn directory.

  • Difference with builtins.filterSource

    The same can be achieved with builtins.filterSource, except:

    • cleanSourceWith recognizes when it is nested, and only ends up creating one final derivation instead of multiple intermediate ones. (To be perfectly honest with you, who cares? It’s a cosmetic advantage, but sure, fine.)
    • filterSource doesn’t take a sha256 argument so in order to be used in pure eval mode, it must itself be wrapped in a builtins.path. (This seems like the more poignant advantage, to me)

Filtering absolute paths vs relative paths

Paths passed to these filter functions are absolute which is never what you want. All solutions online propose using baseNameOf which is a hack. A StackOverflow thread covers the real solution:

files = [
    ./Cargo.toml
    ./Cargo.lock
    ./nested/file.rs
];
src = builtins.filterSource (p: t: builtins.elem (/. + p) files) ./.;

Two important elements:

  • Don’t use relative paths in your own whitelist, but absolute paths (Nix notation ./foo is resolved to a path, which is always absolute)
  • Your filter function gets a string as incoming argument. You must cast it to a path if you want to compare it.

{ ${null} = ...; }: Elide attrset entry

{ ${null} = 3; }
{ }

Useful in conditionals:

{ ${if 1 == 2 then "foo" else null} = 3; }
{ }

Similar:

let pkgs = (import <nixpkgs> {});
in
  pkgs.lib.attrsets.optionalAttrs (1 == 2) { foo = 3; }
{ }

Example: isSubset

# is inner a subset of outer?
isSubset = inner: outer: (builtins.intersectAttrs inner outer) == inner

Synthetic derivation (without source)

Set parameter dontUnpack = true; on pkgs.stdenv.mkDerivation:

pkgs.stdenv.mkDerivation {
  name = "foo";
  dontUnpack = true;
  installPhase = ''
    mkdir -p $out/bin
    echo echo foo > $out/bin/foo
    chmod +x $out/bin/foo
  '';
}

builtins.trace: “printf debugging”

Put this in front of any code you want to debug by seeing its value:

Before:

let
  my-func = x: (x * x);
in
  ...

After:

let
  my-func = x: builtins.trace "Got x: ${x}" (x * x);
in
  ...

Cookbook

Real-world problems I encountered, and their solutions

How to debug stdenv setup

Problem

I’m trying to understand addToEnvHooks, and the larger stdenv derivation setup script in general. I would like to see a trace of the setup file as it gets executed so I can see variables’ values as it goes.

Solution

The real top-level entrypoint for building any derivation using stdenv.mkDerivation is a builder. You can figure out what the builder is on a derivation using the following properties:

nix show-derivation $(nix-instantiate '<nixpkgs>' -A sbcl )

The result will contain some of these fields:

{
  "/nix/store/zn31didn16smk7llxpwxbgifwyz2yi9h-sbcl-2.3.2.drv": {
    "args": [
      "-e",
      "/nix/store/6xg259477c90a229xwmb53pdfkn6ig3g-default-builder.sh"
    ],
    "builder": "/nix/store/yhzzjlywz0c2m89h5pi5d77k2vlqhfim-bash-5.2-p15/bin/bash",
    "env": {
      ...

What I’m looking for is builder and args. In this example I learn that SBCL 2.3.2 from nixpkgs is built by running:

/nix/store/yhzzjlywz0c2m89h5pi5d77k2vlqhfim-bash-5.2-p15/bin/bash \
  -e \
  /nix/store/6xg259477c90a229xwmb53pdfkn6ig3g-default-builder.sh

This is the default builder script, which is itself simple:

cat /nix/store/6xg259477c90a229xwmb53pdfkn6ig3g-default-builder.sh
if [ -f .attrs.sh ]; then
    . .attrs.sh
fi

source $stdenv/setup
genericBuild

The meat of the operation is that $stdenv/setup file. Most of it can be found in the nixpkgs repo’s copy, although the actual file loaded during build will have a few extra os-specific lines added.

Now I read that file, it has all of the code that sets and calls hooks, etc. But it’s complicated and I want to see a trace. Luckily there’s a NIX_DEBUG variable that I can set, and it is read at the top. It is read before any hooks are loaded, including preHook, so this won’t work in my derivation:

stdenv.mkDerivation {
  preHook = "export NIX_DEBUG=6";
  ...
}
  • Solution 1: Just export an envvar

    The cleanest way to do this is to rely on a built-in mechanism offered by stdenv.mkDerivation to set environment variables:

    stdenv.mkDerivation {
      env.NIX_DEBUG = 6;
      ...
    }
    

    Or set a property on the derivation:

    stdenv.mkDerivation {
      NIX_DEBUG = 6;
      ...
    }
    

    Which is technically a little bit different, but it doesn’t matter much. Technically, this sets a property NIX_DEBUG to the value 6 on the derivation. As a convenience, stdenv.mkDerivation will export all(-ish) properties as envvars, so it amounts to the same(ish) thing.

  • Solution 2: Change the build script

    For completeness, here’s another solution: change the build script directly:

    stdenv.mkDerivation {
      builder = pkgs.writeText "custom-build" ''
    export NIX_DEBUG=6
    
    if [ -f .attrs.sh ]; then
        . .attrs.sh
    fi
    
    source $stdenv/setup
    genericBuild
    '';
    
      ...
    }
    

Conclusion

The output was far too verbose to learn anything useful about addEnvHooks, but at least I learned about NIX_DEBUG and changing the builder script. 🤷‍♀️

Is this derivation available on a binary cache?

Problem

I have a binary cache, https://cl-nix-lite.cachix.org, and I want to programatically check if available from that cache or not. By derivation in this case I mean that I’m in a directory with a default.nix and I want to know if running nix-build would build stuff, or if it will just download a prebuilt binary.

Solution

For channels (default.nix):

curl --fail -sL -o /dev/null --head --fail https://cl-nix-lite.cachix.org/$(nix show-derivation $(nix-instantiate ) | jq -r '.[].outputs.out.path' | sed -e 's,/nix/store/,,' -e 's,-.*,,').narinfo

For flakes:

curl --fail -sL -o /dev/null --head --fail https://cl-nix-lite.cachix.org/$(nix show-derivation . | jq -r '.[].outputs.out.path' | sed -e 's,/nix/store/,,' -e 's,-.*,,').narinfo

NixOS

My configuration for NixOS. Store in /etc/nixos/configuration.nix. This is for a gandi.net VPS.

Configuration

{pkgs, ...}:

{
  imports = [
    # On Gandi:
    # <nixpkgs/nixos/modules/virtualisation/openstack-config.nix>
    # ./gandicloud.nix

    # On a VM from a nixos iso:
    ./hardware-configuration.nix
  ];

  nix.settings.experimental-features = [ "nix-command" "flakes" ];

  environment = {
    systemPackages = with pkgs; [
      emacs
      file
      htop
      iotop
      lsof
      moreutils
      mosh
      netcat
      pstree
      pv
      rsync
      tmux
      traceroute
      tree
      wget
    ];
    variables.EDITOR = "emacs";
  };
  system = {
    autoUpgrade = {
      enable = true;
      allowReboot = true;
    };
  };
  programs.mosh.enable = true;

  users.users.asdf = {
    isNormalUser = true;
    description = "Main user";
    extraGroups = ["networkmanager" "wheel"];
    openssh.authorizedKeys.keys = [ "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF0+BDB4kr/I+kAyoWTVRtzGqpRzBTTjVAPD9cYCujUy" ];
    packages = with pkgs; [ ];
  };

  # Bootloader.
  boot.loader.systemd-boot.enable = true; boot.loader.efi.canTouchEfiVariables = true;
  boot.loader.efi.efiSysMountPoint = "/boot/efi";

  systemd.services.foo = {
    script = ''
    echo "Doing some stuff"
    (date ; echo hello bye bye) >> /tmp/hello-bye
  '';
    wantedBy = [ "multi-user.target" ];
  };


  # Set your time zone.
  time.timeZone = "America/New_York";

  # Select internationalisation properties.
  i18n.defaultLocale = "en_US.UTF-8";

  i18n.extraLocaleSettings = {
    LC_ADDRESS = "en_US.UTF-8";
    LC_IDENTIFICATION = "en_US.UTF-8";
    LC_MEASUREMENT = "en_US.UTF-8";
    LC_MONETARY = "en_US.UTF-8";
    LC_NAME = "en_US.UTF-8";
    LC_NUMERIC = "en_US.UTF-8";
    LC_PAPER = "en_US.UTF-8";
    LC_TELEPHONE = "en_US.UTF-8";
    LC_TIME = "en_US.UTF-8";
  };

  services.openssh.enable = true;
  # # Enable the X11 windowing system.
  # services.xserver.enable = true;

  # # Enable the KDE Plasma Desktop Environment.
  # services.xserver.displayManager.sddm.enable = true;
  # services.xserver.desktopManager.plasma5.enable = true;

  # # Configure keymap in X11
  # services.xserver = {
  #   layout = "us";
  #   xkbVariant = "alt-intl";
  # };

  # Configure console keymap
  console.keyMap = "us";

}

After saving, run nixos-rebuild switch (or use test first, check it works, then switch).

(setq-local org-confirm-babel-evaluate nil)
(setq explicit-shell-file-name "/run/current-system/sw/bin/bash")
date
echo hi
hostname

Memory

A 1GB VPS doesn’t have enough memory for nixos-rebuild switch. Nothing will appear to happen.

You can detect this by adding ; echo $? after the command: if it prints 137 you’re OOM. It should output 0.

Add a swap file, e.g. in /root/swap, then run the command again.

Upgrade packages

nixos-rebuild -v --upgrade switch; echo $?

I’ve added ; echo $? here because the failure mode is otherwise silent, and it happened to me on a minimal size VM from gandi. See Configuration.

Upgrade entire OS version

NixOS is pinned to a release by setting the nixos channel. See which one:

nix-channel --list

If you want to upgrade to a newer version, or in this case the “always bleeding edge latest”:

nix-channel --add https://nixos.org/channels/nixos-unstable nixos

And run this to upgrade your system:

nixos-rebuild switch --upgrade


Or leave a comment below: