nix/doc/manual/source/advanced-topics/distributed-builds.md

Remote Builds

A local Nix installation can forward Nix builds to other machines, this allows multiple builds to be performed in parallel.

Remote builds also allow Nix to perform multi-platform builds in a semi-transparent way. For example, if you perform a build for a x86_64-darwin on an i686-linux machine, Nix can automatically forward the build to a x86_64-darwin machine, if one is available.

Requirements

For a local machine to forward a build to a remote machine, the remote machine must:

• Have Nix installed
• Be running an SSH server, e.g. sshd
• Be accessible via SSH from the local machine over the network
• Have the local machine's public SSH key in /etc/ssh/authorized_keys.d/<username>
• Have the username of the SSH user in the trusted-users setting in nix.conf

Testing

To test connecting to a remote Nix instance (in this case mac), run:

nix store info --store ssh://username@mac

To specify an SSH identity file as part of the remote store URI add a query parameter, e.g.

nix store info --store ssh://username@mac?ssh-key=/home/alice/my-key

Since builds should be non-interactive, the key should not have a passphrase. Alternatively, you can load identities ahead of time into ssh-agent or gpg-agent.

In a multi-user installation (default), builds are executed by the Nix Daemon. The Nix Daemon cannot prompt for a passphrase via the terminal or ssh-agent, so the SSH key must not have a passphrase.

In addition, the Nix Daemon's user (typically root) needs to have SSH access to the remote builder.

Access can be verified by running sudo su, and then validating SSH access, e.g. by running ssh mac. SSH identity files for root users are usually stored in /root/.ssh/ (Linux) or /var/root/.ssh (MacOS).

If you get the error

bash: nix: command not found
error: cannot connect to 'mac'

then you need to ensure that the PATH of non-interactive login shells contains Nix.

The list of remote build machines can be specified on the command line or in the Nix configuration file. For example, the following command allows you to build a derivation for x86_64-darwin on a Linux machine:

uname

Linux

nix build --impure \
 --expr '(with import <nixpkgs> { system = "x86_64-darwin"; }; runCommand "foo" {} "uname > $out")' \
 --builders 'ssh://mac x86_64-darwin'

[1/0/1 built, 0.0 MiB DL] building foo on ssh://mac

cat ./result

Darwin

It is possible to specify multiple build machines separated by a semicolon or a newline, e.g.

  --builders 'ssh://mac x86_64-darwin ; ssh://beastie x86_64-freebsd'

Remote build machines can also be configured in nix.conf, e.g.

builders = ssh://mac x86_64-darwin ; ssh://beastie x86_64-freebsd

After making changes to nix.conf, restart the Nix daemon for changes to take effect.

Finally, remote build machines can be configured in a separate configuration file included in builders via the syntax @/path/to/file. For example,

builders = @/etc/nix/machines

causes the list of machines in /etc/nix/machines to be included. (This is the default.)

Resource Management

Adding resource-management to the experimental-features setting in nix.conf enables a basic resource management scheme for system features. This is akin to what can be accomplished with job schedulers like Slurm, where a remote machine can have a limited quantity of a resource that can be temporarily "consumed" by a job. This can be used with memory-heavy builds, or derivations that require exclusive access to particular hardware resources.

Resource management is supported in both the supported features and mandatory features of a remote machine configuration, by appending a colon : to a feature name followed by the quantity that this machine has. This is tracked on a per-store basis, so different users on a multi-user installation share the same pool of resources for their remote build machines. A derivation specifies that it consumes a resource with the same notation in the requiredSystemFeatures attribute.

For example, this builder can provide exclusive access to two GPUs and 128G of memory for remote builds:

builders = ssh://gpu-node x86_64-linux - 32 1 gpu:2,mem:128

A derivation that might use this machine may set its requiredSystemFeatures to ["gpu:1" "mem:4"] to indicate that it requires a GPU and consumes 4G of system memory. A particularly memory-heavy derivation that doesn't need a GPU may still use the machine with a value of ["mem:64"]. This helps ensure that limited system resources are not over-consumed by remote builds. Note that Nix does not do any actual delegation or enforcement of GPU, memory, or other resource usage, that is up to the derivations to manage.

When configuring the system-features setting on the remote machine's nix.conf, only include the name of the consumable feature, not the quantity availble. Resource limits are tracked on the dispatching end within the local store.