treewide: Prevent IFD by default

Import-from-derivation (IFD) has problematic performance, and is disabled in
Nixpkgs by policy. It is arguably good practice for libraries to avoid
it whenever possible, as it has poor ergonomics in some cases,
especially with dry builds, as it requires multiple eval+build phases.

As such, prevent its use in Home Manager by default by putting existing
tests that use IFD behind a config. In CI, run a first pass with IFD
disabled, skipping tests without the config. Then run a second pass with
IFD enabled and including tests with the config. This second pass will
also run tests without the config, but they should be cached from the
previous run, so the cost is not double (only eval time should be paid
twice). It’s necessary to change from using NMT’s `run` to `build` as
`run` itself uses IFD.

Of the tests that have the config:

- kitty/theme-to-themeFile: this is a test for deprecated config, and so
  should be removed eventually anyway
- podman: the implementation relies on IFD to create individual systemd
  units from the derivation output, and so it is not straightforward to
  remove the IFD; doing so would require rethinking how the module works
  to instead have the systemd unit files included as-is rather than as
  individually configured units in the Nix config.

.github/workflows/test.yml

@@ -26,6 +26,11 @@ jobs:
     - run: ./format --ci
     - run: nix-shell --show-trace . -A install
     - run: yes | home-manager -I home-manager=. uninstall
-    - run: nix-shell -j auto --show-trace --arg enableBig false --pure tests -A run.all
+    - name: Run tests
+      run: nix-build -j auto --show-trace --arg enableBig false --pure --option allow-import-from-derivation false tests -A build.all
+      env:
+        GC_INITIAL_HEAP_SIZE: 4294967296
+    - name: Run tests (with IFD)
+      run: nix-build -j auto --show-trace --arg enableBig false --pure --arg enableLegacyIfd true tests -A build.all
       env:
         GC_INITIAL_HEAP_SIZE: 4294967296

docs/manual/contributing/tests.md

@@ -13,20 +13,20 @@ functions available in test scripts, you can look at NMT's
 The full Home Manager test suite can be run by executing
 
 ``` shell
-$ nix-shell --pure tests -A run.all
+$ nix-build --pure --option allow-import-from-derivation false tests -A build.all
 ```
 
 in the project root. List all test cases through
 
 ``` shell
-$ nix-shell --pure tests -A list
+$ nix-build --pure tests --option allow-import-from-derivation false -A list
 ```
 
 and run an individual test, for example `alacritty-empty-settings`,
 through
 
 ``` shell
-$ nix-shell --pure tests -A run.alacritty-empty-settings
+$ nix-build --pure tests --option allow-import-from-derivation false -A build.alacritty-empty-settings
 ```
 
 However, those invocations will impurely source the system's Nixpkgs,
@@ -34,11 +34,17 @@ and may cause failures. To run against the Nixpkgs from the `flake.lock` file,
 use instead e.g.
 
 ``` shell
-$ nix build --reference-lock-file flake.lock ./tests#test-all
+$ nix build --reference-lock-file flake.lock --option allow-import-from-derivation false ./tests#test-all
 ```
 
 or
 
 ``` shell
-$ nix build --reference-lock-file flake.lock ./tests#test-alacritty-empty-settings
+$ nix build --reference-lock-file flake.lock --option allow-import-from-derivation false ./tests#test-alacritty-empty-settings
+```
+
+Some tests may be marked with `enableLegacyIfd`, those may be run by run with e.g.
+
+``` shell
+$ nix-build --pure tests --arg enableLegacyIfd true -A build.mytest
 ```

tests/big-test.nix

@@ -10,4 +10,14 @@
       packages or tests that take long to run.
     '';
   };
+
+  options.test.enableLegacyIfd = lib.mkOption {
+    type = lib.types.bool;
+    default = false;
+    description = ''
+      Whether to enable tests that use import-from-derivation (IFD). Use of IFD
+      in Home Manager is deprecated, and this option should not be used for new
+      tests.
+    '';
+  };
 }

tests/modules/programs/kitty/theme-to-themeFile.nix

@@ -1,11 +1,12 @@
 {
+  config,
   lib,
   options,
   realPkgs,
   ...
 }:
 
-{
+lib.mkIf config.test.enableLegacyIfd {
   programs.kitty = {
     enable = true;
     theme = "Space Gray Eighties";

tests/modules/services/podman-linux/build.nix

@@ -1,6 +1,11 @@
-{ pkgs, ... }:
-
 {
+  config,
+  lib,
+  pkgs,
+  ...
+}:
+
+lib.mkIf config.test.enableLegacyIfd {
   imports = [ ./podman-stubs.nix ];
 
   services.podman = {

tests/modules/services/podman-linux/container.nix

@@ -1,4 +1,6 @@
-{
+{ config, lib, ... }:
+
+lib.mkIf config.test.enableLegacyIfd {
   imports = [ ./podman-stubs.nix ];
 
   services.podman = {

tests/modules/services/podman-linux/image.nix

@@ -1,4 +1,6 @@
-{
+{ config, lib, ... }:
+
+lib.mkIf config.test.enableLegacyIfd {
   imports = [ ./podman-stubs.nix ];
 
   services.podman = {

tests/modules/services/podman-linux/integration.nix

@@ -1,6 +1,11 @@
-{ pkgs, ... }:
-
 {
+  config,
+  lib,
+  pkgs,
+  ...
+}:
+
+lib.mkIf config.test.enableLegacyIfd {
   imports = [ ./podman-stubs.nix ];
 
   services.podman = {

tests/modules/services/podman-linux/manifest.nix

@@ -1,4 +1,6 @@
-{
+{ config, lib, ... }:
+
+lib.mkIf config.test.enableLegacyIfd {
   imports = [ ./podman-stubs.nix ];
 
   services.podman = {

tests/modules/services/podman-linux/network.nix

@@ -1,4 +1,6 @@
-{
+{ config, lib, ... }:
+
+lib.mkIf config.test.enableLegacyIfd {
   imports = [ ./podman-stubs.nix ];
 
   services.podman = {

tests/modules/services/podman-linux/volume.nix

@@ -1,4 +1,6 @@
-{
+{ config, lib, ... }:
+
+lib.mkIf config.test.enableLegacyIfd {
   imports = [ ./podman-stubs.nix ];
 
   services.podman = {