- The user-specific list of [subscribed channels](#subscribed-channels)
- The [downloaded channel contents](#channels)
-- The [Nix expression search path](@docroot@/command-ref/conf-file.md#conf-nix-path), set with the [`-I` option](#opt-i) or the [`NIX_PATH` environment variable](#env-NIX_PATH)
+- The [Nix expression search path](@docroot@/command-ref/conf-file.md#conf-nix-path), set with the [`-I` option](#opt-I) or the [`NIX_PATH` environment variable](#env-NIX_PATH)
> **Note**
>
diff --git a/doc/manual/source/command-ref/nix-env/upgrade.md b/doc/manual/source/command-ref/nix-env/upgrade.md
index 2779363c3..bf4c1a8ed 100644
--- a/doc/manual/source/command-ref/nix-env/upgrade.md
+++ b/doc/manual/source/command-ref/nix-env/upgrade.md
@@ -22,7 +22,7 @@ left untouched; this is not an error. It is also not an error if an
element of *args* matches no installed derivations.
For a description of how *args* is mapped to a set of store paths, see
-[`--install`](#operation---install). If *args* describes multiple
+[`--install`](./install.md). If *args* describes multiple
store paths with the same symbolic name, only the one with the highest
version is installed.
diff --git a/doc/manual/source/command-ref/nix-shell.md b/doc/manual/source/command-ref/nix-shell.md
index f2e2e3593..307f1934a 100644
--- a/doc/manual/source/command-ref/nix-shell.md
+++ b/doc/manual/source/command-ref/nix-shell.md
@@ -19,7 +19,7 @@
This man page describes the command `nix-shell`, which is distinct from `nix
shell`. For documentation on the latter, run `nix shell --help` or see `man
-nix3-shell`.
+nix3-env-shell`.
# Description
diff --git a/doc/manual/source/command-ref/nix-store/gc.md b/doc/manual/source/command-ref/nix-store/gc.md
index f432e00eb..8ec59d906 100644
--- a/doc/manual/source/command-ref/nix-store/gc.md
+++ b/doc/manual/source/command-ref/nix-store/gc.md
@@ -48,8 +48,7 @@ The behaviour of the collector is also influenced by the
configuration file.
By default, the collector prints the total number of freed bytes when it
-finishes (or when it is interrupted). With `--print-dead`, it prints the
-number of bytes that would be freed.
+finishes (or when it is interrupted).
{{#include ./opt-common.md}}
diff --git a/doc/manual/source/development/building.md b/doc/manual/source/development/building.md
index 889d81d80..eb65a7247 100644
--- a/doc/manual/source/development/building.md
+++ b/doc/manual/source/development/building.md
@@ -66,7 +66,7 @@ You can also build Nix for one of the [supported platforms](#platforms).
This section assumes you are using Nix with the [`flakes`] and [`nix-command`] experimental features enabled.
[`flakes`]: @docroot@/development/experimental-features.md#xp-feature-flakes
-[`nix-command`]: @docroot@/development/experimental-features.md#xp-nix-command
+[`nix-command`]: @docroot@/development/experimental-features.md#xp-feature-nix-command
To build all dependencies and start a shell in which all environment variables are set up so that those dependencies can be found:
@@ -256,7 +256,7 @@ You can use any of the other supported environments in place of `nix-cli-ccacheS
## Editor integration
The `clangd` LSP server is installed by default on the `clang`-based `devShell`s.
-See [supported compilation environments](#compilation-environments) and instructions how to set up a shell [with flakes](#nix-with-flakes) or in [classic Nix](#classic-nix).
+See [supported compilation environments](#compilation-environments) and instructions how to set up a shell [with flakes](#building-nix-with-flakes) or in [classic Nix](#building-nix).
To use the LSP with your editor, you will want a `compile_commands.json` file telling `clangd` how we are compiling the code.
Meson's configure always produces this inside the build directory.
diff --git a/doc/manual/source/development/documentation.md b/doc/manual/source/development/documentation.md
index a2a54175d..dd40ef481 100644
--- a/doc/manual/source/development/documentation.md
+++ b/doc/manual/source/development/documentation.md
@@ -240,3 +240,9 @@ $ configurePhase
$ ninja src/external-api-docs/html
$ xdg-open src/external-api-docs/html/index.html
```
+
+If you use direnv, or otherwise want to run `configurePhase` in a transient shell, use:
+
+```bash
+nix-shell -A devShells.x86_64-linux.native-clangStdenv --command 'appendToVar mesonFlags "-Ddoc-gen=true"; mesonConfigurePhase'
+```
diff --git a/doc/manual/source/development/testing.md b/doc/manual/source/development/testing.md
index c0b130155..7c2cbbb5d 100644
--- a/doc/manual/source/development/testing.md
+++ b/doc/manual/source/development/testing.md
@@ -119,7 +119,7 @@ This will:
3. Stop the program when the test fails, allowing the user to then issue arbitrary commands to GDB.
-### Characterisation testing { #characaterisation-testing-unit }
+### Characterisation testing { #characterisation-testing-unit }
See [functional characterisation testing](#characterisation-testing-functional) for a broader discussion of characterisation testing.
diff --git a/doc/manual/source/glossary.md b/doc/manual/source/glossary.md
index e6a294e7d..502e6d4de 100644
--- a/doc/manual/source/glossary.md
+++ b/doc/manual/source/glossary.md
@@ -208,7 +208,7 @@
- [impure derivation]{#gloss-impure-derivation}
- [An experimental feature](#@docroot@/development/experimental-features.md#xp-feature-impure-derivations) that allows derivations to be explicitly marked as impure,
+ [An experimental feature](@docroot@/development/experimental-features.md#xp-feature-impure-derivations) that allows derivations to be explicitly marked as impure,
so that they are always rebuilt, and their outputs not reused by subsequent calls to realise them.
- [Nix database]{#gloss-nix-database}
@@ -279,7 +279,7 @@
See [References](@docroot@/store/store-object.md#references) for details.
-- [referrer]{#gloss-reference}
+- [referrer]{#gloss-referrer}
A reversed edge from one [store object] to another.
@@ -367,8 +367,8 @@
Nix represents files as [file system objects][file system object], and how they belong together is encoded as [references][reference] between [store objects][store object] that contain these file system objects.
- The [Nix language] allows denoting packages in terms of [attribute sets](@docroot@/language/types.md#attribute-set) containing:
- - attributes that refer to the files of a package, typically in the form of [derivation outputs](#output),
+ The [Nix language] allows denoting packages in terms of [attribute sets](@docroot@/language/types.md#type-attrs) containing:
+ - attributes that refer to the files of a package, typically in the form of [derivation outputs](#gloss-output),
- attributes with metadata, such as information about how the package is supposed to be used.
The exact shape of these attribute sets is up to convention.
@@ -383,7 +383,7 @@
[string]: ./language/types.md#type-string
[path]: ./language/types.md#type-path
- [attribute name]: ./language/types.md#attribute-set
+ [attribute name]: ./language/types.md#type-attrs
- [base directory]{#gloss-base-directory}
diff --git a/doc/manual/source/installation/installing-docker.md b/doc/manual/source/installation/installing-docker.md
index 9354c1a72..ccc75be5a 100644
--- a/doc/manual/source/installation/installing-docker.md
+++ b/doc/manual/source/installation/installing-docker.md
@@ -3,19 +3,21 @@
To run the latest stable release of Nix with Docker run the following command:
```console
-$ docker run -ti ghcr.io/nixos/nix
-Unable to find image 'ghcr.io/nixos/nix:latest' locally
-latest: Pulling from ghcr.io/nixos/nix
+$ docker run -ti docker.io/nixos/nix
+Unable to find image 'docker.io/nixos/nix:latest' locally
+latest: Pulling from docker.io/nixos/nix
5843afab3874: Pull complete
b52bf13f109c: Pull complete
1e2415612aa3: Pull complete
Digest: sha256:27f6e7f60227e959ee7ece361f75d4844a40e1cc6878b6868fe30140420031ff
-Status: Downloaded newer image for ghcr.io/nixos/nix:latest
+Status: Downloaded newer image for docker.io/nixos/nix:latest
35ca4ada6e96:/# nix --version
nix (Nix) 2.3.12
35ca4ada6e96:/# exit
```
+> If you want the latest pre-release you can use ghcr.io/nixos/nix and view them at https://github.com/nixos/nix/pkgs/container/nix
+
# What is included in Nix's Docker image?
The official Docker image is created using `pkgs.dockerTools.buildLayeredImage`
diff --git a/doc/manual/source/language/advanced-attributes.md b/doc/manual/source/language/advanced-attributes.md
index c9d64f060..f0b1a4c73 100644
--- a/doc/manual/source/language/advanced-attributes.md
+++ b/doc/manual/source/language/advanced-attributes.md
@@ -333,7 +333,7 @@ Here is more information on the `output*` attributes, and what values they may b
`outputHashAlgo` can only be `null` when `outputHash` follows the SRI format, because in that case the choice of hash algorithm is determined by `outputHash`.
- - [`outputHash`]{#adv-attr-outputHashAlgo}; [`outputHash`]{#adv-attr-outputHashMode}
+ - [`outputHash`]{#adv-attr-outputHash}
This will specify the output hash of the single output of a [fixed-output derivation].
diff --git a/doc/manual/source/language/derivations.md b/doc/manual/source/language/derivations.md
index 43eec680b..2403183fc 100644
--- a/doc/manual/source/language/derivations.md
+++ b/doc/manual/source/language/derivations.md
@@ -16,7 +16,7 @@ It outputs an attribute set, and produces a [store derivation] as a side effect
- [`name`]{#attr-name} ([String](@docroot@/language/types.md#type-string))
A symbolic name for the derivation.
- See [derivation outputs](@docroot@/store/derivation/index.md#outputs) for what this is affects.
+ See [derivation outputs](@docroot@/store/derivation/outputs/index.md#outputs) for what this is affects.
[store path]: @docroot@/store/store-path.md
diff --git a/doc/manual/source/language/identifiers.md b/doc/manual/source/language/identifiers.md
index 584a2f861..67bb1eeec 100644
--- a/doc/manual/source/language/identifiers.md
+++ b/doc/manual/source/language/identifiers.md
@@ -16,7 +16,7 @@ An *identifier* is an [ASCII](https://en.wikipedia.org/wiki/ASCII) character seq
# Names
-A *name* can be written as an [identifier](#identifier) or a [string literal](./string-literals.md).
+A *name* can be written as an [identifier](#identifiers) or a [string literal](./string-literals.md).
> **Syntax**
>
diff --git a/doc/manual/source/language/index.md b/doc/manual/source/language/index.md
index 1eb14e96d..116f928dc 100644
--- a/doc/manual/source/language/index.md
+++ b/doc/manual/source/language/index.md
@@ -137,7 +137,7 @@ This is an incomplete overview of language features, by example.
|
- [Booleans](@docroot@/language/types.md#type-boolean)
+ [Booleans](@docroot@/language/types.md#type-bool)
|
@@ -245,7 +245,7 @@ This is an incomplete overview of language features, by example.
- An [attribute set](@docroot@/language/types.md#attribute-set) with attributes named `x` and `y`
+ An [attribute set](@docroot@/language/types.md#type-attrs) with attributes named `x` and `y`
|
@@ -285,7 +285,7 @@ This is an incomplete overview of language features, by example.
- [Lists](@docroot@/language/types.md#list) with three elements.
+ [Lists](@docroot@/language/types.md#type-list) with three elements.
|
@@ -369,7 +369,7 @@ This is an incomplete overview of language features, by example.
- [Attribute selection](@docroot@/language/types.md#attribute-set) (evaluates to `1`)
+ [Attribute selection](@docroot@/language/types.md#type-attrs) (evaluates to `1`)
|
@@ -381,7 +381,7 @@ This is an incomplete overview of language features, by example.
- [Attribute selection](@docroot@/language/types.md#attribute-set) with default (evaluates to `3`)
+ [Attribute selection](@docroot@/language/types.md#type-attrs) with default (evaluates to `3`)
|
diff --git a/doc/manual/source/language/string-context.md b/doc/manual/source/language/string-context.md
index 0d8fcdefa..65c59d865 100644
--- a/doc/manual/source/language/string-context.md
+++ b/doc/manual/source/language/string-context.md
@@ -111,7 +111,7 @@ It creates an [attribute set] representing the string context, which can be insp
[`builtins.hasContext`]: ./builtins.md#builtins-hasContext
[`builtins.getContext`]: ./builtins.md#builtins-getContext
-[attribute set]: ./types.md#attribute-set
+[attribute set]: ./types.md#type-attrs
## Clearing string contexts
diff --git a/doc/manual/source/language/string-interpolation.md b/doc/manual/source/language/string-interpolation.md
index a503d5f04..8e25d2b63 100644
--- a/doc/manual/source/language/string-interpolation.md
+++ b/doc/manual/source/language/string-interpolation.md
@@ -6,7 +6,7 @@ Such a construct is called *interpolated string*, and the expression inside is a
[string]: ./types.md#type-string
[path]: ./types.md#type-path
-[attribute set]: ./types.md#attribute-set
+[attribute set]: ./types.md#type-attrs
> **Syntax**
>
diff --git a/doc/manual/source/language/syntax.md b/doc/manual/source/language/syntax.md
index 85162db74..b127aca14 100644
--- a/doc/manual/source/language/syntax.md
+++ b/doc/manual/source/language/syntax.md
@@ -51,7 +51,7 @@ See [String literals](string-literals.md).
Path literals can also include [string interpolation], besides being [interpolated into other expressions].
- [interpolated into other expressions]: ./string-interpolation.md#interpolated-expressions
+ [interpolated into other expressions]: ./string-interpolation.md#interpolated-expression
At least one slash (`/`) must appear *before* any interpolated expression for the result to be recognized as a path.
@@ -235,7 +235,7 @@ of object-oriented programming, for example.
## Recursive sets
-Recursive sets are like normal [attribute sets](./types.md#attribute-set), but the attributes can refer to each other.
+Recursive sets are like normal [attribute sets](./types.md#type-attrs), but the attributes can refer to each other.
> *rec-attrset* = `rec {` [ *name* `=` *expr* `;` `]`... `}`
@@ -287,7 +287,7 @@ This evaluates to `"foobar"`.
## Inheriting attributes
-When defining an [attribute set](./types.md#attribute-set) or in a [let-expression](#let-expressions) it is often convenient to copy variables from the surrounding lexical scope (e.g., when you want to propagate attributes).
+When defining an [attribute set](./types.md#type-attrs) or in a [let-expression](#let-expressions) it is often convenient to copy variables from the surrounding lexical scope (e.g., when you want to propagate attributes).
This can be shortened using the `inherit` keyword.
Example:
diff --git a/doc/manual/source/meson.build b/doc/manual/source/meson.build
index 949d26526..294d57ad9 100644
--- a/doc/manual/source/meson.build
+++ b/doc/manual/source/meson.build
@@ -1,3 +1,6 @@
+# Process JSON schema documentation
+subdir('protocols')
+
summary_rl_next = custom_target(
command : [
bash,
diff --git a/doc/manual/source/protocols/derivation-aterm.md b/doc/manual/source/protocols/derivation-aterm.md
index 99e3c2be6..523678e66 100644
--- a/doc/manual/source/protocols/derivation-aterm.md
+++ b/doc/manual/source/protocols/derivation-aterm.md
@@ -1,6 +1,8 @@
# Derivation "ATerm" file format
-For historical reasons, [store derivations][store derivation] are stored on-disk in [ATerm](https://homepages.cwi.nl/~daybuild/daily-books/technology/aterm-guide/aterm-guide.html) format.
+For historical reasons, [store derivations][store derivation] are stored on-disk in "Annotated Term" (ATerm) format
+([guide](https://homepages.cwi.nl/~daybuild/daily-books/technology/aterm-guide/aterm-guide.html),
+[paper](https://doi.org/10.1002/(SICI)1097-024X(200003)30:3%3C259::AID-SPE298%3E3.0.CO;2-Y)).
## The ATerm format used
diff --git a/doc/manual/source/protocols/json/content-address.md b/doc/manual/source/protocols/json/content-address.md
new file mode 100644
index 000000000..2284e30aa
--- /dev/null
+++ b/doc/manual/source/protocols/json/content-address.md
@@ -0,0 +1,21 @@
+{{#include content-address-v1-fixed.md}}
+
+## Examples
+
+### [Text](@docroot@/store/store-object/content-address.html#method-text) method
+
+```json
+{{#include schema/content-address-v1/text.json}}
+```
+
+### [Nix Archive](@docroot@/store/store-object/content-address.html#method-nix-archive) method
+
+```json
+{{#include schema/content-address-v1/nar.json}}
+```
+
+
diff --git a/doc/manual/source/protocols/json/derivation.md b/doc/manual/source/protocols/json/derivation.md
index cc9389f7c..a4a4ea79d 100644
--- a/doc/manual/source/protocols/json/derivation.md
+++ b/doc/manual/source/protocols/json/derivation.md
@@ -1,120 +1,7 @@
-# Derivation JSON Format
+{{#include derivation-v3-fixed.md}}
-> **Warning**
->
-> This JSON format is currently
-> [**experimental**](@docroot@/development/experimental-features.md#xp-feature-nix-command)
-> and subject to change.
+
diff --git a/doc/manual/source/protocols/json/deriving-path.md b/doc/manual/source/protocols/json/deriving-path.md
new file mode 100644
index 000000000..9851b371d
--- /dev/null
+++ b/doc/manual/source/protocols/json/deriving-path.md
@@ -0,0 +1,21 @@
+{{#include deriving-path-v1-fixed.md}}
+
+## Examples
+
+### Constant
+
+```json
+{{#include schema/deriving-path-v1/single_opaque.json}}
+```
+
+### Output of static derivation
+
+```json
+{{#include schema/deriving-path-v1/single_built.json}}
+```
+
+### Output of dynamic derivation
+
+```json
+{{#include schema/deriving-path-v1/single_built_built.json}}
+```
diff --git a/doc/manual/source/protocols/json/fixup-json-schema-generated-doc.sed b/doc/manual/source/protocols/json/fixup-json-schema-generated-doc.sed
new file mode 100644
index 000000000..27895d42a
--- /dev/null
+++ b/doc/manual/source/protocols/json/fixup-json-schema-generated-doc.sed
@@ -0,0 +1,17 @@
+# For some reason, backticks in the JSON schema are being escaped rather
+# than being kept as intentional code spans. This removes all backtick
+# escaping, which is an ugly solution, but one that is fine, because we
+# are not using backticks for any other purpose.
+s/\\`/`/g
+
+# The way that semi-external references are rendered (i.e. ones to
+# sibling schema files, as opposed to separate website ones, is not nice
+# for humans. Replace it with a nice relative link within the manual
+# instead.
+#
+# As we have more such relative links, more replacements of this nature
+# should appear below.
+s^\(./hash-v1.yaml\)\?#/$defs/algorithm^[JSON format for `Hash`](./hash.html#algorithm)^g
+s^\(./hash-v1.yaml\)^[JSON format for `Hash`](./hash.html)^g
+s^\(./content-address-v1.yaml\)\?#/$defs/method^[JSON format for `ContentAddress`](./content-address.html#method)^g
+s^\(./content-address-v1.yaml\)^[JSON format for `ContentAddress`](./content-address.html)^g
diff --git a/doc/manual/source/protocols/json/hash.md b/doc/manual/source/protocols/json/hash.md
new file mode 100644
index 000000000..988c8466b
--- /dev/null
+++ b/doc/manual/source/protocols/json/hash.md
@@ -0,0 +1,33 @@
+{{#include hash-v1-fixed.md}}
+
+## Examples
+
+### SHA-256 with Base64 encoding
+
+```json
+{{#include schema/hash-v1/sha256-base64.json}}
+```
+
+### SHA-256 with Base16 (hexadecimal) encoding
+
+```json
+{{#include schema/hash-v1/sha256-base16.json}}
+```
+
+### SHA-256 with Nix32 encoding
+
+```json
+{{#include schema/hash-v1/sha256-nix32.json}}
+```
+
+### BLAKE3 with Base64 encoding
+
+```json
+{{#include schema/hash-v1/blake3-base64.json}}
+```
+
+
diff --git a/doc/manual/source/protocols/json/json-schema-for-humans-config.yaml b/doc/manual/source/protocols/json/json-schema-for-humans-config.yaml
new file mode 100644
index 000000000..cad098053
--- /dev/null
+++ b/doc/manual/source/protocols/json/json-schema-for-humans-config.yaml
@@ -0,0 +1,17 @@
+# Configuration file for json-schema-for-humans
+#
+# https://github.com/coveooss/json-schema-for-humans/blob/main/docs/examples/examples_md_default/Configuration.md
+
+template_name: md
+show_toc: true
+# impure timestamp and distracting
+with_footer: false
+recursive_detection_depth: 3
+show_breadcrumbs: false
+description_is_markdown: true
+template_md_options:
+ properties_table_columns:
+ - Property
+ - Type
+ - Pattern
+ - Title/Description
diff --git a/doc/manual/source/protocols/json/meson.build b/doc/manual/source/protocols/json/meson.build
new file mode 100644
index 000000000..7ebcff697
--- /dev/null
+++ b/doc/manual/source/protocols/json/meson.build
@@ -0,0 +1,78 @@
+# Tests in: ../../../../src/json-schema-checks
+
+fs = import('fs')
+
+# Find json-schema-for-humans if available
+json_schema_for_humans = find_program('generate-schema-doc', required : false)
+
+# Configuration for json-schema-for-humans
+json_schema_config = files('json-schema-for-humans-config.yaml')
+
+schemas = [
+ 'hash-v1',
+ 'content-address-v1',
+ 'store-path-v1',
+ 'store-object-info-v1',
+ 'derivation-v3',
+ 'deriving-path-v1',
+]
+
+schema_files = files()
+foreach schema_name : schemas
+ schema_files += files('schema' / schema_name + '.yaml')
+endforeach
+
+
+schema_outputs = []
+foreach schema_name : schemas
+ schema_outputs += schema_name + '.md'
+endforeach
+
+json_schema_generated_files = []
+
+# Generate markdown documentation from JSON schema
+# Note: output must be just a filename, not a path
+gen_file = custom_target(
+ schema_name + '-schema-docs.tmp',
+ command : [
+ json_schema_for_humans,
+ '--config-file',
+ json_schema_config,
+ meson.current_source_dir() / 'schema',
+ meson.current_build_dir(),
+ ],
+ input : schema_files + [
+ json_schema_config,
+ ],
+ output : schema_outputs,
+ capture : false,
+ build_by_default : true,
+)
+
+idx = 0
+if json_schema_for_humans.found()
+ foreach schema_name : schemas
+ #schema_file = 'schema' / schema_name + '.yaml'
+
+ # There is one so-so hack, and one horrible hack being done here.
+ sedded_file = custom_target(
+ schema_name + '-schema-docs',
+ command : [
+ 'sed',
+ '-f',
+ # Out of line to avoid https://github.com/mesonbuild/meson/issues/1564
+ files('fixup-json-schema-generated-doc.sed'),
+ '@INPUT@',
+ ],
+ capture : true,
+ input : gen_file[idx],
+ output : schema_name + '-fixed.md',
+ )
+ idx += 1
+ json_schema_generated_files += [ sedded_file ]
+ endforeach
+else
+ warning(
+ 'json-schema-for-humans not found, skipping JSON schema documentation generation',
+ )
+endif
diff --git a/doc/manual/source/protocols/json/schema/content-address-v1 b/doc/manual/source/protocols/json/schema/content-address-v1
new file mode 120000
index 000000000..35a0dd865
--- /dev/null
+++ b/doc/manual/source/protocols/json/schema/content-address-v1
@@ -0,0 +1 @@
+../../../../../../src/libstore-tests/data/content-address
\ No newline at end of file
diff --git a/doc/manual/source/protocols/json/schema/content-address-v1.yaml b/doc/manual/source/protocols/json/schema/content-address-v1.yaml
new file mode 100644
index 000000000..d0f759201
--- /dev/null
+++ b/doc/manual/source/protocols/json/schema/content-address-v1.yaml
@@ -0,0 +1,55 @@
+"$schema": "http://json-schema.org/draft-04/schema"
+"$id": "https://nix.dev/manual/nix/latest/protocols/json/schema/content-address-v1.json"
+title: Content Address
+description: |
+ This schema describes the JSON representation of Nix's `ContentAddress` type, which conveys information about [content-addressing store objects](@docroot@/store/store-object/content-address.md).
+
+ > **Note**
+ >
+ > For current methods of content addressing, this data type is a bit suspicious, because it is neither simply a content address of a file system object (the `method` is richer), nor simply a content address of a store object (the `hash` doesn't account for the references).
+ > It should thus only be used in contexts where the references are also known / otherwise made tamper-resistant.
+
+
+
+type: object
+properties:
+ method:
+ "$ref": "#/$defs/method"
+ hash:
+ title: Content Address
+ description: |
+ This would be the content-address itself.
+
+ For all current methods, this is just a content address of the file system object of the store object, [as described in the store chapter](@docroot@/store/file-system-object/content-address.md), and not of the store object as a whole.
+ In particular, the references of the store object are *not* taken into account with this hash (and currently-supported methods).
+ "$ref": "./hash-v1.yaml"
+required:
+- method
+- hash
+additionalProperties: false
+"$defs":
+ method:
+ type: string
+ enum: [flat, nar, text, git]
+ title: Content-Addressing Method
+ description: |
+ A string representing the [method](@docroot@/store/store-object/content-address.md) of content addressing that is chosen.
+
+ Valid method strings are:
+
+ - [`flat`](@docroot@/store/store-object/content-address.md#method-flat) (provided the contents are a single file)
+ - [`nar`](@docroot@/store/store-object/content-address.md#method-nix-archive)
+ - [`text`](@docroot@/store/store-object/content-address.md#method-text)
+ - [`git`](@docroot@/store/store-object/content-address.md#method-git)
diff --git a/doc/manual/source/protocols/json/schema/derivation-v3.yaml b/doc/manual/source/protocols/json/schema/derivation-v3.yaml
new file mode 100644
index 000000000..fa68adcb1
--- /dev/null
+++ b/doc/manual/source/protocols/json/schema/derivation-v3.yaml
@@ -0,0 +1,291 @@
+"$schema": "http://json-schema.org/draft-04/schema"
+"$id": "https://nix.dev/manual/nix/latest/protocols/json/schema/derivation-v3.json"
+title: Derivation
+description: |
+ Experimental JSON representation of a Nix derivation (version 3).
+
+ This schema describes the JSON representation of Nix's `Derivation` type.
+
+ > **Warning**
+ >
+ > This JSON format is currently
+ > [**experimental**](@docroot@/development/experimental-features.md#xp-feature-nix-command)
+ > and subject to change.
+
+type: object
+required:
+ - name
+ - version
+ - outputs
+ - inputSrcs
+ - inputDrvs
+ - system
+ - builder
+ - args
+ - env
+properties:
+ name:
+ type: string
+ title: Derivation name
+ description: |
+ The name of the derivation.
+ Used when calculating store paths for the derivation’s outputs.
+
+ version:
+ const: 3
+ title: Format version (must be 3)
+ description: |
+ Must be `3`.
+ This is a guard that allows us to continue evolving this format.
+ The choice of `3` is fairly arbitrary, but corresponds to this informal version:
+
+ - Version 0: ATerm format
+
+ - Version 1: Original JSON format, with ugly `"r:sha256"` inherited from ATerm format.
+
+ - Version 2: Separate `method` and `hashAlgo` fields in output specs
+
+ - Version 3: Drop store dir from store paths, just include base name.
+
+ Note that while this format is experimental, the maintenance of versions is best-effort, and not promised to identify every change.
+
+ outputs:
+ type: object
+ title: Output specifications
+ description: |
+ Information about the output paths of the derivation.
+ This is a JSON object with one member per output, where the key is the output name and the value is a JSON object as described.
+
+ > **Example**
+ >
+ > ```json
+ > "outputs": {
+ > "out": {
+ > "method": "nar",
+ > "hashAlgo": "sha256",
+ > "hash": "6fc80dcc62179dbc12fc0b5881275898f93444833d21b89dfe5f7fbcbb1d0d62"
+ > }
+ > }
+ > ```
+ additionalProperties:
+ "$ref": "#/$defs/output/overall"
+
+ inputSrcs:
+ type: array
+ title: Input source paths
+ description: |
+ List of store paths on which this derivation depends.
+
+ > **Example**
+ >
+ > ```json
+ > "inputSrcs": [
+ > "47y241wqdhac3jm5l7nv0x4975mb1975-separate-debug-info.sh",
+ > "56d0w71pjj9bdr363ym3wj1zkwyqq97j-fix-pop-var-context-error.patch"
+ > ]
+ > ```
+ items:
+ $ref: "store-path-v1.yaml"
+
+ inputDrvs:
+ type: object
+ title: Input derivations
+ description: |
+ Mapping of derivation paths to lists of output names they provide.
+
+ > **Example**
+ >
+ > ```json
+ > "inputDrvs": {
+ > "6lkh5yi7nlb7l6dr8fljlli5zfd9hq58-curl-7.73.0.drv": ["dev"],
+ > "fn3kgnfzl5dzym26j8g907gq3kbm8bfh-unzip-6.0.drv": ["out"]
+ > }
+ > ```
+ >
+ > specifies that this derivation depends on the `dev` output of `curl`, and the `out` output of `unzip`.
+ patternProperties:
+ "^[0123456789abcdfghijklmnpqrsvwxyz]{32}-.+\\.drv$":
+ title: Store Path
+ description: |
+ A store path to a derivation, mapped to the outputs of that derivation.
+ oneOf:
+ - "$ref": "#/$defs/outputNames"
+ - "$ref": "#/$defs/dynamicOutputs"
+ additionalProperties: false
+
+ system:
+ type: string
+ title: Build system type
+ description: |
+ The system type on which this derivation is to be built
+ (e.g. `x86_64-linux`).
+
+ builder:
+ type: string
+ title: Build program path
+ description: |
+ Absolute path of the program used to perform the build.
+ Typically this is the `bash` shell
+ (e.g. `/nix/store/r3j288vpmczbl500w6zz89gyfa4nr0b1-bash-4.4-p23/bin/bash`).
+
+ args:
+ type: array
+ title: Builder arguments
+ description: |
+ Command-line arguments passed to the `builder`.
+ items:
+ type: string
+
+ env:
+ type: object
+ title: Environment variables
+ description: |
+ Environment variables passed to the `builder`.
+ additionalProperties:
+ type: string
+
+ structuredAttrs:
+ title: Structured attributes
+ description: |
+ [Structured Attributes](@docroot@/store/derivation/index.md#structured-attrs), only defined if the derivation contains them.
+ Structured attributes are JSON, and thus embedded as-is.
+ type: object
+ additionalProperties: true
+
+"$defs":
+ output:
+ overall:
+ title: Derivation Output
+ description: |
+ A single output of a derivation, with different variants for different output types.
+ oneOf:
+ - "$ref": "#/$defs/output/inputAddressed"
+ - "$ref": "#/$defs/output/caFixed"
+ - "$ref": "#/$defs/output/caFloating"
+ - "$ref": "#/$defs/output/deferred"
+ - "$ref": "#/$defs/output/impure"
+
+ inputAddressed:
+ title: Input-Addressed Output
+ description: |
+ The traditional non-fixed-output derivation type.
+ The output path is determined from the derivation itself.
+
+ See [Input-addressing derivation outputs](@docroot@/store/derivation/outputs/input-address.md) for more details.
+ type: object
+ required:
+ - path
+ properties:
+ path:
+ $ref: "store-path-v1.yaml"
+ title: Output path
+ description: |
+ The output path determined from the derivation itself.
+ additionalProperties: false
+
+ caFixed:
+ title: Fixed Content-Addressed Output
+ description: |
+ The output is content-addressed, and the content-address is fixed in advance.
+
+ See [Fixed-output content-addressing](@docroot@/store/derivation/outputs/content-address.md#fixed) for more details.
+ type: object
+ required:
+ - method
+ - hashAlgo
+ - hash
+ properties:
+ method:
+ "$ref": "./content-address-v1.yaml#/$defs/method"
+ description: |
+ Method of content addressing used for this output.
+ hashAlgo:
+ title: Hash algorithm
+ "$ref": "./hash-v1.yaml#/$defs/algorithm"
+ hash:
+ type: string
+ title: Expected hash value
+ description: |
+ The expected content hash in base-16.
+ additionalProperties: false
+
+ caFloating:
+ title: Floating Content-Addressed Output
+ description: |
+ Floating-output derivations, whose outputs are content
+ addressed, but not fixed, and so the output paths are dynamically calculated from
+ whatever the output ends up being.
+
+ See [Floating Content-Addressing](@docroot@/store/derivation/outputs/content-address.md#floating) for more details.
+ type: object
+ required:
+ - method
+ - hashAlgo
+ properties:
+ method:
+ "$ref": "./content-address-v1.yaml#/$defs/method"
+ description: |
+ Method of content addressing used for this output.
+ hashAlgo:
+ title: Hash algorithm
+ "$ref": "./hash-v1.yaml#/$defs/algorithm"
+ description: |
+ What hash algorithm to use for the given method of content-addressing.
+ additionalProperties: false
+
+ deferred:
+ title: Deferred Output
+ description: |
+ Input-addressed output which depends on a (CA) derivation whose outputs (and thus their content-address
+ are not yet known.
+ type: object
+ properties: {}
+ additionalProperties: false
+
+ impure:
+ title: Impure Output
+ description: |
+ Impure output which is just like a floating content-addressed output, but this derivation runs without sandboxing.
+ As such, we don't record it in the build trace, under the assumption that if we need it again, we should rebuild it, as it might produce something different.
+ required:
+ - impure
+ - method
+ - hashAlgo
+ properties:
+ impure:
+ const: true
+ method:
+ "$ref": "./content-address-v1.yaml#/$defs/method"
+ description: |
+ How the file system objects will be serialized for hashing.
+ hashAlgo:
+ title: Hash algorithm
+ "$ref": "./hash-v1.yaml#/$defs/algorithm"
+ description: |
+ How the serialization will be hashed.
+ additionalProperties: false
+
+ outputName:
+ type: string
+ title: Output name
+ description: Name of the derivation output to depend on
+
+ outputNames:
+ type: array
+ title: Output Names
+ description: Set of names of derivation outputs to depend on
+ items:
+ "$ref": "#/$defs/outputName"
+
+ dynamicOutputs:
+ type: object
+ title: Dynamic Outputs
+ description: |
+ **Experimental feature**: [`dynamic-derivations`](@docroot@/development/experimental-features.md#xp-feature-dynamic-derivations)
+
+ This recursive data type allows for depending on outputs of outputs.
+ properties:
+ outputs:
+ "$ref": "#/$defs/outputNames"
+ dynamicOutputs:
+ "$ref": "#/$defs/dynamicOutputs"
diff --git a/doc/manual/source/protocols/json/schema/deriving-path-v1 b/doc/manual/source/protocols/json/schema/deriving-path-v1
new file mode 120000
index 000000000..92ec6d01a
--- /dev/null
+++ b/doc/manual/source/protocols/json/schema/deriving-path-v1
@@ -0,0 +1 @@
+../../../../../../src/libstore-tests/data/derived-path
\ No newline at end of file
diff --git a/doc/manual/source/protocols/json/schema/deriving-path-v1.yaml b/doc/manual/source/protocols/json/schema/deriving-path-v1.yaml
new file mode 100644
index 000000000..11a784d06
--- /dev/null
+++ b/doc/manual/source/protocols/json/schema/deriving-path-v1.yaml
@@ -0,0 +1,27 @@
+"$schema": "http://json-schema.org/draft-04/schema"
+"$id": "https://nix.dev/manual/nix/latest/protocols/json/schema/deriving-path-v1.json"
+title: Deriving Path
+description: |
+ This schema describes the JSON representation of Nix's [Deriving Path](@docroot@/store/derivation/index.md#deriving-path).
+oneOf:
+ - title: Constant
+ description: |
+ See [Constant](@docroot@/store/derivation/index.md#deriving-path-constant) deriving path.
+ $ref: "store-path-v1.yaml"
+ - title: Output
+ description: |
+ See [Output](@docroot@/store/derivation/index.md#deriving-path-output) deriving path.
+ type: object
+ properties:
+ drvPath:
+ "$ref": "#"
+ description: |
+ A deriving path to a [Derivation](@docroot@/store/derivation/index.md#store-derivation), whose output is being referred to.
+ output:
+ type: string
+ description: |
+ The name of an output produced by that derivation (e.g. "out", "doc", etc.).
+ required:
+ - drvPath
+ - output
+ additionalProperties: false
diff --git a/doc/manual/source/protocols/json/schema/hash-v1 b/doc/manual/source/protocols/json/schema/hash-v1
new file mode 120000
index 000000000..06937e286
--- /dev/null
+++ b/doc/manual/source/protocols/json/schema/hash-v1
@@ -0,0 +1 @@
+../../../../../../src/libutil-tests/data/hash/
\ No newline at end of file
diff --git a/doc/manual/source/protocols/json/schema/hash-v1.yaml b/doc/manual/source/protocols/json/schema/hash-v1.yaml
new file mode 100644
index 000000000..821546dee
--- /dev/null
+++ b/doc/manual/source/protocols/json/schema/hash-v1.yaml
@@ -0,0 +1,54 @@
+"$schema": "http://json-schema.org/draft-04/schema"
+"$id": "https://nix.dev/manual/nix/latest/protocols/json/schema/hash-v1.json"
+title: Hash
+description: |
+ A cryptographic hash value used throughout Nix for content addressing and integrity verification.
+
+ This schema describes the JSON representation of Nix's `Hash` type.
+type: object
+properties:
+ algorithm:
+ "$ref": "#/$defs/algorithm"
+ format:
+ type: string
+ enum:
+ - base64
+ - nix32
+ - base16
+ - sri
+ title: Hash format
+ description: |
+ The encoding format of the hash value.
+
+ - `base64` uses standard Base64 encoding [RFC 4648, section 4](https://datatracker.ietf.org/doc/html/rfc4648#section-4)
+ - `nix32` is Nix-specific base-32 encoding
+ - `base16` is lowercase hexadecimal
+ - `sri` is the [Subresource Integrity format](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity).
+ hash:
+ type: string
+ title: Hash
+ description: |
+ The encoded hash value, itself.
+
+ It is specified in the format specified by the `format` field.
+ It must be the right length for the hash algorithm specified in the `algorithm` field, also.
+ The hash value does not include any algorithm prefix.
+required:
+- algorithm
+- format
+- hash
+additionalProperties: false
+"$defs":
+ algorithm:
+ type: string
+ enum:
+ - blake3
+ - md5
+ - sha1
+ - sha256
+ - sha512
+ title: Hash algorithm
+ description: |
+ The hash algorithm used to compute the hash value.
+
+ `blake3` is currently experimental and requires the [`blake-hashing`](@docroot@/development/experimental-features.md#xp-feature-blake3-hashes) experimental feature.
diff --git a/doc/manual/source/protocols/json/schema/nar-info-v1 b/doc/manual/source/protocols/json/schema/nar-info-v1
new file mode 120000
index 000000000..ded866b6f
--- /dev/null
+++ b/doc/manual/source/protocols/json/schema/nar-info-v1
@@ -0,0 +1 @@
+../../../../../../src/libstore-tests/data/nar-info
\ No newline at end of file
diff --git a/doc/manual/source/protocols/json/schema/store-object-info-v1 b/doc/manual/source/protocols/json/schema/store-object-info-v1
new file mode 120000
index 000000000..fc6f5c3f0
--- /dev/null
+++ b/doc/manual/source/protocols/json/schema/store-object-info-v1
@@ -0,0 +1 @@
+../../../../../../src/libstore-tests/data/path-info
\ No newline at end of file
diff --git a/doc/manual/source/protocols/json/schema/store-object-info-v1.yaml b/doc/manual/source/protocols/json/schema/store-object-info-v1.yaml
new file mode 100644
index 000000000..d79f25043
--- /dev/null
+++ b/doc/manual/source/protocols/json/schema/store-object-info-v1.yaml
@@ -0,0 +1,235 @@
+"$schema": "http://json-schema.org/draft-07/schema"
+"$id": "https://nix.dev/manual/nix/latest/protocols/json/schema/store-object-info-v1.json"
+title: Store Object Info
+description: |
+ Information about a [store object](@docroot@/store/store-object.md).
+
+ This schema describes the JSON representation of store object metadata as returned by commands like [`nix path-info --json`](@docroot@/command-ref/new-cli/nix3-path-info.md).
+
+ > **Warning**
+ >
+ > This JSON format is currently
+ > [**experimental**](@docroot@/development/experimental-features.md#xp-feature-nix-command)
+ > and subject to change.
+
+ ### Field Categories
+
+ Store object information can come in a few different variations.
+
+ Firstly, "impure" fields, which contain non-intrinsic information about the store object, may or may not be included.
+
+ Second, binary cache stores have extra non-intrinsic infomation about the store objects they contain.
+
+ Thirdly, [`nix path-info --json --closure-size`](@docroot@/command-ref/new-cli/nix3-path-info.html#opt-closure-size) can compute some extra information about not just the single store object in question, but the store object and its [closure](@docroot@/glossary.md#gloss-closure).
+
+ The impure and NAR fields are grouped into separate variants below.
+ See their descriptions for additional information.
+ The closure fields however as just included as optional fields, to avoid a combinatorial explosion of variants.
+
+oneOf:
+ - $ref: "#/$defs/base"
+
+ - $ref: "#/$defs/impure"
+
+ - $ref: "#/$defs/narInfo"
+
+$defs:
+ base:
+ title: Store Object Info
+ description: |
+ Basic store object metadata containing only intrinsic properties.
+ This is the minimal set of fields that describe what a store object contains.
+ type: object
+ required:
+ - narHash
+ - narSize
+ - references
+ - ca
+ properties:
+ path:
+ type: string
+ title: Store Path
+ description: |
+ [Store path](@docroot@/store/store-path.md) to the given store object.
+
+ Note: This field may not be present in all contexts, such as when the path is used as the key and the the store object info the value in map.
+
+ narHash:
+ type: string
+ title: NAR Hash
+ description: |
+ Hash of the [file system object](@docroot@/store/file-system-object.md) part of the store object when serialized as a [Nix Archive](@docroot@/store/file-system-object/content-address.md#serial-nix-archive).
+
+ narSize:
+ type: integer
+ minimum: 0
+ title: NAR Size
+ description: |
+ Size of the [file system object](@docroot@/store/file-system-object.md) part of the store object when serialized as a [Nix Archive](@docroot@/store/file-system-object/content-address.md#serial-nix-archive).
+
+ references:
+ type: array
+ title: References
+ description: |
+ An array of [store paths](@docroot@/store/store-path.md), possibly including this one.
+ items:
+ type: string
+
+ ca:
+ type: ["string", "null"]
+ title: Content Address
+ description: |
+ If the store object is [content-addressed](@docroot@/store/store-object/content-address.md),
+ this is the content address of this store object's file system object, used to compute its store path.
+ Otherwise (i.e. if it is [input-addressed](@docroot@/glossary.md#gloss-input-addressed-store-object)), this is `null`.
+ additionalProperties: false
+
+ impure:
+ title: Store Object Info with Impure Fields
+ description: |
+ Store object metadata including impure fields that are not *intrinsic* properties.
+ In other words, the same store object in different stores could have different values for these impure fields.
+ type: object
+ required:
+ - narHash
+ - narSize
+ - references
+ - ca
+ # impure
+ - deriver
+ - registrationTime
+ - ultimate
+ - signatures
+ properties:
+ path: { $ref: "#/$defs/base/properties/path" }
+ narHash: { $ref: "#/$defs/base/properties/narHash" }
+ narSize: { $ref: "#/$defs/base/properties/narSize" }
+ references: { $ref: "#/$defs/base/properties/references" }
+ ca: { $ref: "#/$defs/base/properties/ca" }
+ deriver:
+ type: ["string", "null"]
+ title: Deriver
+ description: |
+ If known, the path to the [store derivation](@docroot@/glossary.md#gloss-store-derivation) from which this store object was produced.
+ Otherwise `null`.
+
+ > This is an "impure" field that may not be included in certain contexts.
+
+ registrationTime:
+ type: ["integer", "null"]
+ title: Registration Time
+ description: |
+ If known, when this derivation was added to the store (Unix timestamp).
+ Otherwise `null`.
+
+ > This is an "impure" field that may not be included in certain contexts.
+
+ ultimate:
+ type: boolean
+ title: Ultimate
+ description: |
+ Whether this store object is trusted because we built it ourselves, rather than substituted a build product from elsewhere.
+
+ > This is an "impure" field that may not be included in certain contexts.
+
+ signatures:
+ type: array
+ title: Signatures
+ description: |
+ Signatures claiming that this store object is what it claims to be.
+ Not relevant for [content-addressed](@docroot@/store/store-object/content-address.md) store objects,
+ but useful for [input-addressed](@docroot@/glossary.md#gloss-input-addressed-store-object) store objects.
+
+ > This is an "impure" field that may not be included in certain contexts.
+ items:
+ type: string
+
+ # Computed closure fields
+ closureSize:
+ type: integer
+ minimum: 0
+ title: Closure Size
+ description: |
+ The total size of this store object and every other object in its [closure](@docroot@/glossary.md#gloss-closure).
+
+ > This field is not stored at all, but computed by traversing the other fields across all the store objects in a closure.
+ additionalProperties: false
+
+ narInfo:
+ title: Store Object Info with Impure fields and NAR Info
+ description: |
+ The store object info in the "binary cache" family of Nix store type contain extra information pertaining to *downloads* of the store object in question.
+ (This store info is called "NAR info", since the downloads take the form of [Nix Archives](@docroot@/store/file-system-object/content-address.md#serial-nix-archive, and the metadata is served in a file with a `.narinfo` extension.)
+
+ This download information, being specific to how the store object happens to be stored and transferred, is also considered to be non-intrinsic / impure.
+ type: object
+ required:
+ - narHash
+ - narSize
+ - references
+ - ca
+ # impure
+ - deriver
+ - registrationTime
+ - ultimate
+ - signatures
+ # nar
+ - url
+ - compression
+ - downloadHash
+ - downloadSize
+ properties:
+ path: { $ref: "#/$defs/base/properties/path" }
+ narHash: { $ref: "#/$defs/base/properties/narHash" }
+ narSize: { $ref: "#/$defs/base/properties/narSize" }
+ references: { $ref: "#/$defs/base/properties/references" }
+ ca: { $ref: "#/$defs/base/properties/ca" }
+ deriver: { $ref: "#/$defs/impure/properties/deriver" }
+ registrationTime: { $ref: "#/$defs/impure/properties/registrationTime" }
+ ultimate: { $ref: "#/$defs/impure/properties/ultimate" }
+ signatures: { $ref: "#/$defs/impure/properties/signatures" }
+ closureSize: { $ref: "#/$defs/impure/properties/closureSize" }
+ url:
+ type: string
+ title: URL
+ description: |
+ Where to download a compressed archive of the file system objects of this store object.
+
+ > This is an impure "`.narinfo`" field that may not be included in certain contexts.
+
+ compression:
+ type: string
+ title: Compression
+ description: |
+ The compression format that the archive is in.
+
+ > This is an impure "`.narinfo`" field that may not be included in certain contexts.
+
+ downloadHash:
+ type: string
+ title: Download Hash
+ description: |
+ A digest for the compressed archive itself, as opposed to the data contained within.
+
+ > This is an impure "`.narinfo`" field that may not be included in certain contexts.
+
+ downloadSize:
+ type: integer
+ minimum: 0
+ title: Download Size
+ description: |
+ The size of the compressed archive itself.
+
+ > This is an impure "`.narinfo`" field that may not be included in certain contexts.
+
+ closureDownloadSize:
+ type: integer
+ minimum: 0
+ title: Closure Download Size
+ description: |
+ The total size of the compressed archive itself for this object, and the compressed archive of every object in this object's [closure](@docroot@/glossary.md#gloss-closure).
+
+ > This is an impure "`.narinfo`" field that may not be included in certain contexts.
+
+ > This field is not stored at all, but computed by traversing the other fields across all the store objects in a closure.
+ additionalProperties: false
diff --git a/doc/manual/source/protocols/json/schema/store-path-v1 b/doc/manual/source/protocols/json/schema/store-path-v1
new file mode 120000
index 000000000..31e7a6b2a
--- /dev/null
+++ b/doc/manual/source/protocols/json/schema/store-path-v1
@@ -0,0 +1 @@
+../../../../../../src/libstore-tests/data/store-path
\ No newline at end of file
diff --git a/doc/manual/source/protocols/json/schema/store-path-v1.yaml b/doc/manual/source/protocols/json/schema/store-path-v1.yaml
new file mode 100644
index 000000000..2012aab99
--- /dev/null
+++ b/doc/manual/source/protocols/json/schema/store-path-v1.yaml
@@ -0,0 +1,32 @@
+"$schema": "http://json-schema.org/draft-07/schema"
+"$id": "https://nix.dev/manual/nix/latest/protocols/json/schema/store-path-v1.json"
+title: Store Path
+description: |
+ A [store path](@docroot@/store/store-path.md) identifying a store object.
+
+ This schema describes the JSON representation of store paths as used in various Nix JSON APIs.
+
+ > **Warning**
+ >
+ > This JSON format is currently
+ > [**experimental**](@docroot@/development/experimental-features.md#xp-feature-nix-command)
+ > and subject to change.
+
+ ## Format
+
+ Store paths in JSON are represented as strings containing just the hash and name portion, without the store directory prefix.
+
+ For example: `"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo.drv"`
+
+ (If the store dir is `/nix/store`, then this corresponds to the path `/nix/store/g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo.drv`.)
+
+ ## Structure
+
+ The format follows this pattern: `${digest}-${name}`
+
+ - **hash**: Digest rendered in a custom variant of [Base32](https://en.wikipedia.org/wiki/Base32) (20 arbitrary bytes become 32 ASCII characters)
+ - **name**: The package name and optional version/suffix information
+
+type: string
+pattern: "^[0123456789abcdfghijklmnpqrsvwxyz]{32}-.+$"
+minLength: 34
diff --git a/doc/manual/source/protocols/json/store-object-info.md b/doc/manual/source/protocols/json/store-object-info.md
index b7348538c..4673dd773 100644
--- a/doc/manual/source/protocols/json/store-object-info.md
+++ b/doc/manual/source/protocols/json/store-object-info.md
@@ -1,102 +1,45 @@
-# Store object info JSON format
+{{#include store-object-info-v1-fixed.md}}
-> **Warning**
->
-> This JSON format is currently
-> [**experimental**](@docroot@/development/experimental-features.md#xp-feature-nix-command)
-> and subject to change.
+## Examples
-Info about a [store object].
+### Minimal store object (content-addressed)
-* `path`:
+```json
+{{#include schema/store-object-info-v1/pure.json}}
+```
- [Store path][store path] to the given store object.
+### Store object with impure fields
-* `narHash`:
+```json
+{{#include schema/store-object-info-v1/impure.json}}
+```
- Hash of the [file system object] part of the store object when serialized as a [Nix Archive].
+### Minimal store object (empty)
-* `narSize`:
+```json
+{{#include schema/store-object-info-v1/empty_pure.json}}
+```
- Size of the [file system object] part of the store object when serialized as a [Nix Archive].
+### Store object with all impure fields
-* `references`:
+```json
+{{#include schema/store-object-info-v1/empty_impure.json}}
+```
- An array of [store paths][store path], possibly including this one.
+### NAR info (minimal)
-* `ca`:
+```json
+{{#include schema/nar-info-v1/pure.json}}
+```
- If the store object is [content-addressed],
- this is the content address of this store object's file system object, used to compute its store path.
- Otherwise (i.e. if it is [input-addressed]), this is `null`.
+### NAR info (with binary cache fields)
-[store path]: @docroot@/store/store-path.md
-[file system object]: @docroot@/store/file-system-object.md
-[Nix Archive]: @docroot@/store/file-system-object/content-address.md#serial-nix-archive
+```json
+{{#include schema/nar-info-v1/impure.json}}
+```
-## Impure fields
+
diff --git a/doc/manual/source/protocols/json/store-path.md b/doc/manual/source/protocols/json/store-path.md
new file mode 100644
index 000000000..cd18f6595
--- /dev/null
+++ b/doc/manual/source/protocols/json/store-path.md
@@ -0,0 +1,15 @@
+{{#include store-path-v1-fixed.md}}
+
+## Examples
+
+### Simple store path
+
+```json
+{{#include schema/store-path-v1/simple.json}}
+```
+
+
diff --git a/doc/manual/source/protocols/meson.build b/doc/manual/source/protocols/meson.build
new file mode 100644
index 000000000..5b5eb900d
--- /dev/null
+++ b/doc/manual/source/protocols/meson.build
@@ -0,0 +1,2 @@
+# Process JSON schema documentation
+subdir('json')
diff --git a/doc/manual/source/protocols/nix-archive.md b/doc/manual/source/protocols/nix-archive/index.md
similarity index 72%
rename from doc/manual/source/protocols/nix-archive.md
rename to doc/manual/source/protocols/nix-archive/index.md
index 02a8dd464..bd2a8e833 100644
--- a/doc/manual/source/protocols/nix-archive.md
+++ b/doc/manual/source/protocols/nix-archive/index.md
@@ -4,7 +4,7 @@ This is the complete specification of the [Nix Archive] format.
The Nix Archive format closely follows the abstract specification of a [file system object] tree,
because it is designed to serialize exactly that data structure.
-[Nix Archive]: @docroot@/store/file-system-object/content-address.md#nix-archive
+[Nix Archive]: @docroot@/store/file-system-object/content-address.md#serial-nix-archive
[file system object]: @docroot@/store/file-system-object.md
The format of this specification is close to [Extended Backus–Naur form](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form), with the exception of the `str(..)` function / parameterized rule, which length-prefixes and pads strings.
@@ -41,3 +41,15 @@ The `str` function / parameterized rule is defined as follows:
- `int(n)` = the 64-bit little endian representation of the number `n`
- `pad(s)` = the byte sequence `s`, padded with 0s to a multiple of 8 byte
+
+## Kaitai Struct Specification
+
+The Nix Archive (NAR) format is also formally described using [Kaitai Struct](https://kaitai.io/), an Interface Description Language (IDL) for defining binary data structures.
+
+> Kaitai Struct provides a language-agnostic, machine-readable specification that can be compiled into parsers for various programming languages (e.g., C++, Python, Java, Rust).
+
+```yaml
+{{#include nar.ksy}}
+```
+
+The source of the spec can be found [here](https://github.com/nixos/nix/blob/master/src/nix-manual/source/protocols/nix-archive/nar.ksy). Contributions and improvements to the spec are welcomed.
\ No newline at end of file
diff --git a/doc/manual/source/protocols/nix-archive/nar.ksy b/doc/manual/source/protocols/nix-archive/nar.ksy
new file mode 100644
index 000000000..6a172b276
--- /dev/null
+++ b/doc/manual/source/protocols/nix-archive/nar.ksy
@@ -0,0 +1,169 @@
+meta:
+ id: nix_nar
+ title: Nix Archive (NAR)
+ file-extension: nar
+ endian: le
+doc: |
+ Nix Archive (NAR) format. A simple, reproducible binary archive
+ format used by the Nix package manager to serialize file system objects.
+doc-ref: 'https://nixos.org/manual/nix/stable/command-ref/nix-store.html#nar-format'
+
+seq:
+ - id: magic
+ type: padded_str
+ doc: "Magic string, must be 'nix-archive-1'."
+ valid:
+ expr: _.body == 'nix-archive-1'
+ - id: root_node
+ type: node
+ doc: "The root of the archive, which is always a single node."
+
+types:
+ padded_str:
+ doc: |
+ A string, prefixed with its length (u8le) and
+ padded with null bytes to the next 8-byte boundary.
+ seq:
+ - id: len_str
+ type: u8
+ - id: body
+ type: str
+ size: len_str
+ encoding: 'ASCII'
+ - id: padding
+ size: (8 - (len_str % 8)) % 8
+
+ node:
+ doc: "A single filesystem node (file, directory, or symlink)."
+ seq:
+ - id: open_paren
+ type: padded_str
+ doc: "Must be '(', a token starting the node definition."
+ valid:
+ expr: _.body == '('
+ - id: type_key
+ type: padded_str
+ doc: "Must be 'type'."
+ valid:
+ expr: _.body == 'type'
+ - id: type_val
+ type: padded_str
+ doc: "The type of the node: 'regular', 'directory', or 'symlink'."
+ - id: body
+ type:
+ switch-on: type_val.body
+ cases:
+ "'directory'": type_directory
+ "'regular'": type_regular
+ "'symlink'": type_symlink
+ - id: close_paren
+ type: padded_str
+ valid:
+ expr: _.body == ')'
+ if: "type_val.body != 'directory'"
+ doc: "Must be ')', a token ending the node definition."
+
+ type_directory:
+ doc: "A directory node, containing a list of entries. Entries must be ordered by their names."
+ seq:
+ - id: entries
+ type: dir_entry
+ repeat: until
+ repeat-until: _.kind.body == ')'
+ types:
+ dir_entry:
+ doc: "A single entry within a directory, or a terminator."
+ seq:
+ - id: kind
+ type: padded_str
+ valid:
+ expr: _.body == 'entry' or _.body == ')'
+ doc: "Must be 'entry' (for a child node) or '' (for terminator)."
+ - id: open_paren
+ type: padded_str
+ valid:
+ expr: _.body == '('
+ if: 'kind.body == "entry"'
+ - id: name_key
+ type: padded_str
+ valid:
+ expr: _.body == 'name'
+ if: 'kind.body == "entry"'
+ - id: name
+ type: padded_str
+ if: 'kind.body == "entry"'
+ - id: node_key
+ type: padded_str
+ valid:
+ expr: _.body == 'node'
+ if: 'kind.body == "entry"'
+ - id: node
+ type: node
+ if: 'kind.body == "entry"'
+ doc: "The child node, present only if kind is 'entry'."
+ - id: close_paren
+ type: padded_str
+ valid:
+ expr: _.body == ')'
+ if: 'kind.body == "entry"'
+ instances:
+ is_terminator:
+ value: kind.body == ')'
+
+ type_regular:
+ doc: "A regular file node."
+ seq:
+ # Read attributes (like 'executable') until we hit 'contents'
+ - id: attributes
+ type: reg_attribute
+ repeat: until
+ repeat-until: _.key.body == "contents"
+ # After the 'contents' token, read the file data
+ - id: file_data
+ type: file_content
+ instances:
+ is_executable:
+ value: 'attributes[0].key.body == "executable"'
+ doc: "True if the file has the 'executable' attribute."
+ types:
+ reg_attribute:
+ doc: "An attribute of the file, e.g., 'executable' or 'contents'."
+ seq:
+ - id: key
+ type: padded_str
+ doc: "Attribute key, e.g., 'executable' or 'contents'."
+ valid:
+ expr: _.body == 'executable' or _.body == 'contents'
+ - id: value
+ type: padded_str
+ if: 'key.body == "executable"'
+ valid:
+ expr: _.body == ''
+ doc: "Must be '' if key is 'executable'."
+ file_content:
+ doc: "The raw data of the file, prefixed by length."
+ seq:
+ - id: len_contents
+ type: u8
+ # # This relies on the property of instances that they are lazily evaluated and cached.
+ - size: 0
+ if: nar_offset < 0
+ - id: contents
+ size: len_contents
+ - id: padding
+ size: (8 - (len_contents % 8)) % 8
+ instances:
+ nar_offset:
+ value: _io.pos
+
+ type_symlink:
+ doc: "A symbolic link node."
+ seq:
+ - id: target_key
+ type: padded_str
+ doc: "Must be 'target'."
+ valid:
+ expr: _.body == 'target'
+ - id: target_val
+ type: padded_str
+ doc: "The destination path of the symlink."
diff --git a/doc/manual/source/release-notes/rl-2.18.md b/doc/manual/source/release-notes/rl-2.18.md
index eb26fc9e7..71b25f408 100644
--- a/doc/manual/source/release-notes/rl-2.18.md
+++ b/doc/manual/source/release-notes/rl-2.18.md
@@ -13,7 +13,7 @@
- The `discard-references` feature has been stabilized.
This means that the
- [unsafeDiscardReferences](@docroot@/development/experimental-features.md#xp-feature-discard-references)
+ [unsafeDiscardReferences](@docroot@/language/advanced-attributes.md#adv-attr-unsafeDiscardReferences)
attribute is no longer guarded by an experimental flag and can be used
freely.
diff --git a/doc/manual/source/release-notes/rl-2.19.md b/doc/manual/source/release-notes/rl-2.19.md
index 06c704324..04f8c9c28 100644
--- a/doc/manual/source/release-notes/rl-2.19.md
+++ b/doc/manual/source/release-notes/rl-2.19.md
@@ -17,8 +17,8 @@
- `nix-shell` shebang lines now support single-quoted arguments.
-- `builtins.fetchTree` is now its own experimental feature, [`fetch-tree`](@docroot@/development/experimental-features.md#xp-fetch-tree).
- This allows stabilising it independently of the rest of what is encompassed by [`flakes`](@docroot@/development/experimental-features.md#xp-fetch-tree).
+- `builtins.fetchTree` is now its own experimental feature, [`fetch-tree`](@docroot@/development/experimental-features.md#xp-feature-fetch-tree).
+ This allows stabilising it independently of the rest of what is encompassed by [`flakes`](@docroot@/development/experimental-features.md#xp-feature-flakes).
- The interface for creating and updating lock files has been overhauled:
diff --git a/doc/manual/source/release-notes/rl-2.23.md b/doc/manual/source/release-notes/rl-2.23.md
index e6b0e9ffc..b358a0fdc 100644
--- a/doc/manual/source/release-notes/rl-2.23.md
+++ b/doc/manual/source/release-notes/rl-2.23.md
@@ -14,7 +14,7 @@
- Modify `nix derivation {add,show}` JSON format [#9866](https://github.com/NixOS/nix/issues/9866) [#10722](https://github.com/NixOS/nix/pull/10722)
- The JSON format for derivations has been slightly revised to better conform to our [JSON guidelines](@docroot@/development/cli-guideline.md#returning-future-proof-json).
+ The JSON format for derivations has been slightly revised to better conform to our [JSON guidelines](@docroot@/development/json-guideline.md).
In particular, the hash algorithm and content addressing method of content-addressed derivation outputs are now separated into two fields `hashAlgo` and `method`,
rather than one field with an arcane `:`-separated format.
diff --git a/doc/manual/source/release-notes/rl-2.24.md b/doc/manual/source/release-notes/rl-2.24.md
index d4af3cb51..e9b46bb22 100644
--- a/doc/manual/source/release-notes/rl-2.24.md
+++ b/doc/manual/source/release-notes/rl-2.24.md
@@ -93,7 +93,7 @@
- Support unit prefixes in configuration settings [#10668](https://github.com/NixOS/nix/pull/10668)
- Configuration settings in Nix now support unit prefixes, allowing for more intuitive and readable configurations. For example, you can now specify [`--min-free 1G`](@docroot@/command-ref/opt-common.md#opt-min-free) to set the minimum free space to 1 gigabyte.
+ Configuration settings in Nix now support unit prefixes, allowing for more intuitive and readable configurations. For example, you can now specify [`--min-free 1G`](@docroot@/command-ref/conf-file.md#conf-min-free) to set the minimum free space to 1 gigabyte.
This enhancement was extracted from [#7851](https://github.com/NixOS/nix/pull/7851) and is also useful for PR [#10661](https://github.com/NixOS/nix/pull/10661).
diff --git a/doc/manual/source/store/build-trace.md b/doc/manual/source/store/build-trace.md
new file mode 100644
index 000000000..beb9bea6a
--- /dev/null
+++ b/doc/manual/source/store/build-trace.md
@@ -0,0 +1,53 @@
+# Build Trace
+
+> **Warning**
+>
+> This entire concept is currently
+> [**experimental**](@docroot@/development/experimental-features.md#xp-feature-ca-derivations)
+> and subject to change.
+
+The *build trace* is a [memoization table](https://en.wikipedia.org/wiki/Memoization) for builds.
+It maps the inputs of builds to the outputs of builds.
+Concretely, that means it maps [derivations][derivation] to maps of [output] names to [store objects][store object].
+
+In general the derivations used as a key should be [*resolved*](./resolution.md).
+A build trace with all-resolved-derivation keys is also called a *base build trace* for extra clarity.
+If all the resolved inputs of a derivation are content-addressed, that means the inputs will be fully determined, leaving no ambiguity for what build was performed.
+(Input-addressed inputs however are still ambiguous. They too should be locked down, but this is left as future work.)
+
+Accordingly, to look up an unresolved derivation, one must first resolve it to get a resolved derivation.
+Resolving itself involves looking up entries in the build trace, so this is a mutually recursive process that will end up inspecting possibly many entries.
+
+Except for the issue with input-addressed paths called out above, base build traces are trivially *coherent* -- incoherence is not possible.
+That means that the claims that each key-value base build try entry makes are independent, and no mapping invalidates another mapping.
+
+Whether the mappings are *true*, i.e. the faithful recording of actual builds performed, is another matter.
+Coherence is about the multiple claims of the build trace being mutually consistent, not about whether the claims are individually true or false.
+
+In general, there is no way to audit a build trace entry except for by performing the build again from scratch.
+And even in that case, a different result doesn't mean the original entry was a "lie", because the derivation being built may be non-deterministic.
+As such, the decision of whether to trust a counterparty's build trace is a fundamentally subject policy choice.
+Build trace entries are typically *signed* in order to enable arbitrary public-key-based trust polices.
+
+## Derived build traces
+
+Implementations that wish to memoize the above may also keep additional *derived* build trace entries that do map unresolved derivations.
+But if they do so, they *must* also keep the underlying base entries with resolved derivation keys around.
+Firstly, this ensures that the derived entries are merely cache, which could be recomputed from scratch.
+Secondly, this ensures the coherence of the derived build trace.
+
+Unlike with base build traces, incoherence with derived build traces is possible.
+The key ingredient is that derivation resolution is only deterministic with respect to a fixed base build trace.
+Without fixing the base build trace, it inherits the subjectivity of base build traces themselves.
+
+Concretely, suppose there are three derivations \\(a\\), \\(b\\), and \\(c\\).
+Let \\(a\\) be a resolved derivation, but let \\(b\\) and \\(c\\) be unresolved and both take as an input an output of \\(a\\).
+Now suppose that derived entries are made for \\(b\\) and \\(c\\) based on two different entries of \\(a\\).
+(This could happen if \\(a\\) is non-deterministic, \\(a\\) and \\(b\\) are built in one store, \\(a\\) and \\(c\\) are built in another store, and then a third store substitutes from both of the first two stores.)
+
+If trusting the derived build trace entries for \\(b\\) and \\(c\\) requires that each's underlying entry for \\(a\\) be also trusted, the two different mappings for \\(a\\) will be caught.
+However, if \\(b\\) and \\(c\\)'s entries can be combined in isolation, there will be nothing to catch the contradiction in their hidden assumptions about \\(a\\)'s output.
+
+[derivation]: ./derivation/index.md
+[output]: ./derivation/outputs/index.md
+[store object]: @docroot@/store/store-object.md
diff --git a/doc/manual/source/store/building.md b/doc/manual/source/store/building.md
index dbfe6b5ca..f2d470e99 100644
--- a/doc/manual/source/store/building.md
+++ b/doc/manual/source/store/building.md
@@ -8,7 +8,7 @@
- Once this is done, the derivation is *normalized*, replacing each input deriving path with its store path, which we now know from realising the input.
-## Builder Execution
+## Builder Execution {#builder-execution}
The [`builder`](./derivation/index.md#builder) is executed as follows:
diff --git a/doc/manual/source/store/derivation/index.md b/doc/manual/source/store/derivation/index.md
index 5b179273d..670f3b2bd 100644
--- a/doc/manual/source/store/derivation/index.md
+++ b/doc/manual/source/store/derivation/index.md
@@ -102,7 +102,7 @@ But rather than somehow scanning all the other fields for inputs, Nix requires t
### System {#system}
-The system type on which the [`builder`](#attr-builder) executable is meant to be run.
+The system type on which the [`builder`](#builder) executable is meant to be run.
A necessary condition for Nix to schedule a given derivation on some [Nix instance] is for the "system" of that derivation to match that instance's [`system` configuration option] or [`extra-platforms` configuration option].
@@ -245,7 +245,7 @@ If those other derivations *also* abide by this common case (and likewise for tr
> note the ".drv"
> ```
-## Extending the model to be higher-order
+## Extending the model to be higher-order {#dynamic}
**Experimental feature**: [`dynamic-derivations`](@docroot@/development/experimental-features.md#xp-feature-dynamic-derivations)
diff --git a/doc/manual/source/store/derivation/outputs/content-address.md b/doc/manual/source/store/derivation/outputs/content-address.md
index 4d5130348..aa65fbe49 100644
--- a/doc/manual/source/store/derivation/outputs/content-address.md
+++ b/doc/manual/source/store/derivation/outputs/content-address.md
@@ -167,10 +167,10 @@ It is only in the potential for that check to fail that they are different.
>
> In a future world where floating content-addressing is also stable, we in principle no longer need separate [fixed](#fixed) content-addressing.
> Instead, we could always use floating content-addressing, and separately assert the precise value content address of a given store object to be used as an input (of another derivation).
-> A stand-alone assertion object of this sort is not yet implemented, but its possible creation is tracked in [Issue #11955](https://github.com/NixOS/nix/issues/11955).
+> A stand-alone assertion object of this sort is not yet implemented, but its possible creation is tracked in [issue #11955](https://github.com/NixOS/nix/issues/11955).
>
> In the current version of Nix, fixed outputs which fail their hash check are still registered as valid store objects, just not registered as outputs of the derivation which produced them.
-> This is an optimization that means if the wrong output hash is specified in a derivation, and then the derivation is recreated with the right output hash, derivation does not need to be rebuilt --- avoiding downloading potentially large amounts of data twice.
+> This is an optimization that means if the wrong output hash is specified in a derivation, and then the derivation is recreated with the right output hash, derivation does not need to be rebuilt — avoiding downloading potentially large amounts of data twice.
> This optimisation prefigures the design above:
> If the output hash assertion was removed outside the derivation itself, Nix could additionally not only register that outputted store object like today, but could also make note that derivation did in fact successfully download some data.
For example, for the "fetch URL" example above, making such a note is tantamount to recording what data is available at the time of download at the given URL.
diff --git a/doc/manual/source/store/derivation/outputs/index.md b/doc/manual/source/store/derivation/outputs/index.md
index 0683f5703..ca2ce6665 100644
--- a/doc/manual/source/store/derivation/outputs/index.md
+++ b/doc/manual/source/store/derivation/outputs/index.md
@@ -43,7 +43,7 @@ In particular, the specification decides:
- if the content is content-addressed, how is it content addressed
-- if the content is content-addressed, [what is its content address](./content-address.md#fixed-content-addressing) (and thus what is its [store path])
+- if the content is content-addressed, [what is its content address](./content-address.md#fixed) (and thus what is its [store path])
## Types of derivations
diff --git a/doc/manual/source/store/derivation/outputs/input-address.md b/doc/manual/source/store/derivation/outputs/input-address.md
index e2e15a801..3fd20f17d 100644
--- a/doc/manual/source/store/derivation/outputs/input-address.md
+++ b/doc/manual/source/store/derivation/outputs/input-address.md
@@ -6,26 +6,221 @@
That is to say, an input-addressed output's store path is a function not of the output itself, but of the derivation that produced it.
Even if two store paths have the same contents, if they are produced in different ways, and one is input-addressed, then they will have different store paths, and thus guaranteed to not be the same store object.
-
+type FirstOrderDerivingPath = ConstantPath | FirstOrderOutputPath;
+type Inputs = Set;
+```
+
+For the algorithm below, we adopt a derivation where the two types of (first order) derived paths are partitioned into two sets, as follows:
+```typescript
+type Derivation = {
+ // inputs: Set; // replaced
+ inputSrcs: Set; // new instead
+ inputDrvOutputs: Set; // new instead
+ // ...other fields...
+};
+```
+
+In the [currently-experimental][xp-feature-dynamic-derivations] higher-order case where outputs of outputs are allowed as [deriving paths][deriving-path] and thus derivation inputs, derivations using that generalization are not valid arguments to this function.
+Those derivations must be (partially) [resolved](@docroot@/store/resolution.md) enough first, to the point where no such higher-order inputs remain.
+Then, and only then, can input addresses be assigned.
+
+```
+function hashQuotientDerivation(drv) -> Hash:
+ assert(drv.outputs are input-addressed)
+ drv′ ← drv with {
+ inputDrvOutputs = ⋃(
+ assert(drvPath is store path)
+ case hashOutputsOrQuotientDerivation(readDrv(drvPath)) of
+ drvHash : Hash →
+ (drvHash.toBase16(), output)
+ outputHashes : Map[String, Hash] →
+ (outputHashes[output].toBase16(), "out")
+ | (drvPath, output) ∈ drv.inputDrvOutputs
+ )
+ }
+ return hashSHA256(printDrv(drv′))
+
+function hashOutputsOrQuotientDerivation(drv) -> Map[String, Hash] | Hash:
+ if drv.outputs are content-addressed:
+ return {
+ outputName ↦ hashSHA256(
+ "fixed:out:" + ca.printMethodAlgo() +
+ ":" + ca.hash.toBase16() +
+ ":" + ca.makeFixedOutputPath(drv.name, outputName))
+ | (outputName ↦ output) ∈ drv.outputs
+ , ca = output.contentAddress // or get from build trace if floating
+ }
+ else: // drv.outputs are input-addressed
+ return hashQuotientDerivation(drv)
+```
+
+### `hashQuotientDerivation`
+
+We replace each element in the derivation's `inputDrvOutputs` using data from a call to `hashOutputsOrQuotientDerivation` on the `drvPath` of that element.
+When `hashOutputsOrQuotientDerivation` returns a single drv hash (because the input derivation in question is input-addressing), we simply swap out the `drvPath` for that hash, and keep the same output name.
+When `hashOutputsOrQuotientDerivation` returns a map of content addresses per-output, we look up the output in question, and pair it with the output name `out`.
+
+The resulting pseudo-derivation (with hashes instead of store paths in `inputDrvs`) is then printed (in the ["ATerm" format](@docroot@/protocols/derivation-aterm.md)) and hashed, and this becomes the hash of the "quotient derivation".
+
+When calculating output hashes, `hashQuotientDerivation` is called on an almost-complete input-addressing derivation, which is just missing its input-addressed outputs paths.
+The derivation hash is then used to calculate output paths for each output.
+
+Those output paths can then be substituted into the almost-complete input-addressed derivation to complete it.
+
+> **Note**
+>
+> There may be an unintentional deviation from specification currently implemented in the `(outputHashes[output].toBase16(), "out")` case.
+> This is not fatal because the deviation would only apply for content-addressing derivations with more than one output, and that only occurs in the floating case, which is [experimental][xp-feature-ca-derivations].
+> Once this bug is fixed, this note will be removed.
+
+### `hashOutputsOrQuotientDerivation`
+
+How does `hashOutputsOrQuotientDerivation` in turn work?
+It consists of two main cases, based on whether the outputs of the derivation are to be input-addressed or content-addressed.
+
+#### Input-addressed outputs case
+
+In the input-addressed case, it just calls `hashQuotientDerivation`, and returns that derivation hash.
+This makes `hashQuotientDerivation` and `hashOutputsOrQuotientDerivation` mutually-recursive.
+
+> **Note**
+>
+> In this case, `hashQuotientDerivation` is being called on a *complete* input-addressing derivation that already has its output paths calculated.
+> The `inputDrvs` substitution takes place anyways.
+
+#### Content-addressed outputs case
+
+If the outputs are [content-addressed](./content-address.md), then it computes a hash for each output derived from the content-address of that output.
+
+> **Note**
+>
+> In the [fixed](./content-address.md#fixed) content-addressing case, the outputs' content addresses are statically specified in advance, so this always just works.
+> (The fixed case is what the pseudo-code shows.)
+>
+> In the [floating](./content-address.md#floating) case, the content addresses are not specified in advance.
+> This is what the "or get from [build trace](@docroot@/store/build-trace.md) if floating" comment refers to.
+> In this case, the algorithm is *stuck* until the input in question is built, and we know what the actual contents of the output in question is.
+>
+> That is OK however, because there is no problem with delaying the assigning of input addresses (which, remember, is what `hashQuotientDerivation` is ultimately for) until all inputs are known.
+
+### Performance
+
+The recursion in the algorithm is potentially inefficient:
+it could call itself once for each path by which a subderivation can be reached, i.e., `O(V^k)` times for a derivation graph with `V` derivations and with out-degree of at most `k`.
+In the actual implementation, [memoisation](https://en.wikipedia.org/wiki/Memoization) is used to reduce this cost to be proportional to the total number of `inputDrvOutputs` encountered.
+
+### Semantic properties
+
+*See [this chapter's appendix](@docroot@/store/math-notation.md) on grammar and metavariable conventions.*
+
+In essence, `hashQuotientDerivation` partitions input-addressing derivations into equivalence classes: every derivation in that equivalence class is mapped to the same derivation hash.
+We can characterize this equivalence relation directly, by working bottom up.
+
+We start by defining an equivalence relation on first-order output deriving paths that refer content-addressed derivation outputs. Two such paths are equivalent if they refer to the same store object:
+
+\\[
+\\begin{prooftree}
+\\AxiomC{$d\_1$ is content-addressing}
+\\AxiomC{$d\_2$ is content-addressing}
+\\AxiomC{$
+ {}^\*(\text{path}(d\_1), o\_1)
+ \=
+ {}^\*(\text{path}(d\_2), o\_2)
+$}
+\\TrinaryInfC{$(\text{path}(d\_1), o\_1) \\,\\sim_{\\mathrm{CA}}\\, (d\_2, o\_2)$}
+\\end{prooftree}
+\\]
+
+where \\({}^*(s, o)\\) denotes the store object that the output deriving path refers to.
+
+We will also need the following construction to lift any equivalence relation on \\(X\\) to an equivalence relation on (finite) sets of \\(X\\) (in short, \\(\\mathcal{P}(X)\\)):
+
+\\[
+\\begin{prooftree}
+\\AxiomC{$\\forall a \\in A. \\exists b \\in B. a \\,\\sim\_X\\, b$}
+\\AxiomC{$\\forall b \\in B. \\exists a \\in A. b \\,\\sim\_X\\, a$}
+\\BinaryInfC{$A \\,\\sim_{\\mathcal{P}(X)}\\, B$}
+\\end{prooftree}
+\\]
+
+Now we can define the equivalence relation \\(\\sim_\\mathrm{IA}\\) on input-addressed derivation outputs. Two input-addressed outputs are equivalent if their derivations are equivalent (via the yet-to-be-defined \\(\\sim_{\\mathrm{IADrv}}\\) relation) and their output names are the same:
+
+\\[
+\\begin{prooftree}
+\\AxiomC{$d\_1$ is input-addressing}
+\\AxiomC{$d\_2$ is input-addressing}
+\\AxiomC{$d\_1 \\,\\sim_{\\mathrm{IADrv}}\\, d\_2$}
+\\AxiomC{$o\_1 = o\_2$}
+\\QuaternaryInfC{$(\text{path}(d\_1), o\_1) \\,\\sim_{\\mathrm{IA}}\\, (\text{path}(d\_2), o\_2)$}
+\\end{prooftree}
+\\]
+
+And now we can define \\(\\sim_{\\mathrm{IADrv}}\\).
+Two input-addressed derivations are equivalent if their content-addressed inputs are equivalent, their input-addressed inputs are also equivalent, and they are otherwise equal:
+
+
+
+\\[
+\\begin{prooftree}
+\\alwaysNoLine
+\\AxiomC{$
+ \\mathrm{caInputs}(d\_1)
+ \\,\\sim_{\\mathcal{P}(\\mathrm{CA})}\\,
+ \\mathrm{caInputs}(d\_2)
+$}
+\\AxiomC{$
+ \\mathrm{iaInputs}(d\_1)
+ \\,\\sim_{\\mathcal{P}(\\mathrm{IA})}\\,
+ \\mathrm{iaInputs}(d\_2)
+$}
+\\BinaryInfC{$
+ d\_1\left[\\mathrm{inputDrvOutputs} := \\{\\}\right]
+ \=
+ d\_2\left[\\mathrm{inputDrvOutputs} := \\{\\}\right]
+$}
+\\alwaysSingleLine
+\\UnaryInfC{$d\_1 \\,\\sim_{\\mathrm{IADrv}}\\, d\_2$}
+\\end{prooftree}
+\\]
+
+where \\(\\mathrm{caInputs}(d)\\) returns the content-addressed inputs of \\(d\\) and \\(\\mathrm{iaInputs}(d)\\) returns the input-addressed inputs.
+
+> **Note**
+>
+> An astute reader might notice that that nowhere does `inputSrcs` enter into these definitions.
+> That means that replacing an input derivation with its outputs directly added to `inputSrcs` always results in a derivation in a different equivalence class, despite the resulting input closure (as would be mounted in the store at build time) being the same.
+> [Issue #9259](https://github.com/NixOS/nix/issues/9259) is about creating a coarser equivalence relation to address this.
+>
+> \\(\\sim_\mathrm{Drv}\\) from [derivation resolution](@docroot@/store/resolution.md) is such an equivalence relation.
+> It is coarser than this one: any two derivations which are "'hash quotient derivation'-equivalent" (\\(\\sim_\mathrm{IADrv}\\)) are also "resolution-equivalent" (\\(\\sim_\mathrm{Drv}\\)).
+> It also relates derivations whose `inputDrvOutputs` have been rewritten into `inputSrcs`.
+
+[deriving-path]: @docroot@/store/derivation/index.md#deriving-path
+[xp-feature-dynamic-derivations]: @docroot@/development/experimental-features.md#xp-feature-dynamic-derivations
[xp-feature-ca-derivations]: @docroot@/development/experimental-features.md#xp-feature-ca-derivations
-[xp-feature-git-hashing]: @docroot@/development/experimental-features.md#xp-feature-git-hashing
-[xp-feature-impure-derivations]: @docroot@/development/experimental-features.md#xp-feature-impure-derivations
diff --git a/doc/manual/source/store/file-system-object/content-address.md b/doc/manual/source/store/file-system-object/content-address.md
index 04a1021f1..5685de03e 100644
--- a/doc/manual/source/store/file-system-object/content-address.md
+++ b/doc/manual/source/store/file-system-object/content-address.md
@@ -46,7 +46,7 @@ be many different serialisations.
For these reasons, Nix has its very own archive format—the Nix Archive (NAR) format,
which is carefully designed to avoid the problems described above.
-The exact specification of the Nix Archive format is in [specified here](../../protocols/nix-archive.md).
+The exact specification of the Nix Archive format is in [specified here](../../protocols/nix-archive/index.md).
## Content addressing File System Objects beyond a single serialisation pass
diff --git a/doc/manual/source/store/math-notation.md b/doc/manual/source/store/math-notation.md
new file mode 100644
index 000000000..723982e73
--- /dev/null
+++ b/doc/manual/source/store/math-notation.md
@@ -0,0 +1,16 @@
+# Appendix: Math notation
+
+A few times in this manual, formal "proof trees" are used for [natural deduction](https://en.wikipedia.org/wiki/Natural_deduction)-style definition of various [relations](https://en.wikipedia.org/wiki/Relation_(mathematics)).
+
+The following grammar and assignment of metavariables to syntactic categories is used in these sections.
+
+\\begin{align}
+s, t &\in \text{store-path} \\\\
+o &\in \text{output-name} \\\\
+i, p &\in \text{deriving-path} \\\\
+d &\in \text{derivation}
+\\end{align}
+
+\\begin{align}
+\text{deriving-path} \quad p &::= s \mid (p, o)
+\\end{align}
diff --git a/doc/manual/source/store/resolution.md b/doc/manual/source/store/resolution.md
new file mode 100644
index 000000000..9a87fea99
--- /dev/null
+++ b/doc/manual/source/store/resolution.md
@@ -0,0 +1,219 @@
+# Derivation Resolution
+
+*See [this chapter's appendix](@docroot@/store/math-notation.md) on grammar and metavariable conventions.*
+
+To *resolve* a derivation is to replace its [inputs] with the simplest inputs — plain store paths — that denote the same store objects.
+
+Derivations that only have store paths as inputs are likewise called *resolved derivations*.
+(They are called that whether they are in fact the output of derivation resolution, or just made that way without non-store-path inputs to begin with.)
+
+## Input Content Equivalence of Derivations
+
+[Deriving paths][deriving-path] intentionally make it possible to refer to the same [store object] in multiple ways.
+This is a consequence of content-addressing, since different derivations can produce the same outputs, and the same data can also be manually added to the store.
+This is also a consequence even of input-addressing, as an output can be referred to by derivation and output name, or directly by its [computed](./derivation/outputs/input-address.md) store path.
+Since dereferencing deriving paths is thus not injective, it induces an equivalence relation on deriving paths.
+
+Let's call this equivalence relation \\(\\sim\\), where \\(p_1 \\sim p_2\\) means that deriving paths \\(p_1\\) and \\(p_2\\) refer to the same store object.
+
+**Content Equivalence**: Two deriving paths are equivalent if they refer to the same store object:
+
+\\[
+\\begin{prooftree}
+\\AxiomC{${}^*p_1 = {}^*p_2$}
+\\UnaryInfC{$p_1 \\,\\sim_\\mathrm{DP}\\, p_2$}
+\\end{prooftree}
+\\]
+
+where \\({}^\*p\\) denotes the store object that deriving path \\(p\\) refers to.
+
+This also induces an equivalence relation on sets of deriving paths:
+
+\\[
+\\begin{prooftree}
+\\AxiomC{$\\{ {}^*p | p \\in P_1 \\} = \\{ {}^*p | p \\in P_2 \\}$}
+\\UnaryInfC{$P_1 \\,\\sim_{\\mathcal{P}(\\mathrm{DP})}\\, P_2$}
+\\end{prooftree}
+\\]
+
+**Input Content Equivalence**: This, in turn, induces an equivalence relation on derivations: two derivations are equivalent if their inputs are equivalent, and they are otherwise equal:
+
+\\[
+\\begin{prooftree}
+\\AxiomC{$\\mathrm{inputs}(d_1) \\,\\sim_{\\mathcal{P}(\\mathrm{DP})}\\, \\mathrm{inputs}(d_2)$}
+\\AxiomC{$
+ d\_1\left[\\mathrm{inputs} := \\{\\}\right]
+ \=
+ d\_2\left[\\mathrm{inputs} := \\{\\}\right]
+$}
+\\BinaryInfC{$d_1 \\,\\sim_\\mathrm{Drv}\\, d_2$}
+\\end{prooftree}
+\\]
+
+Derivation resolution always maps derivations to input-content-equivalent derivations.
+
+## Resolution relation
+
+Dereferencing a derived path — \\({}^\*p\\) above — was just introduced as a black box.
+But actually it is a multi-step process of looking up build results in the [build trace] that itself depends on resolving the lookup keys.
+Resolution is thus a recursive multi-step process that is worth diagramming formally.
+
+We can do this with a small-step binary transition relation; let's call it \\(\rightsquigarrow\\).
+We can then conclude dereferenced equality like this:
+
+\\[
+\\begin{prooftree}
+\\AxiomC{$p\_1 \\rightsquigarrow^* p$}
+\\AxiomC{$p\_2 \\rightsquigarrow^* p$}
+\\BinaryInfC{${}^*p\_1 = {}^*p\_2$}
+\\end{prooftree}
+\\]
+
+I.e. by showing that both original items resolve (over 0 or more small steps, hence the \\({}^*\\)) to the same exact item.
+
+With this motivation, let's now formalize a [small-step](https://en.wikipedia.org/wiki/Operational_semantics#Small-step_semantics) system of reduction rules for resolution.
+
+### Formal rules
+
+### \\(\text{resolved}\\) unary relation
+
+\\[
+\\begin{prooftree}
+\\AxiomC{$s \in \text{store-path}$}
+\\UnaryInfC{$s$ resolved}
+\\end{prooftree}
+\\]
+
+\\[
+\\begin{prooftree}
+\\AxiomC{$\forall i \in \mathrm{inputs}(d). i \text{ resolved}$}
+\\UnaryInfC{$d$ resolved}
+\\end{prooftree}
+\\]
+
+### \\(\rightsquigarrow\\) binary relation
+
+> **Remark**
+>
+> Actually, to be completely formal we would need to keep track of the build trace we are choosing to resolve against.
+>
+> We could do that by making \\(\rightsquigarrow\\) a ternary relation, which would pass the build trace to itself until it finally uses it in that one rule.
+> This would add clutter more than insight, so we didn't bother to write it.
+>
+> There are other options too, like saying the whole reduction rule system is parameterized on the build trace, essentially [currying](https://en.wikipedia.org/wiki/Currying) the ternary \\(\rightsquigarrow\\) into a function from build traces to the binary relation written above.
+
+#### Core build trace lookup rule
+
+\\[
+\\begin{prooftree}
+\\AxiomC{$s \in \text{store-path}$}
+\\AxiomC{${}^*s \in \text{derivation}$}
+\\AxiomC{${}^*s$ resolved}
+\\AxiomC{$\mathrm{build\text{-}trace}[s][o] = t$}
+\\QuaternaryInfC{$(s, o) \rightsquigarrow t$}
+\\RightLabel{\\scriptsize output path resolution}
+\\end{prooftree}
+\\]
+
+#### Inductive rules
+
+\\[
+\\begin{prooftree}
+\\AxiomC{$i \\rightsquigarrow i'$}
+\\AxiomC{$i \\in \\mathrm{inputs}(d)$}
+\\BinaryInfC{$d \\rightsquigarrow d[i \\mapsto i']$}
+\\end{prooftree}
+\\]
+
+\\[
+\\begin{prooftree}
+\\AxiomC{$d \\rightsquigarrow d'$}
+\\UnaryInfC{$(\\mathrm{path}(d), o) \\rightsquigarrow (\\mathrm{path}(d'), o)$}
+\\end{prooftree}
+\\]
+
+\\[
+\\begin{prooftree}
+\\AxiomC{$p \\rightsquigarrow p'$}
+\\UnaryInfC{$(p, o) \\rightsquigarrow (p', o)$}
+\\end{prooftree}
+\\]
+
+### Properties
+
+Like all well-behaved evaluation relations, partial resolution is [*confluent*](https://en.wikipedia.org/wiki/Confluence_(abstract_rewriting)).
+Also, if we take the symmetric closure of \\(\\rightsquigarrow^\*\\), we end up with the equivalence relations of the previous section.
+Resolution respects content equivalence for deriving paths, and input content equivalence for derivations.
+
+> **Remark**
+>
+> We chose to define from scratch an "resolved" unary relation explicitly above.
+> But it can also be defined as the normal forms of the \\(\\rightsquigarrow^\*\\) relation:
+>
+> \\[ a \text{ resolved} \Leftrightarrow \forall b. b \rightsquigarrow^* a \Rightarrow b = a\\]
+>
+> In prose, resolved terms are terms which \\(\\rightsquigarrow^\*\\) only relates on the left side to the same term on the right side; they are the terms which can be resolved no further.
+
+## Partial versus Complete Resolution
+
+Similar to evaluation, we can also speak of *partial* versus *complete* derivation resolution.
+Partial derivation resolution is what we've actually formalized above with \\(\\rightsquigarrow^\*\\).
+Complete resolution is resolution ending in a resolved term (deriving path or derivation).
+(Which is a normal form of the relation, per the remark above.)
+
+With partial resolution, a derivation is related to equivalent derivations with the same or simpler inputs, but not all those inputs will be plain store paths.
+This is useful when the input refers to a floating content addressed output we have not yet built — we don't know what (content-address) store path will used for that derivation, so we are "stuck" trying to resolve the deriving path in question.
+(In the above formalization, this happens when the build trace is missing the keys we wish to look up in it.)
+
+Complete resolution is a *functional* relation, i.e. values on the left are uniquely related with values on the right.
+It is not however, a *total* relation (in general, assuming arbitrary build traces).
+This is discussed in the next section.
+
+## Termination
+
+For static derivations graphs, complete resolution is indeed total, because it always terminates for all inputs.
+(A relation that is both total and functional is a function.)
+
+For [dynamic][xp-feature-dynamic-derivations] derivation graphs, however, this is not the case — resolution is not guaranteed to terminate.
+The issue isn't rewriting deriving paths themselves:
+a single rewrite to normalize an output deriving path to a constant one always exists, and always proceeds in one step.
+The issue is that dynamic derivations (i.e. those that are filled-in the graph by a previous resolution) may have more transitive dependencies than the original derivation.
+
+> **Example**
+>
+> Suppose we have this deriving path
+> ```json
+> {
+> "drvPath": {
+> "drvPath": "...-foo.drv",
+> "output": "bar.drv"
+> },
+> "output": "baz"
+> }
+> ```
+> and derivation `foo` is already resolved.
+> When we resolve deriving path we'll end up with something like.
+> ```json
+> {
+> "drvPath": "...-foo-bar.drv",
+> "output": "baz"
+> }
+> ```
+> So far is just an atomic single rewrite, with no termination issues.
+> But the derivation `foo-bar` may have its *own* dynamic derivation inputs.
+> Resolution must resolve that derivation first before the above deriving path can finally be normalized to a plain `...-foo-bar-baz` store path.
+
+The important thing to notice is that while "build trace" *keys* must be resolved.
+The *value* those keys are mapped to have no such constraints.
+An arbitrary store object has no notion of being resolved or not.
+But, an arbitrary store object can be read back as a derivation (as will in fact be done in case for dynamic derivations / nested output deriving paths).
+And those derivations need *not* be resolved.
+
+It is those dynamic non-resolved derivations which are the source of non-termination.
+By the same token, they are also the reason why dynamic derivations offer greater expressive power.
+
+[store object]: @docroot@/store/store-object.md
+[inputs]: @docroot@/store/derivation/index.md#inputs
+[build trace]: @docroot@/store/build-trace.md
+[deriving-path]: @docroot@/store/derivation/index.md#deriving-path
+[xp-feature-dynamic-derivations]: @docroot@/development/experimental-features.md#xp-feature-dynamic-derivations
diff --git a/doc/manual/source/store/store-object/content-address.md b/doc/manual/source/store/store-object/content-address.md
index 36e841fa3..7834ac510 100644
--- a/doc/manual/source/store/store-object/content-address.md
+++ b/doc/manual/source/store/store-object/content-address.md
@@ -1,7 +1,7 @@
# Content-Addressing Store Objects
Just [like][fso-ca] [File System Objects][File System Object],
-[Store Objects][Store Object] can also be [content-addressed](@docroot@/glossary.md#gloss-content-addressed),
+[Store Objects][Store Object] can also be [content-addressed](@docroot@/glossary.md#gloss-content-address),
unless they are [input-addressed](@docroot@/glossary.md#gloss-input-addressed-store-object).
For store objects, the content address we produce will take the form of a [Store Path] rather than regular hash.
@@ -107,7 +107,7 @@ References (to other store objects and self-references alike) are supported so l
>
> This method is part of the [`git-hashing`][xp-feature-git-hashing] experimental feature.
-This uses the corresponding [Git](../file-system-object/content-address.md#serial-git) method of file system object content addressing.
+This uses the corresponding [Git](../file-system-object/content-address.md#git) method of file system object content addressing.
References are not supported.
diff --git a/doc/manual/source/store/store-path.md b/doc/manual/source/store/store-path.md
index beec2389b..4061f3653 100644
--- a/doc/manual/source/store/store-path.md
+++ b/doc/manual/source/store/store-path.md
@@ -6,7 +6,7 @@
>
> A rendered store path
-Nix implements references to [store objects](./index.md#store-object) as *store paths*.
+Nix implements references to [store objects](./store-object.md) as *store paths*.
Think of a store path as an [opaque], [unique identifier]:
The only way to obtain store path is by adding or building store objects.
diff --git a/doc/manual/theme/head.hbs b/doc/manual/theme/head.hbs
new file mode 100644
index 000000000..e514a9977
--- /dev/null
+++ b/doc/manual/theme/head.hbs
@@ -0,0 +1,15 @@
+
+
+
diff --git a/flake.lock b/flake.lock
index cc2b2f27e..63290ef86 100644
--- a/flake.lock
+++ b/flake.lock
@@ -63,16 +63,16 @@
},
"nixpkgs": {
"locked": {
- "lastModified": 1756178832,
- "narHash": "sha256-O2CIn7HjZwEGqBrwu9EU76zlmA5dbmna7jL1XUmAId8=",
+ "lastModified": 1761597516,
+ "narHash": "sha256-wxX7u6D2rpkJLWkZ2E932SIvDJW8+ON/0Yy8+a5vsDU=",
"owner": "NixOS",
"repo": "nixpkgs",
- "rev": "d98ce345cdab58477ca61855540999c86577d19d",
+ "rev": "daf6dc47aa4b44791372d6139ab7b25269184d55",
"type": "github"
},
"original": {
"owner": "NixOS",
- "ref": "nixos-25.05-small",
+ "ref": "nixos-25.05",
"repo": "nixpkgs",
"type": "github"
}
diff --git a/flake.nix b/flake.nix
index fd623c807..08f518983 100644
--- a/flake.nix
+++ b/flake.nix
@@ -1,7 +1,7 @@
{
description = "The purely functional package manager";
- inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05-small";
+ inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05";
inputs.nixpkgs-regression.url = "github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2";
inputs.nixpkgs-23-11.url = "github:NixOS/nixpkgs/a62e6edd6d5e1fa0329b8653c801147986f8d446";
@@ -413,6 +413,14 @@
supportsCross = false;
};
+ "nix-json-schema-checks" = {
+ supportsCross = false;
+ };
+
+ "nix-kaitai-struct-checks" = {
+ supportsCross = false;
+ };
+
"nix-perl-bindings" = {
supportsCross = false;
};
@@ -467,6 +475,27 @@
}
);
+ apps = forAllSystems (
+ system:
+ let
+ pkgs = nixpkgsFor.${system}.native;
+ opener = if pkgs.stdenv.isDarwin then "open" else "xdg-open";
+ in
+ {
+ open-manual = {
+ type = "app";
+ program = "${pkgs.writeShellScript "open-nix-manual" ''
+ path="${self.packages.${system}.nix-manual.site}/index.html"
+ if ! ${opener} "$path"; then
+ echo "Failed to open manual with ${opener}. Manual is located at:"
+ echo "$path"
+ fi
+ ''}";
+ meta.description = "Open the Nix manual in your browser";
+ };
+ }
+ );
+
devShells =
let
makeShell = import ./packaging/dev-shell.nix { inherit lib devFlake; };
diff --git a/meson.build b/meson.build
index 736756157..f3158ea6d 100644
--- a/meson.build
+++ b/meson.build
@@ -60,3 +60,4 @@ if get_option('unit-tests')
subproject('libflake-tests')
endif
subproject('nix-functional-tests')
+subproject('json-schema-checks')
diff --git a/misc/zsh/completion.zsh b/misc/zsh/completion.zsh
index eb26a16cb..6146455fe 100644
--- a/misc/zsh/completion.zsh
+++ b/misc/zsh/completion.zsh
@@ -1,5 +1,5 @@
-# shellcheck disable=all
#compdef nix
+# shellcheck disable=all
function _nix() {
local ifs_bk="$IFS"
diff --git a/nix-meson-build-support/common/meson.build b/nix-meson-build-support/common/meson.build
index 99bfbd486..1405974d2 100644
--- a/nix-meson-build-support/common/meson.build
+++ b/nix-meson-build-support/common/meson.build
@@ -42,8 +42,8 @@ if cxx.get_id() == 'clang'
add_project_arguments('-fpch-instantiate-templates', language : 'cpp')
endif
-# Darwin ld doesn't like "X.Y.Zpre"
-nix_soversion = meson.project_version().split('pre')[0]
+# Darwin ld doesn't like "X.Y.ZpreABCD+W"
+nix_soversion = meson.project_version().split('+')[0].split('pre')[0]
subdir('assert-fail')
subdir('asan-options')
diff --git a/packaging/components.nix b/packaging/components.nix
index c621b7073..bbd6208b9 100644
--- a/packaging/components.nix
+++ b/packaging/components.nix
@@ -438,6 +438,16 @@ in
*/
nix-external-api-docs = callPackage ../src/external-api-docs/package.nix { version = fineVersion; };
+ /**
+ JSON schema validation checks
+ */
+ nix-json-schema-checks = callPackage ../src/json-schema-checks/package.nix { };
+
+ /**
+ Kaitai struct schema validation checks
+ */
+ nix-kaitai-struct-checks = callPackage ../src/kaitai-struct-checks/package.nix { };
+
nix-perl-bindings = callPackage ../src/perl/package.nix { };
/**
diff --git a/packaging/dev-shell.nix b/packaging/dev-shell.nix
index 5fb4f14d2..153e7a3eb 100644
--- a/packaging/dev-shell.nix
+++ b/packaging/dev-shell.nix
@@ -108,6 +108,7 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
++ pkgs.nixComponents2.nix-internal-api-docs.nativeBuildInputs
++ pkgs.nixComponents2.nix-external-api-docs.nativeBuildInputs
++ pkgs.nixComponents2.nix-functional-tests.externalNativeBuildInputs
+ ++ pkgs.nixComponents2.nix-json-schema-checks.externalNativeBuildInputs
++ lib.optional (
!buildCanExecuteHost
# Hack around https://github.com/nixos/nixpkgs/commit/bf7ad8cfbfa102a90463433e2c5027573b462479
diff --git a/packaging/hydra.nix b/packaging/hydra.nix
index bc75b5dfb..67e2c0dfd 100644
--- a/packaging/hydra.nix
+++ b/packaging/hydra.nix
@@ -62,6 +62,8 @@ let
"nix-cmd"
"nix-cli"
"nix-functional-tests"
+ "nix-json-schema-checks"
+ "nix-kaitai-struct-checks"
]
++ lib.optionals enableBindings [
"nix-perl-bindings"
diff --git a/src/external-api-docs/README.md b/src/external-api-docs/README.md
index 8760ac88b..1940cc1c0 100644
--- a/src/external-api-docs/README.md
+++ b/src/external-api-docs/README.md
@@ -15,7 +15,7 @@ programmatically:
1. Embedding the evaluator
2. Writing language plug-ins
-Embedding means you link the Nix C libraries in your program and use them from
+Embedding means you link the Nix C API libraries in your program and use them from
there. Adding a plug-in means you make a library that gets loaded by the Nix
language evaluator, specified through a configuration option.
diff --git a/src/json-schema-checks/.version b/src/json-schema-checks/.version
new file mode 120000
index 000000000..b7badcd0c
--- /dev/null
+++ b/src/json-schema-checks/.version
@@ -0,0 +1 @@
+../../.version
\ No newline at end of file
diff --git a/src/json-schema-checks/content-address b/src/json-schema-checks/content-address
new file mode 120000
index 000000000..194a265a1
--- /dev/null
+++ b/src/json-schema-checks/content-address
@@ -0,0 +1 @@
+../../src/libstore-tests/data/content-address
\ No newline at end of file
diff --git a/src/json-schema-checks/derivation b/src/json-schema-checks/derivation
new file mode 120000
index 000000000..3dc1cbe06
--- /dev/null
+++ b/src/json-schema-checks/derivation
@@ -0,0 +1 @@
+../../src/libstore-tests/data/derivation
\ No newline at end of file
diff --git a/src/json-schema-checks/deriving-path b/src/json-schema-checks/deriving-path
new file mode 120000
index 000000000..4f50b2ee9
--- /dev/null
+++ b/src/json-schema-checks/deriving-path
@@ -0,0 +1 @@
+../../src/libstore-tests/data/derived-path
\ No newline at end of file
diff --git a/src/json-schema-checks/hash b/src/json-schema-checks/hash
new file mode 120000
index 000000000..d68763879
--- /dev/null
+++ b/src/json-schema-checks/hash
@@ -0,0 +1 @@
+../../src/libutil-tests/data/hash
\ No newline at end of file
diff --git a/src/json-schema-checks/meson.build b/src/json-schema-checks/meson.build
new file mode 100644
index 000000000..67f553162
--- /dev/null
+++ b/src/json-schema-checks/meson.build
@@ -0,0 +1,204 @@
+# Run with:
+# meson test --suite json-schema
+# Run with: (without shell / configure)
+# nix build .#nix-json-schema-checks
+
+project(
+ 'nix-json-schema-checks',
+ version : files('.version'),
+ meson_version : '>= 1.1',
+ license : 'LGPL-2.1-or-later',
+)
+
+fs = import('fs')
+
+# Note: The 'jsonschema' package provides the 'jv' command
+jv = find_program('jv', required : true)
+
+# The schema directory is a committed symlink to the actual schema location
+schema_dir = meson.current_source_dir() / 'schema'
+
+# Get all example files
+schemas = [
+ {
+ 'stem' : 'hash',
+ 'schema' : schema_dir / 'hash-v1.yaml',
+ 'files' : [
+ 'sha256-base64.json',
+ 'sha256-base16.json',
+ 'sha256-nix32.json',
+ 'blake3-base64.json',
+ ],
+ },
+ {
+ 'stem' : 'content-address',
+ 'schema' : schema_dir / 'content-address-v1.yaml',
+ 'files' : [
+ 'text.json',
+ 'nar.json',
+ ],
+ },
+ {
+ 'stem' : 'store-path',
+ 'schema' : schema_dir / 'store-path-v1.yaml',
+ 'files' : [
+ 'simple.json',
+ ],
+ },
+ {
+ 'stem' : 'deriving-path',
+ 'schema' : schema_dir / 'deriving-path-v1.yaml',
+ 'files' : [
+ 'single_opaque.json',
+ 'single_built.json',
+ 'single_built_built.json',
+ ],
+ },
+]
+
+# Derivation and Derivation output
+schemas += [
+ # Match overall
+ {
+ 'stem' : 'derivation',
+ 'schema' : schema_dir / 'derivation-v3.yaml',
+ 'files' : [
+ 'dyn-dep-derivation.json',
+ 'simple-derivation.json',
+ ],
+ },
+ {
+ 'stem' : 'derivation',
+ 'schema' : schema_dir / 'derivation-v3.yaml#/$defs/output/overall',
+ 'files' : [
+ 'output-caFixedFlat.json',
+ 'output-caFixedNAR.json',
+ 'output-caFixedText.json',
+ 'output-caFloating.json',
+ 'output-deferred.json',
+ 'output-impure.json',
+ 'output-inputAddressed.json',
+ ],
+ },
+ # Match exact variant
+ {
+ 'stem' : 'derivation',
+ 'schema' : schema_dir / 'derivation-v3.yaml#/$defs/output/inputAddressed',
+ 'files' : [
+ 'output-inputAddressed.json',
+ ],
+ },
+ {
+ 'stem' : 'derivation',
+ 'schema' : schema_dir / 'derivation-v3.yaml#/$defs/output/caFixed',
+ 'files' : [
+ 'output-caFixedFlat.json',
+ 'output-caFixedNAR.json',
+ 'output-caFixedText.json',
+ ],
+ },
+ {
+ 'stem' : 'derivation',
+ 'schema' : schema_dir / 'derivation-v3.yaml#/$defs/output/caFloating',
+ 'files' : [
+ 'output-caFloating.json',
+ ],
+ },
+ {
+ 'stem' : 'derivation',
+ 'schema' : schema_dir / 'derivation-v3.yaml#/$defs/output/deferred',
+ 'files' : [
+ 'output-deferred.json',
+ ],
+ },
+ {
+ 'stem' : 'derivation',
+ 'schema' : schema_dir / 'derivation-v3.yaml#/$defs/output/impure',
+ 'files' : [
+ 'output-impure.json',
+ ],
+ },
+]
+
+# Store object info
+schemas += [
+ # Match overall
+ {
+ 'stem' : 'store-object-info',
+ 'schema' : schema_dir / 'store-object-info-v1.yaml',
+ 'files' : [
+ 'pure.json',
+ 'impure.json',
+ 'empty_pure.json',
+ 'empty_impure.json',
+ ],
+ },
+ {
+ 'stem' : 'nar-info',
+ 'schema' : schema_dir / 'store-object-info-v1.yaml',
+ 'files' : [
+ 'pure.json',
+ 'impure.json',
+ ],
+ },
+ # Match exact variant
+ {
+ 'stem' : 'store-object-info',
+ 'schema' : schema_dir / 'store-object-info-v1.yaml#/$defs/base',
+ 'files' : [
+ 'pure.json',
+ 'empty_pure.json',
+ ],
+ },
+ {
+ 'stem' : 'store-object-info',
+ 'schema' : schema_dir / 'store-object-info-v1.yaml#/$defs/impure',
+ 'files' : [
+ 'impure.json',
+ 'empty_impure.json',
+ ],
+ },
+ {
+ 'stem' : 'nar-info',
+ 'schema' : schema_dir / 'store-object-info-v1.yaml#/$defs/base',
+ 'files' : [
+ 'pure.json',
+ ],
+ },
+ {
+ 'stem' : 'nar-info',
+ 'schema' : schema_dir / 'store-object-info-v1.yaml#/$defs/narInfo',
+ 'files' : [
+ 'impure.json',
+ ],
+ },
+]
+
+# Validate each example against the schema
+foreach schema : schemas
+ stem = schema['stem']
+ schema_file = schema['schema']
+ if '#' not in schema_file
+ # Validate the schema itself against JSON Schema Draft 04
+ test(
+ stem + '-schema-valid',
+ jv,
+ args : [
+ 'http://json-schema.org/draft-04/schema',
+ schema_file,
+ ],
+ suite : 'json-schema',
+ )
+ endif
+ foreach example : schema['files']
+ test(
+ stem + '-example-' + fs.stem(example),
+ jv,
+ args : [
+ schema_file,
+ files(stem / example),
+ ],
+ suite : 'json-schema',
+ )
+ endforeach
+endforeach
diff --git a/src/json-schema-checks/nar-info b/src/json-schema-checks/nar-info
new file mode 120000
index 000000000..0ba4d5870
--- /dev/null
+++ b/src/json-schema-checks/nar-info
@@ -0,0 +1 @@
+../../src/libstore-tests/data/nar-info
\ No newline at end of file
diff --git a/src/json-schema-checks/package.nix b/src/json-schema-checks/package.nix
new file mode 100644
index 000000000..160db003f
--- /dev/null
+++ b/src/json-schema-checks/package.nix
@@ -0,0 +1,56 @@
+# Run with: nix build .#nix-json-schema-checks
+{
+ lib,
+ mkMesonDerivation,
+
+ meson,
+ ninja,
+ jsonschema,
+
+ # Configuration Options
+
+ version,
+}:
+
+mkMesonDerivation (finalAttrs: {
+ pname = "nix-json-schema-checks";
+ inherit version;
+
+ workDir = ./.;
+ fileset = lib.fileset.unions [
+ ../../.version
+ ../../doc/manual/source/protocols/json/schema
+ ../../src/libutil-tests/data/hash
+ ../../src/libstore-tests/data/content-address
+ ../../src/libstore-tests/data/store-path
+ ../../src/libstore-tests/data/derivation
+ ../../src/libstore-tests/data/derived-path
+ ../../src/libstore-tests/data/path-info
+ ../../src/libstore-tests/data/nar-info
+ ./.
+ ];
+
+ outputs = [ "out" ];
+
+ passthru.externalNativeBuildInputs = [
+ jsonschema
+ ];
+
+ nativeBuildInputs = [
+ meson
+ ninja
+ ]
+ ++ finalAttrs.passthru.externalNativeBuildInputs;
+
+ doCheck = true;
+
+ mesonCheckFlags = [ "--print-errorlogs" ];
+
+ postInstall = ''
+ touch $out
+ '';
+
+ meta = {
+ platforms = lib.platforms.all;
+ };
+})
diff --git a/src/json-schema-checks/schema b/src/json-schema-checks/schema
new file mode 120000
index 000000000..473e47b1b
--- /dev/null
+++ b/src/json-schema-checks/schema
@@ -0,0 +1 @@
+../../doc/manual/source/protocols/json/schema
\ No newline at end of file
diff --git a/src/json-schema-checks/store-object-info b/src/json-schema-checks/store-object-info
new file mode 120000
index 000000000..a3c9e07c4
--- /dev/null
+++ b/src/json-schema-checks/store-object-info
@@ -0,0 +1 @@
+../../src/libstore-tests/data/path-info
\ No newline at end of file
diff --git a/src/json-schema-checks/store-path b/src/json-schema-checks/store-path
new file mode 120000
index 000000000..003b1dbbb
--- /dev/null
+++ b/src/json-schema-checks/store-path
@@ -0,0 +1 @@
+../../src/libstore-tests/data/store-path
\ No newline at end of file
diff --git a/src/kaitai-struct-checks/.version b/src/kaitai-struct-checks/.version
new file mode 120000
index 000000000..b7badcd0c
--- /dev/null
+++ b/src/kaitai-struct-checks/.version
@@ -0,0 +1 @@
+../../.version
\ No newline at end of file
diff --git a/src/kaitai-struct-checks/meson.build b/src/kaitai-struct-checks/meson.build
new file mode 100644
index 000000000..f705a6744
--- /dev/null
+++ b/src/kaitai-struct-checks/meson.build
@@ -0,0 +1,77 @@
+# Run with:
+# meson test --suite kaitai-struct
+# Run with: (without shell / configure)
+# nix build .#nix-kaitai-struct-checks
+
+project(
+ 'nix-kaitai-struct-checks',
+ 'cpp',
+ version : files('.version'),
+ default_options : [
+ 'cpp_std=c++23',
+ # TODO(Qyriad): increase the warning level
+ 'warning_level=1',
+ 'errorlogs=true', # Please print logs for tests that fail
+ ],
+ meson_version : '>= 1.1',
+ license : 'LGPL-2.1-or-later',
+)
+
+kaitai_runtime_dep = dependency('kaitai-struct-cpp-stl-runtime', required : true)
+gtest_dep = dependency('gtest')
+gtest_main_dep = dependency('gtest_main', required : true)
+
+# Find the Kaitai Struct compiler
+ksc = find_program('ksc', required : true)
+
+kaitai_generated_srcs = custom_target(
+ 'kaitai-generated-sources',
+ input : [ 'nar.ksy' ],
+ output : [ 'nix_nar.cpp', 'nix_nar.h' ],
+ command : [
+ ksc,
+ '@INPUT@',
+ '--target', 'cpp_stl',
+ '--outdir',
+ meson.current_build_dir(),
+ ],
+)
+
+nar_kaitai_lib = library(
+ 'nix-nar-kaitai-lib',
+ kaitai_generated_srcs,
+ dependencies : [ kaitai_runtime_dep ],
+ install : true,
+)
+
+nar_kaitai_dep = declare_dependency(
+ link_with : nar_kaitai_lib,
+ sources : kaitai_generated_srcs[1],
+)
+
+# The nar directory is a committed symlink to the actual nars location
+nars_dir = meson.current_source_dir() / 'nars'
+
+# Get all example files
+nars = [
+ 'dot.nar',
+]
+
+test_deps = [
+ nar_kaitai_dep,
+ kaitai_runtime_dep,
+ gtest_main_dep,
+]
+
+this_exe = executable(
+ meson.project_name(),
+ 'test-parse-nar.cc',
+ dependencies : test_deps,
+)
+
+test(
+ meson.project_name(),
+ this_exe,
+ env : [ 'NIX_NARS_DIR=' + nars_dir ],
+ protocol : 'gtest',
+)
diff --git a/src/kaitai-struct-checks/nar.ksy b/src/kaitai-struct-checks/nar.ksy
new file mode 120000
index 000000000..c3a79a3b6
--- /dev/null
+++ b/src/kaitai-struct-checks/nar.ksy
@@ -0,0 +1 @@
+../../doc/manual/source/protocols/nix-archive/nar.ksy
\ No newline at end of file
diff --git a/src/kaitai-struct-checks/nars b/src/kaitai-struct-checks/nars
new file mode 120000
index 000000000..ed0b4ecc7
--- /dev/null
+++ b/src/kaitai-struct-checks/nars
@@ -0,0 +1 @@
+../libutil-tests/data/nars
\ No newline at end of file
diff --git a/src/kaitai-struct-checks/nix-meson-build-support b/src/kaitai-struct-checks/nix-meson-build-support
new file mode 120000
index 000000000..0b140f56b
--- /dev/null
+++ b/src/kaitai-struct-checks/nix-meson-build-support
@@ -0,0 +1 @@
+../../nix-meson-build-support
\ No newline at end of file
diff --git a/src/kaitai-struct-checks/package.nix b/src/kaitai-struct-checks/package.nix
new file mode 100644
index 000000000..97d56aabd
--- /dev/null
+++ b/src/kaitai-struct-checks/package.nix
@@ -0,0 +1,76 @@
+# Run with: nix build .#nix-kaitai-struct-checks
+# or: `nix develop .#nix-kaitai-struct-checks` to enter a dev shell
+{
+ lib,
+ mkMesonDerivation,
+ gtest,
+ meson,
+ ninja,
+ pkg-config,
+ kaitai-struct-compiler,
+ fetchzip,
+ kaitai-struct-cpp-stl-runtime,
+ # Configuration Options
+ version,
+}:
+let
+ inherit (lib) fileset;
+in
+mkMesonDerivation (finalAttrs: {
+ pname = "nix-kaitai-struct-checks";
+ inherit version;
+
+ workDir = ./.;
+ fileset = lib.fileset.unions [
+ ../../nix-meson-build-support
+ ./nix-meson-build-support
+ ./.version
+ ../../.version
+ ../../doc/manual/source/protocols/nix-archive/nar.ksy
+ ./nars
+ ../../src/libutil-tests/data
+ ./meson.build
+ ./nar.ksy
+ (fileset.fileFilter (file: file.hasExt "cc") ./.)
+ (fileset.fileFilter (file: file.hasExt "hh") ./.)
+ ];
+
+ outputs = [ "out" ];
+
+ passthru.externalNativeBuildInputs = [
+ # This can go away when we bump up to 25.11
+ (kaitai-struct-compiler.overrideAttrs (finalAttrs: {
+ version = "0.11";
+ src = fetchzip {
+ url = "https://github.com/kaitai-io/kaitai_struct_compiler/releases/download/${version}/kaitai-struct-compiler-${version}.zip";
+ sha256 = "sha256-j9TEilijqgIiD0GbJfGKkU1FLio9aTopIi1v8QT1b+A=";
+ };
+ }))
+ ];
+
+ passthru.externalBuildInputs = [
+ gtest
+ kaitai-struct-cpp-stl-runtime
+ ];
+
+ buildInputs = finalAttrs.passthru.externalBuildInputs;
+
+ nativeBuildInputs = [
+ meson
+ ninja
+ pkg-config
+ ]
+ ++ finalAttrs.passthru.externalNativeBuildInputs;
+
+ doCheck = true;
+
+ mesonCheckFlags = [ "--print-errorlogs" ];
+
+ postInstall = ''
+ touch $out
+ '';
+
+ meta = {
+ platforms = lib.platforms.all;
+ };
+})
diff --git a/src/kaitai-struct-checks/test-parse-nar.cc b/src/kaitai-struct-checks/test-parse-nar.cc
new file mode 100644
index 000000000..456ffb127
--- /dev/null
+++ b/src/kaitai-struct-checks/test-parse-nar.cc
@@ -0,0 +1,48 @@
+#include
+#include
+#include
+#include
+#include
+
+#include
+
+#include
+#include
+#include
+
+#include "nix_nar.h"
+
+static const std::vector NarFiles = {
+ "empty.nar",
+ "dot.nar",
+ "dotdot.nar",
+ "executable-after-contents.nar",
+ "invalid-tag-instead-of-contents.nar",
+ "name-after-node.nar",
+ "nul-character.nar",
+ "slash.nar",
+};
+
+class NarParseTest : public ::testing::TestWithParam
+{};
+
+TEST_P(NarParseTest, ParseSucceeds)
+{
+ const auto nar_file = GetParam();
+
+ const char * nars_dir_env = std::getenv("NIX_NARS_DIR");
+ if (nars_dir_env == nullptr) {
+ FAIL() << "NIX_NARS_DIR environment variable not set.";
+ }
+
+ const std::filesystem::path nar_file_path = std::filesystem::path(nars_dir_env) / "dot.nar";
+ ASSERT_TRUE(std::filesystem::exists(nar_file_path)) << "Missing test file: " << nar_file_path;
+
+ std::ifstream ifs(nar_file_path, std::ifstream::binary);
+ ASSERT_TRUE(ifs.is_open()) << "Failed to open file: " << nar_file;
+ kaitai::kstream ks(&ifs);
+ nix_nar_t nar(&ks);
+ ASSERT_TRUE(nar.root_node() != nullptr) << "Failed to parse NAR file: " << nar_file;
+}
+
+INSTANTIATE_TEST_SUITE_P(AllNarFiles, NarParseTest, ::testing::ValuesIn(NarFiles));
diff --git a/src/libexpr-c/nix_api_expr.h b/src/libexpr-c/nix_api_expr.h
index 2be739955..3623ee076 100644
--- a/src/libexpr-c/nix_api_expr.h
+++ b/src/libexpr-c/nix_api_expr.h
@@ -4,11 +4,14 @@
* @brief Bindings to the Nix language evaluator
*
* See *[Embedding the Nix Evaluator](@ref nix_evaluator_example)* for an example.
- * @{
*/
/** @file
* @brief Main entry for the libexpr C bindings
*/
+/** @defgroup libexpr_init Initialization
+ * @ingroup libexpr
+ * @{
+ */
#include "nix_api_store.h"
#include "nix_api_util.h"
@@ -45,7 +48,10 @@ typedef struct nix_eval_state_builder nix_eval_state_builder;
*/
typedef struct EvalState EvalState; // nix::EvalState
+/** @} */
+
/** @brief A Nix language value, or thunk that may evaluate to a value.
+ * @ingroup value
*
* Values are the primary objects manipulated in the Nix language.
* They are considered to be immutable from a user's perspective, but the process of evaluating a value changes its
@@ -56,7 +62,8 @@ typedef struct EvalState EvalState; // nix::EvalState
*
* The evaluator manages its own memory, but your use of the C API must follow the reference counting rules.
*
- * @see value_manip
+ * @struct nix_value
+ * @see value_create, value_extract
* @see nix_value_incref, nix_value_decref
*/
typedef struct nix_value nix_value;
@@ -65,6 +72,7 @@ NIX_DEPRECATED("use nix_value instead") typedef nix_value Value;
// Function prototypes
/**
* @brief Initialize the Nix language evaluator.
+ * @ingroup libexpr_init
*
* This function must be called at least once,
* at some point before constructing a EvalState for the first time.
@@ -77,6 +85,7 @@ nix_err nix_libexpr_init(nix_c_context * context);
/**
* @brief Parses and evaluates a Nix expression from a string.
+ * @ingroup value_create
*
* @param[out] context Optional, stores error information
* @param[in] state The state of the evaluation.
@@ -93,6 +102,7 @@ nix_err nix_expr_eval_from_string(
/**
* @brief Calls a Nix function with an argument.
+ * @ingroup value_create
*
* @param[out] context Optional, stores error information
* @param[in] state The state of the evaluation.
@@ -107,6 +117,7 @@ nix_err nix_value_call(nix_c_context * context, EvalState * state, nix_value * f
/**
* @brief Calls a Nix function with multiple arguments.
+ * @ingroup value_create
*
* Technically these are functions that return functions. It is common for Nix
* functions to be curried, so this function is useful for calling them.
@@ -126,10 +137,12 @@ nix_err nix_value_call_multi(
/**
* @brief Calls a Nix function with multiple arguments.
+ * @ingroup value_create
*
* Technically these are functions that return functions. It is common for Nix
* functions to be curried, so this function is useful for calling them.
*
+ * @def NIX_VALUE_CALL
* @param[out] context Optional, stores error information
* @param[in] state The state of the evaluation.
* @param[out] value The result of the function call.
@@ -147,6 +160,7 @@ nix_err nix_value_call_multi(
/**
* @brief Forces the evaluation of a Nix value.
+ * @ingroup value_create
*
* The Nix interpreter is lazy, and not-yet-evaluated values can be
* of type NIX_TYPE_THUNK instead of their actual value.
@@ -180,18 +194,20 @@ nix_err nix_value_force_deep(nix_c_context * context, EvalState * state, nix_val
/**
* @brief Create a new nix_eval_state_builder
+ * @ingroup libexpr_init
*
* The settings are initialized to their default value.
* Values can be sourced elsewhere with nix_eval_state_builder_load.
*
* @param[out] context Optional, stores error information
* @param[in] store The Nix store to use.
- * @return A new nix_eval_state_builder or NULL on failure.
+ * @return A new nix_eval_state_builder or NULL on failure. Call nix_eval_state_builder_free() when you're done.
*/
nix_eval_state_builder * nix_eval_state_builder_new(nix_c_context * context, Store * store);
/**
* @brief Read settings from the ambient environment
+ * @ingroup libexpr_init
*
* Settings are sourced from environment variables and configuration files,
* as documented in the Nix manual.
@@ -204,6 +220,7 @@ nix_err nix_eval_state_builder_load(nix_c_context * context, nix_eval_state_buil
/**
* @brief Set the lookup path for `<...>` expressions
+ * @ingroup libexpr_init
*
* @param[in] context Optional, stores error information
* @param[in] builder The builder to modify.
@@ -214,18 +231,21 @@ nix_err nix_eval_state_builder_set_lookup_path(
/**
* @brief Create a new Nix language evaluator state
+ * @ingroup libexpr_init
*
- * Remember to nix_eval_state_builder_free after building the state.
+ * The builder becomes unusable after this call. Remember to call nix_eval_state_builder_free()
+ * after building the state.
*
* @param[out] context Optional, stores error information
* @param[in] builder The builder to use and free
- * @return A new Nix state or NULL on failure.
+ * @return A new Nix state or NULL on failure. Call nix_state_free() when you're done.
* @see nix_eval_state_builder_new, nix_eval_state_builder_free
*/
EvalState * nix_eval_state_build(nix_c_context * context, nix_eval_state_builder * builder);
/**
* @brief Free a nix_eval_state_builder
+ * @ingroup libexpr_init
*
* Does not fail.
*
@@ -235,19 +255,21 @@ void nix_eval_state_builder_free(nix_eval_state_builder * builder);
/**
* @brief Create a new Nix language evaluator state
+ * @ingroup libexpr_init
*
* For more control, use nix_eval_state_builder
*
* @param[out] context Optional, stores error information
* @param[in] lookupPath Null-terminated array of strings corresponding to entries in NIX_PATH.
* @param[in] store The Nix store to use.
- * @return A new Nix state or NULL on failure.
+ * @return A new Nix state or NULL on failure. Call nix_state_free() when you're done.
* @see nix_state_builder_new
*/
EvalState * nix_state_create(nix_c_context * context, const char ** lookupPath, Store * store);
/**
* @brief Frees a Nix state.
+ * @ingroup libexpr_init
*
* Does not fail.
*
@@ -256,6 +278,7 @@ EvalState * nix_state_create(nix_c_context * context, const char ** lookupPath,
void nix_state_free(EvalState * state);
/** @addtogroup GC
+ * @ingroup libexpr
* @brief Reference counting and garbage collector operations
*
* The Nix language evaluator uses a garbage collector. To ease C interop, we implement
@@ -286,6 +309,9 @@ nix_err nix_gc_incref(nix_c_context * context, const void * object);
/**
* @brief Decrement the garbage collector reference counter for the given object
*
+ * @deprecated We are phasing out the general nix_gc_decref() in favor of type-specified free functions, such as
+ * nix_value_decref().
+ *
* We also provide typed `nix_*_decref` functions, which are
* - safer to use
* - easier to integrate when deriving bindings
@@ -314,12 +340,11 @@ void nix_gc_now();
*/
void nix_gc_register_finalizer(void * obj, void * cd, void (*finalizer)(void * obj, void * cd));
-/** @} */
+/** @} */ // doxygen group GC
+
// cffi end
#ifdef __cplusplus
}
#endif
-/** @} */
-
#endif // NIX_API_EXPR_H
diff --git a/src/libexpr-c/nix_api_external.h b/src/libexpr-c/nix_api_external.h
index f4a327281..96c479d57 100644
--- a/src/libexpr-c/nix_api_external.h
+++ b/src/libexpr-c/nix_api_external.h
@@ -2,11 +2,12 @@
#define NIX_API_EXTERNAL_H
/** @ingroup libexpr
* @addtogroup Externals
- * @brief Deal with external values
+ * @brief Externals let Nix expressions work with foreign values that aren't part of the normal Nix value data model
* @{
*/
/** @file
* @brief libexpr C bindings dealing with external values
+ * @see Externals
*/
#include "nix_api_expr.h"
@@ -115,7 +116,7 @@ typedef struct NixCExternalValueDesc
* @brief Try to compare two external values
*
* Optional, the default is always false.
- * If the other object was not a Nix C external value, this comparison will
+ * If the other object was not a Nix C API external value, this comparison will
* also return false
* @param[in] self the void* passed to nix_create_external_value
* @param[in] other the void* passed to the other object's
@@ -168,7 +169,7 @@ typedef struct NixCExternalValueDesc
/**
* @brief Create an external value, that can be given to nix_init_external
*
- * Owned by the GC. Use nix_gc_decref when you're done with the pointer.
+ * Call nix_gc_decref() when you're done with the pointer.
*
* @param[out] context Optional, stores error information
* @param[in] desc a NixCExternalValueDesc, you should keep this alive as long
@@ -180,10 +181,11 @@ typedef struct NixCExternalValueDesc
ExternalValue * nix_create_external_value(nix_c_context * context, NixCExternalValueDesc * desc, void * v);
/**
- * @brief Extract the pointer from a nix c external value.
+ * @brief Extract the pointer from a Nix C API external value.
* @param[out] context Optional, stores error information
* @param[in] b The external value
- * @returns The pointer, or null if the external value was not from nix c.
+ * @returns The pointer, valid while the external value is valid, or null if the external value was not from the Nix C
+ * API.
* @see nix_get_external
*/
void * nix_get_external_value_content(nix_c_context * context, ExternalValue * b);
diff --git a/src/libexpr-c/nix_api_value.cc b/src/libexpr-c/nix_api_value.cc
index 3b8c7dd04..e231c36f4 100644
--- a/src/libexpr-c/nix_api_value.cc
+++ b/src/libexpr-c/nix_api_value.cc
@@ -235,7 +235,7 @@ nix_get_string(nix_c_context * context, const nix_value * value, nix_get_string_
try {
auto & v = check_value_in(value);
assert(v.type() == nix::nString);
- call_nix_get_string_callback(v.c_str(), callback, user_data);
+ call_nix_get_string_callback(v.string_view(), callback, user_data);
}
NIXC_CATCH_ERRS
}
diff --git a/src/libexpr-c/nix_api_value.h b/src/libexpr-c/nix_api_value.h
index 835eaec6e..5bd45da90 100644
--- a/src/libexpr-c/nix_api_value.h
+++ b/src/libexpr-c/nix_api_value.h
@@ -1,9 +1,6 @@
#ifndef NIX_API_VALUE_H
#define NIX_API_VALUE_H
-/** @addtogroup libexpr
- * @{
- */
/** @file
* @brief libexpr C bindings dealing with values
*/
@@ -20,18 +17,89 @@ extern "C" {
#endif
// cffi start
+/** @defgroup value Value
+ * @ingroup libexpr
+ * @brief nix_value type and core operations for working with Nix values
+ * @see value_create
+ * @see value_extract
+ */
+
+/** @defgroup value_create Value Creation
+ * @ingroup libexpr
+ * @brief Functions for allocating and initializing Nix values
+ *
+ * Values are usually created with `nix_alloc_value` followed by `nix_init_*` functions.
+ * In primop callbacks, allocation is already done and only initialization is needed.
+ */
+
+/** @defgroup value_extract Value Extraction
+ * @ingroup libexpr
+ * @brief Functions for extracting data from Nix values
+ */
+
+/** @defgroup primops PrimOps and Builtins
+ * @ingroup libexpr
+ */
+
// Type definitions
+/** @brief Represents the state of a Nix value
+ *
+ * Thunk values (NIX_TYPE_THUNK) change to their final, unchanging type when forced.
+ *
+ * @see https://nix.dev/manual/nix/latest/language/evaluation.html
+ * @enum ValueType
+ * @ingroup value
+ */
typedef enum {
+ /** Unevaluated expression
+ *
+ * Thunks often contain an expression and closure, but may contain other
+ * representations too.
+ *
+ * Their state is mutable, unlike that of the other types.
+ */
NIX_TYPE_THUNK,
+ /**
+ * A 64 bit signed integer.
+ */
NIX_TYPE_INT,
+ /** @brief IEEE 754 double precision floating point number
+ * @see https://nix.dev/manual/nix/latest/language/types.html#type-float
+ */
NIX_TYPE_FLOAT,
+ /** @brief Boolean true or false value
+ * @see https://nix.dev/manual/nix/latest/language/types.html#type-bool
+ */
NIX_TYPE_BOOL,
+ /** @brief String value with context
+ *
+ * String content may contain arbitrary bytes, not necessarily UTF-8.
+ * @see https://nix.dev/manual/nix/latest/language/types.html#type-string
+ */
NIX_TYPE_STRING,
+ /** @brief Filesystem path
+ * @see https://nix.dev/manual/nix/latest/language/types.html#type-path
+ */
NIX_TYPE_PATH,
+ /** @brief Null value
+ * @see https://nix.dev/manual/nix/latest/language/types.html#type-null
+ */
NIX_TYPE_NULL,
+ /** @brief Attribute set (key-value mapping)
+ * @see https://nix.dev/manual/nix/latest/language/types.html#type-attrs
+ */
NIX_TYPE_ATTRS,
+ /** @brief Ordered list of values
+ * @see https://nix.dev/manual/nix/latest/language/types.html#type-list
+ */
NIX_TYPE_LIST,
+ /** @brief Function (lambda or builtin)
+ * @see https://nix.dev/manual/nix/latest/language/types.html#type-function
+ */
NIX_TYPE_FUNCTION,
+ /** @brief External value from C++ plugins or C API
+ * @see Externals
+ */
NIX_TYPE_EXTERNAL
} ValueType;
@@ -39,22 +107,41 @@ typedef enum {
typedef struct nix_value nix_value;
typedef struct EvalState EvalState;
+/** @deprecated Use nix_value instead */
[[deprecated("use nix_value instead")]] typedef nix_value Value;
// type defs
/** @brief Stores an under-construction set of bindings
- * @ingroup value_manip
+ * @ingroup value_create
*
- * Do not reuse.
+ * Each builder can only be used once. After calling nix_make_attrs(), the builder
+ * becomes invalid and must not be used again. Call nix_bindings_builder_free() to release it.
+ *
+ * Typical usage pattern:
+ * 1. Create with nix_make_bindings_builder()
+ * 2. Insert attributes with nix_bindings_builder_insert()
+ * 3. Create final attribute set with nix_make_attrs()
+ * 4. Free builder with nix_bindings_builder_free()
+ *
+ * @struct BindingsBuilder
* @see nix_make_bindings_builder, nix_bindings_builder_free, nix_make_attrs
* @see nix_bindings_builder_insert
*/
typedef struct BindingsBuilder BindingsBuilder;
/** @brief Stores an under-construction list
- * @ingroup value_manip
+ * @ingroup value_create
*
- * Do not reuse.
+ * Each builder can only be used once. After calling nix_make_list(), the builder
+ * becomes invalid and must not be used again. Call nix_list_builder_free() to release it.
+ *
+ * Typical usage pattern:
+ * 1. Create with nix_make_list_builder()
+ * 2. Insert elements with nix_list_builder_insert()
+ * 3. Create final list with nix_make_list()
+ * 4. Free builder with nix_list_builder_free()
+ *
+ * @struct ListBuilder
* @see nix_make_list_builder, nix_list_builder_free, nix_make_list
* @see nix_list_builder_insert
*/
@@ -63,25 +150,28 @@ typedef struct ListBuilder ListBuilder;
/** @brief PrimOp function
* @ingroup primops
*
- * Owned by the GC
- * @see nix_alloc_primop, nix_init_primop
+ * Can be released with nix_gc_decref() when necessary.
+ * @struct PrimOp
+ * @see nix_alloc_primop, nix_init_primop, nix_register_primop
*/
typedef struct PrimOp PrimOp;
/** @brief External Value
* @ingroup Externals
*
- * Owned by the GC
+ * Can be released with nix_gc_decref() when necessary.
+ * @struct ExternalValue
+ * @see nix_create_external_value, nix_init_external, nix_get_external
*/
typedef struct ExternalValue ExternalValue;
/** @brief String without placeholders, and realised store paths
+ * @struct nix_realised_string
+ * @see nix_string_realise, nix_realised_string_free
*/
typedef struct nix_realised_string nix_realised_string;
-/** @defgroup primops Adding primops
- * @{
- */
/** @brief Function pointer for primops
+ * @ingroup primops
*
* When you want to return an error, call nix_set_err_msg(context, NIX_ERR_UNKNOWN, "your error message here").
*
@@ -97,9 +187,9 @@ typedef void (*PrimOpFun)(
void * user_data, nix_c_context * context, EvalState * state, nix_value ** args, nix_value * ret);
/** @brief Allocate a PrimOp
+ * @ingroup primops
*
- * Owned by the garbage collector.
- * Use nix_gc_decref() when you're done with the returned PrimOp.
+ * Call nix_gc_decref() when you're done with the returned PrimOp.
*
* @param[out] context Optional, stores error information
* @param[in] fun callback
@@ -121,35 +211,38 @@ PrimOp * nix_alloc_primop(
void * user_data);
/** @brief add a primop to the `builtins` attribute set
+ * @ingroup primops
*
* Only applies to States created after this call.
*
- * Moves your PrimOp content into the global evaluator
- * registry, meaning your input PrimOp pointer is no longer usable.
- * You are free to remove your references to it,
- * after which it will be garbage collected.
+ * Moves your PrimOp content into the global evaluator registry, meaning
+ * your input PrimOp pointer becomes invalid. The PrimOp must not be used
+ * with nix_init_primop() before or after this call, as this would cause
+ * undefined behavior.
+ * You must call nix_gc_decref() on the original PrimOp pointer
+ * after this call to release your reference.
*
* @param[out] context Optional, stores error information
- * @return primop, or null in case of errors
- *
+ * @param[in] primOp PrimOp to register
+ * @return error code, NIX_OK on success
*/
nix_err nix_register_primop(nix_c_context * context, PrimOp * primOp);
-/** @} */
// Function prototypes
/** @brief Allocate a Nix value
+ * @ingroup value_create
*
- * Owned by the GC. Use nix_gc_decref() when you're done with the pointer
+ * Call nix_value_decref() when you're done with the pointer
* @param[out] context Optional, stores error information
* @param[in] state nix evaluator state
* @return value, or null in case of errors
- *
*/
nix_value * nix_alloc_value(nix_c_context * context, EvalState * state);
/**
* @brief Increment the garbage collector reference counter for the given `nix_value`.
+ * @ingroup value
*
* The Nix language evaluator C API keeps track of alive objects by reference counting.
* When you're done with a refcounted pointer, call nix_value_decref().
@@ -161,21 +254,19 @@ nix_err nix_value_incref(nix_c_context * context, nix_value * value);
/**
* @brief Decrement the garbage collector reference counter for the given object
+ * @ingroup value
+ *
+ * When the counter reaches zero, the `nix_value` object becomes invalid.
+ * The data referenced by `nix_value` may not be deallocated until the memory
+ * garbage collector has run, but deallocation is not guaranteed.
*
* @param[out] context Optional, stores error information
* @param[in] value The object to stop referencing
*/
nix_err nix_value_decref(nix_c_context * context, nix_value * value);
-/** @addtogroup value_manip Manipulating values
- * @brief Functions to inspect and change Nix language values, represented by nix_value.
- * @{
- */
-/** @anchor getters
- * @name Getters
- */
-/**@{*/
/** @brief Get value type
+ * @ingroup value_extract
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect
* @return type of nix value
@@ -183,14 +274,15 @@ nix_err nix_value_decref(nix_c_context * context, nix_value * value);
ValueType nix_get_type(nix_c_context * context, const nix_value * value);
/** @brief Get type name of value as defined in the evaluator
+ * @ingroup value_extract
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect
- * @return type name, owned string
- * @todo way to free the result
+ * @return type name string, free with free()
*/
const char * nix_get_typename(nix_c_context * context, const nix_value * value);
/** @brief Get boolean value
+ * @ingroup value_extract
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect
* @return true or false, error info via context
@@ -198,6 +290,7 @@ const char * nix_get_typename(nix_c_context * context, const nix_value * value);
bool nix_get_bool(nix_c_context * context, const nix_value * value);
/** @brief Get the raw string
+ * @ingroup value_extract
*
* This may contain placeholders.
*
@@ -205,21 +298,21 @@ bool nix_get_bool(nix_c_context * context, const nix_value * value);
* @param[in] value Nix value to inspect
* @param[in] callback Called with the string value.
* @param[in] user_data optional, arbitrary data, passed to the callback when it's called.
- * @return string
* @return error code, NIX_OK on success.
*/
nix_err
nix_get_string(nix_c_context * context, const nix_value * value, nix_get_string_callback callback, void * user_data);
/** @brief Get path as string
+ * @ingroup value_extract
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect
- * @return string, if the type is NIX_TYPE_PATH
- * @return NULL in case of error.
+ * @return string valid while value is valid, NULL in case of error
*/
const char * nix_get_path_string(nix_c_context * context, const nix_value * value);
/** @brief Get the length of a list
+ * @ingroup value_extract
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect
* @return length of list, error info via context
@@ -227,6 +320,7 @@ const char * nix_get_path_string(nix_c_context * context, const nix_value * valu
unsigned int nix_get_list_size(nix_c_context * context, const nix_value * value);
/** @brief Get the element count of an attrset
+ * @ingroup value_extract
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect
* @return attrset element count, error info via context
@@ -234,6 +328,7 @@ unsigned int nix_get_list_size(nix_c_context * context, const nix_value * value)
unsigned int nix_get_attrs_size(nix_c_context * context, const nix_value * value);
/** @brief Get float value in 64 bits
+ * @ingroup value_extract
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect
* @return float contents, error info via context
@@ -241,6 +336,7 @@ unsigned int nix_get_attrs_size(nix_c_context * context, const nix_value * value
double nix_get_float(nix_c_context * context, const nix_value * value);
/** @brief Get int value
+ * @ingroup value_extract
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect
* @return int contents, error info via context
@@ -248,15 +344,18 @@ double nix_get_float(nix_c_context * context, const nix_value * value);
int64_t nix_get_int(nix_c_context * context, const nix_value * value);
/** @brief Get external reference
+ * @ingroup value_extract
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect
- * @return reference to external, NULL in case of error
+ * @return reference valid while value is valid. Call nix_gc_incref() if you need it to live longer, then only in that
+ * case call nix_gc_decref() when done. NULL in case of error
*/
ExternalValue * nix_get_external(nix_c_context * context, nix_value * value);
/** @brief Get the ix'th element of a list
+ * @ingroup value_extract
*
- * Owned by the GC. Use nix_gc_decref when you're done with the pointer
+ * Call nix_value_decref() when you're done with the pointer
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect
* @param[in] state nix evaluator state
@@ -266,11 +365,12 @@ ExternalValue * nix_get_external(nix_c_context * context, nix_value * value);
nix_value * nix_get_list_byidx(nix_c_context * context, const nix_value * value, EvalState * state, unsigned int ix);
/** @brief Get the ix'th element of a list without forcing evaluation of the element
+ * @ingroup value_extract
*
* Returns the list element without forcing its evaluation, allowing access to lazy values.
* The list value itself must already be evaluated.
*
- * Owned by the GC. Use nix_gc_decref when you're done with the pointer
+ * Call nix_value_decref() when you're done with the pointer
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect (must be an evaluated list)
* @param[in] state nix evaluator state
@@ -281,8 +381,9 @@ nix_value *
nix_get_list_byidx_lazy(nix_c_context * context, const nix_value * value, EvalState * state, unsigned int ix);
/** @brief Get an attr by name
+ * @ingroup value_extract
*
- * Use nix_gc_decref when you're done with the pointer
+ * Call nix_value_decref() when you're done with the pointer
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect
* @param[in] state nix evaluator state
@@ -292,11 +393,12 @@ nix_get_list_byidx_lazy(nix_c_context * context, const nix_value * value, EvalSt
nix_value * nix_get_attr_byname(nix_c_context * context, const nix_value * value, EvalState * state, const char * name);
/** @brief Get an attribute value by attribute name, without forcing evaluation of the attribute's value
+ * @ingroup value_extract
*
* Returns the attribute value without forcing its evaluation, allowing access to lazy values.
* The attribute set value itself must already be evaluated.
*
- * Use nix_gc_decref when you're done with the pointer
+ * Call nix_value_decref() when you're done with the pointer
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect (must be an evaluated attribute set)
* @param[in] state nix evaluator state
@@ -307,6 +409,7 @@ nix_value *
nix_get_attr_byname_lazy(nix_c_context * context, const nix_value * value, EvalState * state, const char * name);
/** @brief Check if an attribute name exists on a value
+ * @ingroup value_extract
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect
* @param[in] state nix evaluator state
@@ -316,6 +419,7 @@ nix_get_attr_byname_lazy(nix_c_context * context, const nix_value * value, EvalS
bool nix_has_attr_byname(nix_c_context * context, const nix_value * value, EvalState * state, const char * name);
/** @brief Get an attribute by index
+ * @ingroup value_extract
*
* Also gives you the name.
*
@@ -329,18 +433,19 @@ bool nix_has_attr_byname(nix_c_context * context, const nix_value * value, EvalS
* lexicographic order by Unicode scalar value for valid UTF-8). We recommend
* applying this same ordering for consistency.
*
- * Use nix_gc_decref when you're done with the pointer
+ * Call nix_value_decref() when you're done with the pointer
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect
* @param[in] state nix evaluator state
* @param[in] i attribute index
- * @param[out] name will store a pointer to the attribute name
+ * @param[out] name will store a pointer to the attribute name, valid until state is freed
* @return value, NULL in case of errors
*/
nix_value *
nix_get_attr_byidx(nix_c_context * context, nix_value * value, EvalState * state, unsigned int i, const char ** name);
/** @brief Get an attribute by index, without forcing evaluation of the attribute's value
+ * @ingroup value_extract
*
* Also gives you the name.
*
@@ -357,18 +462,19 @@ nix_get_attr_byidx(nix_c_context * context, nix_value * value, EvalState * state
* lexicographic order by Unicode scalar value for valid UTF-8). We recommend
* applying this same ordering for consistency.
*
- * Use nix_gc_decref when you're done with the pointer
+ * Call nix_value_decref() when you're done with the pointer
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect (must be an evaluated attribute set)
* @param[in] state nix evaluator state
* @param[in] i attribute index
- * @param[out] name will store a pointer to the attribute name
+ * @param[out] name will store a pointer to the attribute name, valid until state is freed
* @return value, NULL in case of errors
*/
nix_value * nix_get_attr_byidx_lazy(
nix_c_context * context, nix_value * value, EvalState * state, unsigned int i, const char ** name);
/** @brief Get an attribute name by index
+ * @ingroup value_extract
*
* Returns the attribute name without forcing evaluation of the attribute's value.
*
@@ -382,16 +488,14 @@ nix_value * nix_get_attr_byidx_lazy(
* lexicographic order by Unicode scalar value for valid UTF-8). We recommend
* applying this same ordering for consistency.
*
- * Owned by the nix EvalState
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect
* @param[in] state nix evaluator state
* @param[in] i attribute index
- * @return name, NULL in case of errors
+ * @return name string valid until state is freed, NULL in case of errors
*/
const char * nix_get_attr_name_byidx(nix_c_context * context, nix_value * value, EvalState * state, unsigned int i);
-/**@}*/
/** @name Initializers
*
* Values are typically "returned" by initializing already allocated memory that serves as the return value.
@@ -401,6 +505,7 @@ const char * nix_get_attr_name_byidx(nix_c_context * context, nix_value * value,
*/
/**@{*/
/** @brief Set boolean value
+ * @ingroup value_create
* @param[out] context Optional, stores error information
* @param[out] value Nix value to modify
* @param[in] b the boolean value
@@ -409,6 +514,7 @@ const char * nix_get_attr_name_byidx(nix_c_context * context, nix_value * value,
nix_err nix_init_bool(nix_c_context * context, nix_value * value, bool b);
/** @brief Set a string
+ * @ingroup value_create
* @param[out] context Optional, stores error information
* @param[out] value Nix value to modify
* @param[in] str the string, copied
@@ -417,6 +523,7 @@ nix_err nix_init_bool(nix_c_context * context, nix_value * value, bool b);
nix_err nix_init_string(nix_c_context * context, nix_value * value, const char * str);
/** @brief Set a path
+ * @ingroup value_create
* @param[out] context Optional, stores error information
* @param[out] value Nix value to modify
* @param[in] str the path string, copied
@@ -425,6 +532,7 @@ nix_err nix_init_string(nix_c_context * context, nix_value * value, const char *
nix_err nix_init_path_string(nix_c_context * context, EvalState * s, nix_value * value, const char * str);
/** @brief Set a float
+ * @ingroup value_create
* @param[out] context Optional, stores error information
* @param[out] value Nix value to modify
* @param[in] d the float, 64-bits
@@ -433,6 +541,7 @@ nix_err nix_init_path_string(nix_c_context * context, EvalState * s, nix_value *
nix_err nix_init_float(nix_c_context * context, nix_value * value, double d);
/** @brief Set an int
+ * @ingroup value_create
* @param[out] context Optional, stores error information
* @param[out] value Nix value to modify
* @param[in] i the int
@@ -441,6 +550,7 @@ nix_err nix_init_float(nix_c_context * context, nix_value * value, double d);
nix_err nix_init_int(nix_c_context * context, nix_value * value, int64_t i);
/** @brief Set null
+ * @ingroup value_create
* @param[out] context Optional, stores error information
* @param[out] value Nix value to modify
* @return error code, NIX_OK on success.
@@ -448,6 +558,7 @@ nix_err nix_init_int(nix_c_context * context, nix_value * value, int64_t i);
nix_err nix_init_null(nix_c_context * context, nix_value * value);
/** @brief Set the value to a thunk that will perform a function application when needed.
+ * @ingroup value_create
*
* Thunks may be put into attribute sets and lists to perform some computation lazily; on demand.
* However, note that in some places, a thunk must not be returned, such as in the return value of a PrimOp.
@@ -464,6 +575,7 @@ nix_err nix_init_null(nix_c_context * context, nix_value * value);
nix_err nix_init_apply(nix_c_context * context, nix_value * value, nix_value * fn, nix_value * arg);
/** @brief Set an external value
+ * @ingroup value_create
* @param[out] context Optional, stores error information
* @param[out] value Nix value to modify
* @param[in] val the external value to set. Will be GC-referenced by the value.
@@ -472,18 +584,25 @@ nix_err nix_init_apply(nix_c_context * context, nix_value * value, nix_value * f
nix_err nix_init_external(nix_c_context * context, nix_value * value, ExternalValue * val);
/** @brief Create a list from a list builder
+ * @ingroup value_create
+ *
+ * After this call, the list builder becomes invalid and cannot be used again.
+ * The only necessary next step is to free it with nix_list_builder_free().
+ *
* @param[out] context Optional, stores error information
- * @param[in] list_builder list builder to use. Make sure to unref this afterwards.
+ * @param[in] list_builder list builder to use
* @param[out] value Nix value to modify
* @return error code, NIX_OK on success.
+ * @see nix_list_builder_free
*/
nix_err nix_make_list(nix_c_context * context, ListBuilder * list_builder, nix_value * value);
/** @brief Create a list builder
+ * @ingroup value_create
* @param[out] context Optional, stores error information
* @param[in] state nix evaluator state
* @param[in] capacity how many bindings you'll add. Don't exceed.
- * @return owned reference to a list builder. Make sure to unref when you're done.
+ * @return list builder. Call nix_list_builder_free() when you're done.
*/
ListBuilder * nix_make_list_builder(nix_c_context * context, EvalState * state, size_t capacity);
@@ -505,14 +624,21 @@ nix_list_builder_insert(nix_c_context * context, ListBuilder * list_builder, uns
void nix_list_builder_free(ListBuilder * list_builder);
/** @brief Create an attribute set from a bindings builder
+ * @ingroup value_create
+ *
+ * After this call, the bindings builder becomes invalid and cannot be used again.
+ * The only necessary next step is to free it with nix_bindings_builder_free().
+ *
* @param[out] context Optional, stores error information
* @param[out] value Nix value to modify
- * @param[in] b bindings builder to use. Make sure to unref this afterwards.
+ * @param[in] b bindings builder to use
* @return error code, NIX_OK on success.
+ * @see nix_bindings_builder_free
*/
nix_err nix_make_attrs(nix_c_context * context, nix_value * value, BindingsBuilder * b);
/** @brief Set primop
+ * @ingroup value_create
* @param[out] context Optional, stores error information
* @param[out] value Nix value to modify
* @param[in] op primop, will be gc-referenced by the value
@@ -521,6 +647,7 @@ nix_err nix_make_attrs(nix_c_context * context, nix_value * value, BindingsBuild
*/
nix_err nix_init_primop(nix_c_context * context, nix_value * value, PrimOp * op);
/** @brief Copy from another value
+ * @ingroup value_create
* @param[out] context Optional, stores error information
* @param[out] value Nix value to modify
* @param[in] source value to copy from
@@ -530,12 +657,11 @@ nix_err nix_copy_value(nix_c_context * context, nix_value * value, const nix_val
/**@}*/
/** @brief Create a bindings builder
-* @param[out] context Optional, stores error information
-* @param[in] state nix evaluator state
-* @param[in] capacity how many bindings you'll add. Don't exceed.
-* @return owned reference to a bindings builder. Make sure to unref when you're
-done.
-*/
+ * @param[out] context Optional, stores error information
+ * @param[in] state nix evaluator state
+ * @param[in] capacity how many bindings you'll add. Don't exceed.
+ * @return bindings builder. Call nix_bindings_builder_free() when you're done.
+ */
BindingsBuilder * nix_make_bindings_builder(nix_c_context * context, EvalState * state, size_t capacity);
/** @brief Insert bindings into a builder
@@ -554,7 +680,6 @@ nix_bindings_builder_insert(nix_c_context * context, BindingsBuilder * builder,
* @param[in] builder the builder to free
*/
void nix_bindings_builder_free(BindingsBuilder * builder);
-/**@}*/
/** @brief Realise a string context.
*
@@ -571,13 +696,13 @@ void nix_bindings_builder_free(BindingsBuilder * builder);
* @param[in] isIFD If true, disallow derivation outputs if setting `allow-import-from-derivation` is false.
You should set this to true when this call is part of a primop.
You should set this to false when building for your application's purpose.
- * @return NULL if failed, are a new nix_realised_string, which must be freed with nix_realised_string_free
+ * @return NULL if failed, or a new nix_realised_string, which must be freed with nix_realised_string_free
*/
nix_realised_string * nix_string_realise(nix_c_context * context, EvalState * state, nix_value * value, bool isIFD);
/** @brief Start of the string
* @param[in] realised_string
- * @return pointer to the start of the string. It may not be null-terminated.
+ * @return pointer to the start of the string, valid until realised_string is freed. It may not be null-terminated.
*/
const char * nix_realised_string_get_buffer_start(nix_realised_string * realised_string);
@@ -596,7 +721,7 @@ size_t nix_realised_string_get_store_path_count(nix_realised_string * realised_s
/** @brief Get a store path. The store paths are stored in an arbitrary order.
* @param[in] realised_string
* @param[in] index index of the store path, must be less than the count
- * @return store path
+ * @return store path valid until realised_string is freed
*/
const StorePath * nix_realised_string_get_store_path(nix_realised_string * realised_string, size_t index);
@@ -610,5 +735,4 @@ void nix_realised_string_free(nix_realised_string * realised_string);
}
#endif
-/** @} */
#endif // NIX_API_VALUE_H
diff --git a/src/libexpr-test-support/include/nix/expr/tests/libexpr.hh b/src/libexpr-test-support/include/nix/expr/tests/libexpr.hh
index a1320e14a..daae00802 100644
--- a/src/libexpr-test-support/include/nix/expr/tests/libexpr.hh
+++ b/src/libexpr-test-support/include/nix/expr/tests/libexpr.hh
@@ -106,7 +106,7 @@ MATCHER_P(IsStringEq, s, fmt("The string is equal to \"%1%\"", s))
if (arg.type() != nString) {
return false;
}
- return std::string_view(arg.c_str()) == s;
+ return arg.string_view() == s;
}
MATCHER_P(IsIntEq, v, fmt("The string is equal to \"%1%\"", v))
diff --git a/src/libexpr-tests/primops.cc b/src/libexpr-tests/primops.cc
index 74d676844..f00b4f475 100644
--- a/src/libexpr-tests/primops.cc
+++ b/src/libexpr-tests/primops.cc
@@ -771,7 +771,7 @@ TEST_F(PrimOpTest, derivation)
ASSERT_EQ(v.type(), nFunction);
ASSERT_TRUE(v.isLambda());
ASSERT_NE(v.lambda().fun, nullptr);
- ASSERT_TRUE(v.lambda().fun->hasFormals());
+ ASSERT_TRUE(v.lambda().fun->getFormals());
}
TEST_F(PrimOpTest, currentTime)
diff --git a/src/libexpr-tests/trivial.cc b/src/libexpr-tests/trivial.cc
index a287ce4d1..d112c269a 100644
--- a/src/libexpr-tests/trivial.cc
+++ b/src/libexpr-tests/trivial.cc
@@ -1,4 +1,5 @@
#include "nix/expr/tests/libexpr.hh"
+#include "nix/util/tests/gmock-matchers.hh"
namespace nix {
// Testing of trivial expressions
@@ -160,7 +161,8 @@ TEST_F(TrivialExpressionTest, assertPassed)
ASSERT_THAT(v, IsIntEq(123));
}
-class AttrSetMergeTrvialExpressionTest : public TrivialExpressionTest, public testing::WithParamInterface
+class AttrSetMergeTrvialExpressionTest : public TrivialExpressionTest,
+ public ::testing::WithParamInterface
{};
TEST_P(AttrSetMergeTrvialExpressionTest, attrsetMergeLazy)
@@ -196,7 +198,7 @@ TEST_P(AttrSetMergeTrvialExpressionTest, attrsetMergeLazy)
INSTANTIATE_TEST_SUITE_P(
attrsetMergeLazy,
AttrSetMergeTrvialExpressionTest,
- testing::Values("{ a.b = 1; a.c = 2; }", "{ a = { b = 1; }; a = { c = 2; }; }"));
+ ::testing::Values("{ a.b = 1; a.c = 2; }", "{ a = { b = 1; }; a = { c = 2; }; }"));
// The following macros ultimately define 48 tests (16 variations on three
// templates). Each template tests an expression that can be written in 2^4
@@ -339,4 +341,18 @@ TEST_F(TrivialExpressionTest, orCantBeUsed)
{
ASSERT_THROW(eval("let or = 1; in or"), Error);
}
+
+TEST_F(TrivialExpressionTest, tooManyFormals)
+{
+ std::string expr = "let f = { ";
+ for (uint32_t i = 0; i <= std::numeric_limits::max(); ++i) {
+ expr += fmt("arg%d, ", i);
+ }
+ expr += " }: 0 in; f {}";
+ ASSERT_THAT(
+ [&]() { eval(expr); },
+ ::testing::ThrowsMessage(::nix::testing::HasSubstrIgnoreANSIMatcher(
+ "too many formal arguments, implementation supports at most 65535")));
+}
+
} /* namespace nix */
diff --git a/src/libexpr-tests/value/print.cc b/src/libexpr-tests/value/print.cc
index 1959fddf2..0006da2ff 100644
--- a/src/libexpr-tests/value/print.cc
+++ b/src/libexpr-tests/value/print.cc
@@ -10,7 +10,7 @@ using namespace testing;
struct ValuePrintingTests : LibExprTest
{
template
- void test(Value v, std::string_view expected, A... args)
+ void test(Value & v, std::string_view expected, A... args)
{
std::stringstream out;
v.print(state, out, args...);
@@ -110,9 +110,8 @@ TEST_F(ValuePrintingTests, vLambda)
PosTable::Origin origin = state.positions.addOrigin(std::monostate(), 1);
auto posIdx = state.positions.add(origin, 0);
auto body = ExprInt(0);
- auto formals = Formals{};
- ExprLambda eLambda(posIdx, createSymbol("a"), &formals, &body);
+ ExprLambda eLambda(posIdx, createSymbol("a"), &body);
Value vLambda;
vLambda.mkLambda(&env, &eLambda);
@@ -500,9 +499,8 @@ TEST_F(ValuePrintingTests, ansiColorsLambda)
PosTable::Origin origin = state.positions.addOrigin(std::monostate(), 1);
auto posIdx = state.positions.add(origin, 0);
auto body = ExprInt(0);
- auto formals = Formals{};
- ExprLambda eLambda(posIdx, createSymbol("a"), &formals, &body);
+ ExprLambda eLambda(posIdx, createSymbol("a"), &body);
Value vLambda;
vLambda.mkLambda(&env, &eLambda);
@@ -625,10 +623,11 @@ TEST_F(ValuePrintingTests, ansiColorsAttrsElided)
vThree.mkInt(3);
builder.insert(state.symbols.create("three"), &vThree);
- vAttrs.mkAttrs(builder.finish());
+ Value vAttrs2;
+ vAttrs2.mkAttrs(builder.finish());
test(
- vAttrs,
+ vAttrs2,
"{ one = " ANSI_CYAN "1" ANSI_NORMAL "; " ANSI_FAINT "«2 attributes elided»" ANSI_NORMAL " }",
PrintOptions{.ansiColors = true, .maxAttrs = 1});
}
diff --git a/src/libexpr-tests/value/value.cc b/src/libexpr-tests/value/value.cc
index 63501dd49..229e449db 100644
--- a/src/libexpr-tests/value/value.cc
+++ b/src/libexpr-tests/value/value.cc
@@ -1,6 +1,7 @@
#include "nix/expr/value.hh"
#include "nix/store/tests/libstore.hh"
+#include
namespace nix {
@@ -22,4 +23,21 @@ TEST_F(ValueTest, vInt)
ASSERT_EQ(true, vInt.isValid());
}
+TEST_F(ValueTest, staticString)
+{
+ Value vStr1;
+ Value vStr2;
+ vStr1.mkStringNoCopy("foo");
+ vStr2.mkStringNoCopy("foo");
+
+ auto sd1 = vStr1.string_view();
+ auto sd2 = vStr2.string_view();
+
+ // The strings should be the same
+ ASSERT_EQ(sd1, sd2);
+
+ // The strings should also be backed by the same (static) allocation
+ ASSERT_EQ(sd1.data(), sd2.data());
+}
+
} // namespace nix
diff --git a/src/libexpr/eval-cache.cc b/src/libexpr/eval-cache.cc
index 480ca72c7..de74d2143 100644
--- a/src/libexpr/eval-cache.cc
+++ b/src/libexpr/eval-cache.cc
@@ -406,7 +406,7 @@ Value & AttrCursor::forceValue()
if (root->db && (!cachedValue || std::get_if(&cachedValue->second))) {
if (v.type() == nString)
- cachedValue = {root->db->setString(getKey(), v.c_str(), v.context()), string_t{v.c_str(), {}}};
+ cachedValue = {root->db->setString(getKey(), v.string_view(), v.context()), string_t{v.string_view(), {}}};
else if (v.type() == nPath) {
auto path = v.path().path;
cachedValue = {root->db->setString(getKey(), path.abs()), string_t{path.abs(), {}}};
@@ -541,7 +541,7 @@ std::string AttrCursor::getString()
if (v.type() != nString && v.type() != nPath)
root->state.error("'%s' is not a string but %s", getAttrPathStr(), showType(v)).debugThrow();
- return v.type() == nString ? v.c_str() : v.path().to_string();
+ return v.type() == nString ? std::string(v.string_view()) : v.path().to_string();
}
string_t AttrCursor::getStringWithContext()
@@ -580,7 +580,7 @@ string_t AttrCursor::getStringWithContext()
if (v.type() == nString) {
NixStringContext context;
copyContext(v, context);
- return {v.c_str(), std::move(context)};
+ return {std::string{v.string_view()}, std::move(context)};
} else if (v.type() == nPath)
return {v.path().to_string(), {}};
else
diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc
index 800c29fad..7036df957 100644
--- a/src/libexpr/eval.cc
+++ b/src/libexpr/eval.cc
@@ -1496,15 +1496,13 @@ void EvalState::callFunction(Value & fun, std::span args, Value & vRes,
ExprLambda & lambda(*vCur.lambda().fun);
- auto size = (!lambda.arg ? 0 : 1) + (lambda.hasFormals() ? lambda.formals->formals.size() : 0);
+ auto size = (!lambda.arg ? 0 : 1) + (lambda.getFormals() ? lambda.getFormals()->formals.size() : 0);
Env & env2(mem.allocEnv(size));
env2.up = vCur.lambda().env;
Displacement displ = 0;
- if (!lambda.hasFormals())
- env2.values[displ++] = args[0];
- else {
+ if (auto formals = lambda.getFormals()) {
try {
forceAttrs(*args[0], lambda.pos, "while evaluating the value passed for the lambda argument");
} catch (Error & e) {
@@ -1520,7 +1518,7 @@ void EvalState::callFunction(Value & fun, std::span args, Value & vRes,
there is no matching actual argument but the formal
argument has a default, use the default. */
size_t attrsUsed = 0;
- for (auto & i : lambda.formals->formals) {
+ for (auto & i : formals->formals) {
auto j = args[0]->attrs()->get(i.name);
if (!j) {
if (!i.def) {
@@ -1542,13 +1540,13 @@ void EvalState::callFunction(Value & fun, std::span args, Value & vRes,
/* Check that each actual argument is listed as a formal
argument (unless the attribute match specifies a `...'). */
- if (!lambda.formals->ellipsis && attrsUsed != args[0]->attrs()->size()) {
+ if (!formals->ellipsis && attrsUsed != args[0]->attrs()->size()) {
/* Nope, so show the first unexpected argument to the
user. */
for (auto & i : *args[0]->attrs())
- if (!lambda.formals->has(i.name)) {
+ if (!formals->has(i.name)) {
StringSet formalNames;
- for (auto & formal : lambda.formals->formals)
+ for (auto & formal : formals->formals)
formalNames.insert(std::string(symbols[formal.name]));
auto suggestions = Suggestions::bestMatches(formalNames, symbols[i.name]);
error(
@@ -1563,6 +1561,8 @@ void EvalState::callFunction(Value & fun, std::span args, Value & vRes,
}
unreachable();
}
+ } else {
+ env2.values[displ++] = args[0];
}
nrFunctionCalls++;
@@ -1747,14 +1747,15 @@ void EvalState::autoCallFunction(const Bindings & args, Value & fun, Value & res
}
}
- if (!fun.isLambda() || !fun.lambda().fun->hasFormals()) {
+ if (!fun.isLambda() || !fun.lambda().fun->getFormals()) {
res = fun;
return;
}
+ auto formals = fun.lambda().fun->getFormals();
- auto attrs = buildBindings(std::max(static_cast(fun.lambda().fun->formals->formals.size()), args.size()));
+ auto attrs = buildBindings(std::max(static_cast(formals->formals.size()), args.size()));
- if (fun.lambda().fun->formals->ellipsis) {
+ if (formals->ellipsis) {
// If the formals have an ellipsis (eg the function accepts extra args) pass
// all available automatic arguments (which includes arguments specified on
// the command line via --arg/--argstr)
@@ -1762,7 +1763,7 @@ void EvalState::autoCallFunction(const Bindings & args, Value & fun, Value & res
attrs.insert(v);
} else {
// Otherwise, only pass the arguments that the function accepts
- for (auto & i : fun.lambda().fun->formals->formals) {
+ for (auto & i : formals->formals) {
auto j = args.get(i.name);
if (j) {
attrs.insert(*j);
@@ -2020,7 +2021,7 @@ void EvalState::concatLists(
void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
{
NixStringContext context;
- std::vector s;
+ std::vector strings;
size_t sSize = 0;
NixInt n{0};
NixFloat nf = 0;
@@ -2028,32 +2029,11 @@ void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
bool first = !forceString;
ValueType firstType = nString;
- const auto str = [&] {
- std::string result;
- result.reserve(sSize);
- for (const auto & part : s)
- result += *part;
- return result;
- };
- /* c_str() is not str().c_str() because we want to create a string
- Value. allocating a GC'd string directly and moving it into a
- Value lets us avoid an allocation and copy. */
- const auto c_str = [&] {
- char * result = allocString(sSize + 1);
- char * tmp = result;
- for (const auto & part : s) {
- memcpy(tmp, part->data(), part->size());
- tmp += part->size();
- }
- *tmp = 0;
- return result;
- };
-
// List of returned strings. References to these Values must NOT be persisted.
- SmallTemporaryValueVector values(es->size());
+ SmallTemporaryValueVector values(es.size());
Value * vTmpP = values.data();
- for (auto & [i_pos, i] : *es) {
+ for (auto & [i_pos, i] : es) {
Value & vTmp = *vTmpP++;
i->eval(state, env, vTmp);
@@ -2096,33 +2076,46 @@ void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
.withFrame(env, *this)
.debugThrow();
} else {
- if (s.empty())
- s.reserve(es->size());
+ if (strings.empty())
+ strings.reserve(es.size());
/* skip canonization of first path, which would only be not
canonized in the first place if it's coming from a ./${foo} type
path */
auto part = state.coerceToString(
i_pos, vTmp, context, "while evaluating a path segment", false, firstType == nString, !first);
sSize += part->size();
- s.emplace_back(std::move(part));
+ strings.emplace_back(std::move(part));
}
first = false;
}
- if (firstType == nInt)
+ if (firstType == nInt) {
v.mkInt(n);
- else if (firstType == nFloat)
+ } else if (firstType == nFloat) {
v.mkFloat(nf);
- else if (firstType == nPath) {
+ } else if (firstType == nPath) {
if (!context.empty())
state.error("a string that refers to a store path cannot be appended to a path")
.atPos(pos)
.withFrame(env, *this)
.debugThrow();
- v.mkPath(state.rootPath(CanonPath(str())));
- } else
- v.mkStringMove(c_str(), context);
+ std::string result_str;
+ result_str.reserve(sSize);
+ for (const auto & part : strings) {
+ result_str += *part;
+ }
+ v.mkPath(state.rootPath(CanonPath(result_str)));
+ } else {
+ char * result_str = allocString(sSize + 1);
+ char * tmp = result_str;
+ for (const auto & part : strings) {
+ memcpy(tmp, part->data(), part->size());
+ tmp += part->size();
+ }
+ *tmp = 0;
+ v.mkStringMove(result_str, context);
+ }
}
void ExprPos::eval(EvalState & state, Env & env, Value & v)
@@ -2160,30 +2153,28 @@ void EvalState::forceValueDeep(Value & v)
{
std::set seen;
- std::function recurse;
-
- recurse = [&](Value & v) {
+ [&, &state(*this)](this const auto & recurse, Value & v) {
if (!seen.insert(&v).second)
return;
- forceValue(v, v.determinePos(noPos));
+ state.forceValue(v, v.determinePos(noPos));
if (v.type() == nAttrs) {
for (auto & i : *v.attrs())
try {
// If the value is a thunk, we're evaling. Otherwise no trace necessary.
- auto dts = debugRepl && i.value->isThunk() ? makeDebugTraceStacker(
- *this,
- *i.value->thunk().expr,
- *i.value->thunk().env,
- i.pos,
- "while evaluating the attribute '%1%'",
- symbols[i.name])
- : nullptr;
+ auto dts = state.debugRepl && i.value->isThunk() ? makeDebugTraceStacker(
+ state,
+ *i.value->thunk().expr,
+ *i.value->thunk().env,
+ i.pos,
+ "while evaluating the attribute '%1%'",
+ state.symbols[i.name])
+ : nullptr;
recurse(*i.value);
} catch (Error & e) {
- addErrorTrace(e, i.pos, "while evaluating the attribute '%1%'", symbols[i.name]);
+ state.addErrorTrace(e, i.pos, "while evaluating the attribute '%1%'", state.symbols[i.name]);
throw;
}
}
@@ -2192,9 +2183,7 @@ void EvalState::forceValueDeep(Value & v)
for (auto v2 : v.listView())
recurse(*v2);
}
- };
-
- recurse(v);
+ }(v);
}
NixInt EvalState::forceInt(Value & v, const PosIdx pos, std::string_view errorCtx)
@@ -2377,12 +2366,15 @@ BackedStringView EvalState::coerceToString(
}
if (v.type() == nPath) {
- return !canonicalizePath && !copyToStore
- ? // FIXME: hack to preserve path literals that end in a
- // slash, as in /foo/${x}.
- v.pathStr()
- : copyToStore ? store->printStorePath(copyPathToStore(context, v.path()))
- : std::string(v.path().path.abs());
+ if (!canonicalizePath && !copyToStore) {
+ // FIXME: hack to preserve path literals that end in a
+ // slash, as in /foo/${x}.
+ return v.pathStrView();
+ } else if (copyToStore) {
+ return store->printStorePath(copyPathToStore(context, v.path()));
+ } else {
+ return std::string{v.path().path.abs()};
+ }
}
if (v.type() == nAttrs) {
@@ -2635,7 +2627,7 @@ void EvalState::assertEqValues(Value & v1, Value & v2, const PosIdx pos, std::st
return;
case nString:
- if (strcmp(v1.c_str(), v2.c_str()) != 0) {
+ if (v1.string_view() != v2.string_view()) {
error(
"string '%s' is not equal to string '%s'",
ValuePrinter(*this, v1, errorPrintOptions),
@@ -2652,7 +2644,7 @@ void EvalState::assertEqValues(Value & v1, Value & v2, const PosIdx pos, std::st
ValuePrinter(*this, v2, errorPrintOptions))
.debugThrow();
}
- if (strcmp(v1.pathStr(), v2.pathStr()) != 0) {
+ if (v1.pathStrView() != v2.pathStrView()) {
error(
"path '%s' is not equal to path '%s'",
ValuePrinter(*this, v1, errorPrintOptions),
@@ -2818,12 +2810,12 @@ bool EvalState::eqValues(Value & v1, Value & v2, const PosIdx pos, std::string_v
return v1.boolean() == v2.boolean();
case nString:
- return strcmp(v1.c_str(), v2.c_str()) == 0;
+ return v1.string_view() == v2.string_view();
case nPath:
return
// FIXME: compare accessors by their fingerprint.
- v1.pathAccessor() == v2.pathAccessor() && strcmp(v1.pathStr(), v2.pathStr()) == 0;
+ v1.pathAccessor() == v2.pathAccessor() && v1.pathStrView() == v2.pathStrView();
case nNull:
return true;
@@ -3067,7 +3059,7 @@ Expr * EvalState::parseExprFromFile(const SourcePath & path)
return parseExprFromFile(path, staticBaseEnv);
}
-Expr * EvalState::parseExprFromFile(const SourcePath & path, std::shared_ptr & staticEnv)
+Expr * EvalState::parseExprFromFile(const SourcePath & path, const std::shared_ptr & staticEnv)
{
auto buffer = path.resolveSymlinks().readFile();
// readFile hopefully have left some extra space for terminators
@@ -3075,8 +3067,8 @@ Expr * EvalState::parseExprFromFile(const SourcePath & path, std::shared_ptr & staticEnv)
+Expr * EvalState::parseExprFromString(
+ std::string s_, const SourcePath & basePath, const std::shared_ptr & staticEnv)
{
// NOTE this method (and parseStdin) must take care to *fully copy* their input
// into their respective Pos::Origin until the parser stops overwriting its input
@@ -3210,7 +3202,11 @@ std::optional EvalState::resolveLookupPathPath(const LookupPath::Pat
}
Expr * EvalState::parse(
- char * text, size_t length, Pos::Origin origin, const SourcePath & basePath, std::shared_ptr & staticEnv)
+ char * text,
+ size_t length,
+ Pos::Origin origin,
+ const SourcePath & basePath,
+ const std::shared_ptr & staticEnv)
{
DocCommentMap tmpDocComments; // Only used when not origin is not a SourcePath
DocCommentMap * docComments = &tmpDocComments;
@@ -3220,8 +3216,8 @@ Expr * EvalState::parse(
docComments = &it->second;
}
- auto result = parseExprFromBuf(
- text, length, origin, basePath, mem.exprs.alloc, symbols, settings, positions, *docComments, rootFS);
+ auto result =
+ parseExprFromBuf(text, length, origin, basePath, mem.exprs, symbols, settings, positions, *docComments, rootFS);
result->bindVars(*this, staticEnv);
diff --git a/src/libexpr/get-drvs.cc b/src/libexpr/get-drvs.cc
index 5a7281b2b..c4a2b00af 100644
--- a/src/libexpr/get-drvs.cc
+++ b/src/libexpr/get-drvs.cc
@@ -168,7 +168,7 @@ PackageInfo::Outputs PackageInfo::queryOutputs(bool withPaths, bool onlyOutputsT
for (auto elem : outTI->listView()) {
if (elem->type() != nString)
throw errMsg;
- auto out = outputs.find(elem->c_str());
+ auto out = outputs.find(elem->string_view());
if (out == outputs.end())
throw errMsg;
result.insert(*out);
@@ -245,7 +245,7 @@ std::string PackageInfo::queryMetaString(const std::string & name)
Value * v = queryMeta(name);
if (!v || v->type() != nString)
return "";
- return v->c_str();
+ return std::string{v->string_view()};
}
NixInt PackageInfo::queryMetaInt(const std::string & name, NixInt def)
@@ -258,7 +258,7 @@ NixInt PackageInfo::queryMetaInt(const std::string & name, NixInt def)
if (v->type() == nString) {
/* Backwards compatibility with before we had support for
integer meta fields. */
- if (auto n = string2Int(v->c_str()))
+ if (auto n = string2Int(v->string_view()))
return NixInt{*n};
}
return def;
@@ -274,7 +274,7 @@ NixFloat PackageInfo::queryMetaFloat(const std::string & name, NixFloat def)
if (v->type() == nString) {
/* Backwards compatibility with before we had support for
float meta fields. */
- if (auto n = string2Float(v->c_str()))
+ if (auto n = string2Float(v->string_view()))
return *n;
}
return def;
diff --git a/src/libexpr/include/nix/expr/attr-set.hh b/src/libexpr/include/nix/expr/attr-set.hh
index 46eecd9bd..f57302c42 100644
--- a/src/libexpr/include/nix/expr/attr-set.hh
+++ b/src/libexpr/include/nix/expr/attr-set.hh
@@ -5,6 +5,7 @@
#include "nix/expr/symbol-table.hh"
#include
+#include
#include
#include
@@ -463,12 +464,48 @@ private:
return bindings->baseLayer;
}
+ /**
+ * If the bindings gets "layered" on top of another we need to recalculate
+ * the number of unique attributes in the chain.
+ *
+ * This is done by either iterating over the base "layer" and the newly added
+ * attributes and counting duplicates. If the base "layer" is big this approach
+ * is inefficient and we fall back to doing per-element binary search in the base
+ * "layer".
+ */
void finishSizeIfNecessary()
{
- if (hasBaseLayer())
- /* NOTE: Do not use std::ranges::distance, since Bindings is a sized
- range, but we are calculating this size here. */
- bindings->numAttrsInChain = std::distance(bindings->begin(), bindings->end());
+ if (!hasBaseLayer())
+ return;
+
+ auto & base = *bindings->baseLayer;
+ auto attrs = std::span(bindings->attrs, bindings->numAttrs);
+
+ Bindings::size_type duplicates = 0;
+
+ /* If the base bindings is smaller than the newly added attributes
+ iterate using std::set_intersection to run in O(|base| + |attrs|) =
+ O(|attrs|). Otherwise use an O(|attrs| * log(|base|)) per-attr binary
+ search to check for duplicates. Note that if we are in this code path then
+ |attrs| <= bindingsUpdateLayerRhsSizeThreshold, which 16 by default. We are
+ optimizing for the case when a small attribute set gets "layered" on top of
+ a much larger one. When attrsets are already small it's fine to do a linear
+ scan, but we should avoid expensive iterations over large "base" attrsets. */
+ if (attrs.size() > base.size()) {
+ std::set_intersection(
+ base.begin(),
+ base.end(),
+ attrs.begin(),
+ attrs.end(),
+ boost::make_function_output_iterator([&]([[maybe_unused]] auto && _) { ++duplicates; }));
+ } else {
+ for (const auto & attr : attrs) {
+ if (base.get(attr.name))
+ ++duplicates;
+ }
+ }
+
+ bindings->numAttrsInChain = base.numAttrsInChain + attrs.size() - duplicates;
}
public:
diff --git a/src/libexpr/include/nix/expr/eval.hh b/src/libexpr/include/nix/expr/eval.hh
index 76ce62b87..0c7f9cf09 100644
--- a/src/libexpr/include/nix/expr/eval.hh
+++ b/src/libexpr/include/nix/expr/eval.hh
@@ -191,7 +191,7 @@ std::ostream & operator<<(std::ostream & os, const ValueType t);
struct RegexCache;
-std::shared_ptr makeRegexCache();
+ref makeRegexCache();
struct DebugTrace
{
@@ -372,6 +372,7 @@ public:
const fetchers::Settings & fetchSettings;
const EvalSettings & settings;
+
SymbolTable symbols;
PosTable positions;
@@ -418,7 +419,7 @@ public:
RootValue vImportedDrvToDerivation = nullptr;
- ref inputCache;
+ const ref inputCache;
/**
* Debugger
@@ -471,18 +472,18 @@ private:
/* Cache for calls to addToStore(); maps source paths to the store
paths. */
- ref> srcToStore;
+ const ref> srcToStore;
/**
* A cache that maps paths to "resolved" paths for importing Nix
* expressions, i.e. `/foo` to `/foo/default.nix`.
*/
- ref> importResolutionCache;
+ const ref> importResolutionCache;
/**
* A cache from resolved paths to values.
*/
- ref,
@@ -504,7 +505,7 @@ private:
/**
* Cache used by prim_match().
*/
- std::shared_ptr regexCache;
+ const ref regexCache;
public:
@@ -592,12 +593,13 @@ public:
* Parse a Nix expression from the specified file.
*/
Expr * parseExprFromFile(const SourcePath & path);
- Expr * parseExprFromFile(const SourcePath & path, std::shared_ptr & staticEnv);
+ Expr * parseExprFromFile(const SourcePath & path, const std::shared_ptr & staticEnv);
/**
* Parse a Nix expression from the specified string.
*/
- Expr * parseExprFromString(std::string s, const SourcePath & basePath, std::shared_ptr & staticEnv);
+ Expr *
+ parseExprFromString(std::string s, const SourcePath & basePath, const std::shared_ptr & staticEnv);
Expr * parseExprFromString(std::string s, const SourcePath & basePath);
Expr * parseStdin();
@@ -766,7 +768,7 @@ public:
#if NIX_USE_BOEHMGC
/** A GC root for the baseEnv reference. */
- std::shared_ptr baseEnvP;
+ const std::shared_ptr baseEnvP;
#endif
public:
@@ -780,7 +782,7 @@ public:
/**
* The same, but used during parsing to resolve variables.
*/
- std::shared_ptr staticBaseEnv; // !!! should be private
+ const std::shared_ptr staticBaseEnv; // !!! should be private
/**
* Internal primops not exposed to the user.
@@ -862,7 +864,7 @@ private:
size_t length,
Pos::Origin origin,
const SourcePath & basePath,
- std::shared_ptr & staticEnv);
+ const std::shared_ptr & staticEnv);
/**
* Current Nix call stack depth, used with `max-call-depth` setting to throw stack overflow hopefully before we run
diff --git a/src/libexpr/include/nix/expr/get-drvs.hh b/src/libexpr/include/nix/expr/get-drvs.hh
index 3d42188bf..4beccabe2 100644
--- a/src/libexpr/include/nix/expr/get-drvs.hh
+++ b/src/libexpr/include/nix/expr/get-drvs.hh
@@ -15,7 +15,7 @@ namespace nix {
struct PackageInfo
{
public:
- typedef std::map> Outputs;
+ typedef std::map, std::less<>> Outputs;
private:
EvalState * state;
diff --git a/src/libexpr/include/nix/expr/nixexpr.hh b/src/libexpr/include/nix/expr/nixexpr.hh
index 863a1369d..08d39cd87 100644
--- a/src/libexpr/include/nix/expr/nixexpr.hh
+++ b/src/libexpr/include/nix/expr/nixexpr.hh
@@ -13,6 +13,8 @@
#include "nix/expr/eval-error.hh"
#include "nix/util/pos-idx.hh"
#include "nix/expr/counter.hh"
+#include "nix/util/pos-table.hh"
+#include "nix/util/error.hh"
namespace nix {
@@ -89,13 +91,6 @@ std::string showAttrPath(const SymbolTable & symbols, std::span
using UpdateQueue = SmallTemporaryValueVector;
-class Exprs
-{
- std::pmr::monotonic_buffer_resource buffer;
-public:
- std::pmr::polymorphic_allocator alloc{&buffer};
-};
-
/* Abstract syntax of Nix expressions. */
struct Expr
@@ -442,8 +437,14 @@ struct ExprAttrs : Expr
struct ExprList : Expr
{
- std::vector elems;
- ExprList() {};
+ std::span elems;
+
+ ExprList(std::pmr::polymorphic_allocator & alloc, std::vector exprs)
+ : elems({alloc.allocate_object(exprs.size()), exprs.size()})
+ {
+ std::ranges::copy(exprs, elems.begin());
+ };
+
COMMON_METHODS
Value * maybeThunk(EvalState & state, Env & env) override;
@@ -460,7 +461,7 @@ struct Formal
Expr * def;
};
-struct Formals
+struct FormalsBuilder
{
typedef std::vector Formals_;
/**
@@ -475,6 +476,23 @@ struct Formals
formals.begin(), formals.end(), arg, [](const Formal & f, const Symbol & sym) { return f.name < sym; });
return it != formals.end() && it->name == arg;
}
+};
+
+struct Formals
+{
+ std::span formals;
+ bool ellipsis;
+
+ Formals(std::span formals, bool ellipsis)
+ : formals(formals)
+ , ellipsis(ellipsis) {};
+
+ bool has(Symbol arg) const
+ {
+ auto it = std::lower_bound(
+ formals.begin(), formals.end(), arg, [](const Formal & f, const Symbol & sym) { return f.name < sym; });
+ return it != formals.end() && it->name == arg;
+ }
std::vector lexicographicOrder(const SymbolTable & symbols) const
{
@@ -492,31 +510,71 @@ struct ExprLambda : Expr
PosIdx pos;
Symbol name;
Symbol arg;
- Formals * formals;
+
+private:
+ bool hasFormals;
+ bool ellipsis;
+ uint16_t nFormals;
+ Formal * formalsStart;
+public:
+
+ std::optional getFormals() const
+ {
+ if (hasFormals)
+ return Formals{{formalsStart, nFormals}, ellipsis};
+ else
+ return std::nullopt;
+ }
+
Expr * body;
DocComment docComment;
- ExprLambda(PosIdx pos, Symbol arg, Formals * formals, Expr * body)
+ ExprLambda(
+ const PosTable & positions,
+ std::pmr::polymorphic_allocator & alloc,
+ PosIdx pos,
+ Symbol arg,
+ const FormalsBuilder & formals,
+ Expr * body)
: pos(pos)
, arg(arg)
- , formals(formals)
- , body(body) {};
-
- ExprLambda(PosIdx pos, Formals * formals, Expr * body)
- : pos(pos)
- , formals(formals)
+ , hasFormals(true)
+ , ellipsis(formals.ellipsis)
+ , nFormals(formals.formals.size())
+ , formalsStart(alloc.allocate_object(nFormals))
, body(body)
{
- }
+ if (formals.formals.size() > nFormals) [[unlikely]] {
+ auto err = Error(
+ "too many formal arguments, implementation supports at most %1%",
+ std::numeric_limits::max());
+ if (pos)
+ err.atPos(positions[pos]);
+ throw err;
+ }
+ std::uninitialized_copy_n(formals.formals.begin(), nFormals, formalsStart);
+ };
+
+ ExprLambda(PosIdx pos, Symbol arg, Expr * body)
+ : pos(pos)
+ , arg(arg)
+ , hasFormals(false)
+ , ellipsis(false)
+ , nFormals(0)
+ , formalsStart(nullptr)
+ , body(body) {};
+
+ ExprLambda(
+ const PosTable & positions,
+ std::pmr::polymorphic_allocator & alloc,
+ PosIdx pos,
+ FormalsBuilder formals,
+ Expr * body)
+ : ExprLambda(positions, alloc, pos, Symbol(), formals, body) {};
void setName(Symbol name) override;
std::string showNamePos(const EvalState & state) const;
- inline bool hasFormals() const
- {
- return formals != nullptr;
- }
-
PosIdx getPos() const override
{
return pos;
@@ -572,8 +630,8 @@ struct ExprLet : Expr
struct ExprWith : Expr
{
PosIdx pos;
+ uint32_t prevWith;
Expr *attrs, *body;
- size_t prevWith;
ExprWith * parentWith;
ExprWith(const PosIdx & pos, Expr * attrs, Expr * body)
: pos(pos)
@@ -695,11 +753,19 @@ struct ExprConcatStrings : Expr
{
PosIdx pos;
bool forceString;
- std::vector> * es;
- ExprConcatStrings(const PosIdx & pos, bool forceString, std::vector> * es)
+ std::span> es;
+
+ ExprConcatStrings(
+ std::pmr::polymorphic_allocator & alloc,
+ const PosIdx & pos,
+ bool forceString,
+ const std::vector> & es)
: pos(pos)
, forceString(forceString)
- , es(es) {};
+ , es({alloc.allocate_object>(es.size()), es.size()})
+ {
+ std::ranges::copy(es, this->es.begin());
+ };
PosIdx getPos() const override
{
@@ -737,6 +803,49 @@ struct ExprBlackHole : Expr
extern ExprBlackHole eBlackHole;
+class Exprs
+{
+ std::pmr::monotonic_buffer_resource buffer;
+public:
+ std::pmr::polymorphic_allocator alloc{&buffer};
+
+ template
+ [[gnu::always_inline]]
+ C * add(auto &&... args)
+ {
+ return alloc.new_object(std::forward(args)...);
+ }
+
+ // we define some calls to add explicitly so that the argument can be passed in as initializer lists
+ template
+ [[gnu::always_inline]]
+ C * add(const PosIdx & pos, Expr * fun, std::vector && args)
+ requires(std::same_as)
+ {
+ return alloc.new_object(pos, fun, std::move(args));
+ }
+
+ template
+ [[gnu::always_inline]]
+ C * add(const PosIdx & pos, Expr * fun, std::vector && args, PosIdx && cursedOrEndPos)
+ requires(std::same_as)
+ {
+ return alloc.new_object(pos, fun, std::move(args), std::move(cursedOrEndPos));
+ }
+
+ template
+ [[gnu::always_inline]]
+ C *
+ add(std::pmr::polymorphic_allocator & alloc,
+ const PosIdx & pos,
+ bool forceString,
+ const std::vector> & es)
+ requires(std::same_as)
+ {
+ return alloc.new_object(alloc, pos, forceString, es);
+ }
+};
+
/* Static environments are used to map variable names onto (level,
displacement) pairs used to obtain the value of the variable at
runtime. */
diff --git a/src/libexpr/include/nix/expr/parser-state.hh b/src/libexpr/include/nix/expr/parser-state.hh
index 55dce3047..89c424f82 100644
--- a/src/libexpr/include/nix/expr/parser-state.hh
+++ b/src/libexpr/include/nix/expr/parser-state.hh
@@ -78,7 +78,7 @@ struct LexerState
struct ParserState
{
const LexerState & lexerState;
- std::pmr::polymorphic_allocator & alloc;
+ Exprs & exprs;
SymbolTable & symbols;
PosTable & positions;
Expr * result;
@@ -93,7 +93,7 @@ struct ParserState
void addAttr(
ExprAttrs * attrs, AttrPath && attrPath, const ParserLocation & loc, Expr * e, const ParserLocation & exprLoc);
void addAttr(ExprAttrs * attrs, AttrPath & attrPath, const Symbol & symbol, ExprAttrs::AttrDef && def);
- Formals * validateFormals(Formals * formals, PosIdx pos = noPos, Symbol arg = {});
+ void validateFormals(FormalsBuilder & formals, PosIdx pos = noPos, Symbol arg = {});
Expr * stripIndentation(const PosIdx pos, std::vector>> && es);
PosIdx at(const ParserLocation & loc);
};
@@ -132,11 +132,11 @@ inline void ParserState::addAttr(
dupAttr(attrPath, pos, j->second.pos);
}
} else {
- nested = new ExprAttrs;
+ nested = exprs.add();
attrs->attrs[i->symbol] = ExprAttrs::AttrDef(nested, pos);
}
} else {
- nested = new ExprAttrs;
+ nested = exprs.add();
attrs->dynamicAttrs.push_back(ExprAttrs::DynamicAttrDef(i->expr, nested, pos));
}
attrs = nested;
@@ -213,17 +213,17 @@ ParserState::addAttr(ExprAttrs * attrs, AttrPath & attrPath, const Symbol & symb
}
}
-inline Formals * ParserState::validateFormals(Formals * formals, PosIdx pos, Symbol arg)
+inline void ParserState::validateFormals(FormalsBuilder & formals, PosIdx pos, Symbol arg)
{
- std::sort(formals->formals.begin(), formals->formals.end(), [](const auto & a, const auto & b) {
+ std::sort(formals.formals.begin(), formals.formals.end(), [](const auto & a, const auto & b) {
return std::tie(a.name, a.pos) < std::tie(b.name, b.pos);
});
std::optional> duplicate;
- for (size_t i = 0; i + 1 < formals->formals.size(); i++) {
- if (formals->formals[i].name != formals->formals[i + 1].name)
+ for (size_t i = 0; i + 1 < formals.formals.size(); i++) {
+ if (formals.formals[i].name != formals.formals[i + 1].name)
continue;
- std::pair thisDup{formals->formals[i].name, formals->formals[i + 1].pos};
+ std::pair thisDup{formals.formals[i].name, formals.formals[i + 1].pos};
duplicate = std::min(thisDup, duplicate.value_or(thisDup));
}
if (duplicate)
@@ -231,18 +231,16 @@ inline Formals * ParserState::validateFormals(Formals * formals, PosIdx pos, Sym
{.msg = HintFmt("duplicate formal function argument '%1%'", symbols[duplicate->first]),
.pos = positions[duplicate->second]});
- if (arg && formals->has(arg))
+ if (arg && formals.has(arg))
throw ParseError(
{.msg = HintFmt("duplicate formal function argument '%1%'", symbols[arg]), .pos = positions[pos]});
-
- return formals;
}
inline Expr *
ParserState::stripIndentation(const PosIdx pos, std::vector>> && es)
{
if (es.empty())
- return new ExprString("");
+ return exprs.add("");
/* Figure out the minimum indentation. Note that by design
whitespace-only final lines are not taken into account. (So
@@ -282,7 +280,7 @@ ParserState::stripIndentation(const PosIdx pos, std::vector>;
+ std::vector> es2{};
atStartOfLine = true;
size_t curDropped = 0;
size_t n = es.size();
@@ -290,7 +288,7 @@ ParserState::stripIndentation(const PosIdx pos, std::vectoremplace_back(i->first, e);
+ es2.emplace_back(i->first, e);
};
const auto trimString = [&](const StringToken & t) {
std::string s2;
@@ -324,7 +322,7 @@ ParserState::stripIndentation(const PosIdx pos, std::vectoremplace_back(i->first, new ExprString(alloc, s2));
+ es2.emplace_back(i->first, exprs.add(exprs.alloc, s2));
}
};
for (; i != es.end(); ++i, --n) {
@@ -333,19 +331,17 @@ ParserState::stripIndentation(const PosIdx pos, std::vectorsize() == 0) {
- auto * const result = new ExprString("");
- delete es2;
+ if (es2.size() == 0) {
+ auto * const result = exprs.add("");
return result;
}
/* If this is a single string, then don't do a concatenation. */
- if (es2->size() == 1 && dynamic_cast((*es2)[0].second)) {
- auto * const result = (*es2)[0].second;
- delete es2;
+ if (es2.size() == 1 && dynamic_cast((es2)[0].second)) {
+ auto * const result = (es2)[0].second;
return result;
}
- return new ExprConcatStrings(pos, true, es2);
+ return exprs.add(exprs.alloc, pos, true, std::move(es2));
}
inline PosIdx LexerState::at(const ParserLocation & loc)
diff --git a/src/libexpr/include/nix/expr/value.hh b/src/libexpr/include/nix/expr/value.hh
index 22d85dc99..706a4fe3f 100644
--- a/src/libexpr/include/nix/expr/value.hh
+++ b/src/libexpr/include/nix/expr/value.hh
@@ -1109,7 +1109,7 @@ public:
std::string_view string_view() const noexcept
{
- return std::string_view(getStorage().c_str);
+ return std::string_view{getStorage().c_str};
}
const char * c_str() const noexcept
@@ -1177,6 +1177,11 @@ public:
return getStorage().path;
}
+ std::string_view pathStrView() const noexcept
+ {
+ return std::string_view{getStorage().path};
+ }
+
SourceAccessor * pathAccessor() const noexcept
{
return getStorage().accessor;
diff --git a/src/libexpr/lexer.l b/src/libexpr/lexer.l
index f420fc13f..810503bdc 100644
--- a/src/libexpr/lexer.l
+++ b/src/libexpr/lexer.l
@@ -142,11 +142,11 @@ or { return OR_KW; }
return PIPE_INTO;
}
-{ID} { yylval->id = {yytext, (size_t) yyleng}; return ID; }
+{ID} { yylval->emplace(yytext, (size_t) yyleng); return ID; }
{INT} { errno = 0;
std::optional numMay = string2Int(yytext);
if (numMay.has_value()) {
- yylval->n = NixInt{*numMay};
+ yylval->emplace(*numMay);
} else {
throw ParseError(ErrorInfo{
.msg = HintFmt("invalid integer '%1%'", yytext),
@@ -156,7 +156,7 @@ or { return OR_KW; }
return INT_LIT;
}
{FLOAT} { errno = 0;
- yylval->nf = strtod(yytext, 0);
+ yylval->emplace(strtod(yytext, 0));
if (errno != 0)
throw ParseError(ErrorInfo{
.msg = HintFmt("invalid float '%1%'", yytext),
@@ -183,7 +183,7 @@ or { return OR_KW; }
/* It is impossible to match strings ending with '$' with one
regex because trailing contexts are only valid at the end
of a rule. (A sane but undocumented limitation.) */
- yylval->str = unescapeStr(yytext, yyleng, [&]() { return state->positions[CUR_POS]; });
+ yylval->emplace(unescapeStr(yytext, yyleng, [&]() { return state->positions[CUR_POS]; }));
return STR;
}
\$\{ { PUSH_STATE(DEFAULT); return DOLLAR_CURLY; }
@@ -198,27 +198,27 @@ or { return OR_KW; }
\'\'(\ *\n)? { PUSH_STATE(IND_STRING); return IND_STRING_OPEN; }
([^\$\']|\$[^\{\']|\'[^\'\$])+ {
- yylval->str = {yytext, (size_t) yyleng, true};
- forceNoNullByte(yylval->str, [&]() { return state->positions[CUR_POS]; });
+ yylval->emplace(yytext, (size_t) yyleng, true);
+ forceNoNullByte(yylval->as(), [&]() { return state->positions[CUR_POS]; });
return IND_STR;
}
\'\'\$ |
\$ {
- yylval->str = {"$", 1};
+ yylval->emplace("$", 1);
return IND_STR;
}
\'\'\' {
- yylval->str = {"''", 2};
+ yylval->emplace("''", 2);
return IND_STR;
}
\'\'\\{ANY} {
- yylval->str = unescapeStr(yytext + 2, yyleng - 2, [&]() { return state->positions[CUR_POS]; });
+ yylval->emplace(unescapeStr(yytext + 2, yyleng - 2, [&]() { return state->positions[CUR_POS]; }));
return IND_STR;
}
\$\{ { PUSH_STATE(DEFAULT); return DOLLAR_CURLY; }
\'\' { POP_STATE(); return IND_STRING_CLOSE; }
\' {
- yylval->str = {"'", 1};
+ yylval->emplace("'", 1);
return IND_STR;
}
@@ -232,23 +232,31 @@ or { return OR_KW; }
{PATH_SEG} {
POP_STATE();
PUSH_STATE(INPATH_SLASH);
- yylval->path = {yytext, (size_t) yyleng};
+ yylval->emplace(yytext, (size_t) yyleng);
return PATH;
}
{HPATH_START} {
POP_STATE();
PUSH_STATE(INPATH_SLASH);
- yylval->path = {yytext, (size_t) yyleng};
+ yylval->emplace(yytext, (size_t) yyleng);
return HPATH;
}
+{ANY} |
+<> {
+ /* This should be unreachable: PATH_START is only entered after matching
+ PATH_SEG or HPATH_START, and we rewind to re-parse those same patterns.
+ This rule exists to satisfy flex's %option nodefault requirement. */
+ unreachable();
+}
+
{PATH} {
if (yytext[yyleng-1] == '/')
PUSH_STATE(INPATH_SLASH);
else
PUSH_STATE(INPATH);
- yylval->path = {yytext, (size_t) yyleng};
+ yylval->emplace(yytext, (size_t) yyleng);
return PATH;
}
{HPATH} {
@@ -256,7 +264,7 @@ or { return OR_KW; }
PUSH_STATE(INPATH_SLASH);
else
PUSH_STATE(INPATH);
- yylval->path = {yytext, (size_t) yyleng};
+ yylval->emplace(yytext, (size_t) yyleng);
return HPATH;
}
@@ -272,7 +280,7 @@ or { return OR_KW; }
PUSH_STATE(INPATH_SLASH);
else
PUSH_STATE(INPATH);
- yylval->str = {yytext, (size_t) yyleng};
+ yylval->emplace(yytext, (size_t) yyleng);
return STR;
}
{ANY} |
@@ -294,8 +302,8 @@ or { return OR_KW; }
});
}
-{SPATH} { yylval->path = {yytext, (size_t) yyleng}; return SPATH; }
-{URI} { yylval->uri = {yytext, (size_t) yyleng}; return URI; }
+{SPATH} { yylval->emplace(yytext, (size_t) yyleng); return SPATH; }
+{URI} { yylval->emplace(yytext, (size_t) yyleng); return URI; }
%{
// Doc comment rule
diff --git a/src/libexpr/meson.build b/src/libexpr/meson.build
index 1314ab65b..18c4c7fa3 100644
--- a/src/libexpr/meson.build
+++ b/src/libexpr/meson.build
@@ -183,17 +183,62 @@ subdir('primops')
subdir('nix-meson-build-support/export-all-symbols')
subdir('nix-meson-build-support/windows-version')
+# Turns out that Bison/Flex are particularly sensitive to compilers
+# failing to inline functions. For that reason we crank up the inlining
+# threshold manually for optimized builds. Yes, this can be considered 'ricing'
+# the compiler, but it does pay off.
+#
+# NOTE: missed inlining can be spotted (for Clang) using -Rpass-missed=inline
+# and -fdump-ipa-inline-missed (for GCC).
+parser_library_cpp_args = []
+
+if not get_option('debug')
+ if cxx.get_id() == 'clang'
+ # The default as of LLVM 21 is 225:
+ # llc --help-hidden | grep inline-threshold
+ parser_library_cpp_args += [
+ '-mllvm',
+ '-inline-threshold=5000',
+ ]
+ elif cxx.get_id() == 'gcc'
+ parser_library_cpp_args += [
+ '--param=max-inline-insns-single=1000',
+ '--param=max-inline-insns-auto=1000',
+ '--param=inline-unit-growth=400',
+ ]
+ endif
+endif
+
+# Working around https://github.com/mesonbuild/meson/issues/1367.
+parser_library = static_library(
+ 'nixexpr-parser',
+ parser_tab,
+ lexer_tab,
+ cpp_args : parser_library_cpp_args,
+ dependencies : deps_public + deps_private + deps_other,
+ include_directories : include_dirs,
+ # 1. Stdlib and regular assertions regress parser performance significantly, so build without
+ # them for this one library when building in a release configuration.
+ # 2. Disable LTO for GCC because then inlining flags won't apply, since LTO in GCC is done
+ # by plonking down GIMPLE in the archive.
+ override_options : [
+ 'b_ndebug=@0@'.format(not get_option('debug')),
+ 'b_lto=@0@'.format(get_option('b_lto') and cxx.get_id() != 'gcc'),
+ ],
+)
+
this_library = library(
'nixexpr',
sources,
config_priv_h,
- parser_tab,
- lexer_tab,
+ parser_tab[1],
+ lexer_tab[1],
generated_headers,
soversion : nix_soversion,
dependencies : deps_public + deps_private + deps_other,
include_directories : include_dirs,
link_args : linker_export_flags,
+ link_whole : [ parser_library ],
prelink : true, # For C++ static initializers
install : true,
cpp_pch : do_pch ? [ 'pch/precompiled-headers.hh' ] : [],
diff --git a/src/libexpr/nixexpr.cc b/src/libexpr/nixexpr.cc
index 5b9d17d49..db5e52c3a 100644
--- a/src/libexpr/nixexpr.cc
+++ b/src/libexpr/nixexpr.cc
@@ -45,7 +45,7 @@ void ExprString::show(const SymbolTable & symbols, std::ostream & str) const
void ExprPath::show(const SymbolTable & symbols, std::ostream & str) const
{
- str << v.pathStr();
+ str << v.pathStrView();
}
void ExprVar::show(const SymbolTable & symbols, std::ostream & str) const
@@ -154,7 +154,7 @@ void ExprList::show(const SymbolTable & symbols, std::ostream & str) const
void ExprLambda::show(const SymbolTable & symbols, std::ostream & str) const
{
str << "(";
- if (hasFormals()) {
+ if (auto formals = getFormals()) {
str << "{ ";
bool first = true;
// the natural Symbol ordering is by creation time, which can lead to the
@@ -171,7 +171,7 @@ void ExprLambda::show(const SymbolTable & symbols, std::ostream & str) const
i.def->show(symbols, str);
}
}
- if (formals->ellipsis) {
+ if (ellipsis) {
if (!first)
str << ", ";
str << "...";
@@ -246,7 +246,7 @@ void ExprConcatStrings::show(const SymbolTable & symbols, std::ostream & str) co
{
bool first = true;
str << "(";
- for (auto & i : *es) {
+ for (auto & i : es) {
if (first)
first = false;
else
@@ -452,14 +452,14 @@ void ExprLambda::bindVars(EvalState & es, const std::shared_ptr
es.exprEnvs.insert(std::make_pair(this, env));
auto newEnv =
- std::make_shared(nullptr, env, (hasFormals() ? formals->formals.size() : 0) + (!arg ? 0 : 1));
+ std::make_shared(nullptr, env, (getFormals() ? getFormals()->formals.size() : 0) + (!arg ? 0 : 1));
Displacement displ = 0;
if (arg)
newEnv->vars.emplace_back(arg, displ++);
- if (hasFormals()) {
+ if (auto formals = getFormals()) {
for (auto & i : formals->formals)
newEnv->vars.emplace_back(i.name, displ++);
@@ -523,6 +523,7 @@ void ExprWith::bindVars(EvalState & es, const std::shared_ptr &
prevWith = 0;
for (curEnv = env.get(), level = 1; curEnv; curEnv = curEnv->up.get(), level++)
if (curEnv->isWith) {
+ assert(level <= std::numeric_limits::max());
prevWith = level;
break;
}
@@ -564,7 +565,7 @@ void ExprConcatStrings::bindVars(EvalState & es, const std::shared_ptres)
+ for (auto & i : this->es)
i.second->bindVars(es, env);
}
diff --git a/src/libexpr/parser.y b/src/libexpr/parser.y
index 9186fcf4b..8b56eba15 100644
--- a/src/libexpr/parser.y
+++ b/src/libexpr/parser.y
@@ -14,6 +14,10 @@
%code requires {
+// bison adds a bunch of switch statements with default:
+#pragma GCC diagnostic push
+#pragma GCC diagnostic ignored "-Wswitch-enum"
+
#ifndef BISON_HEADER
#define BISON_HEADER
@@ -59,7 +63,7 @@ Expr * parseExprFromBuf(
size_t length,
Pos::Origin origin,
const SourcePath & basePath,
- std::pmr::polymorphic_allocator & alloc,
+ Exprs & exprs,
SymbolTable & symbols,
const EvalSettings & settings,
PosTable & positions,
@@ -109,57 +113,39 @@ static void setDocPosition(const LexerState & lexerState, ExprLambda * lambda, P
}
}
-static Expr * makeCall(PosIdx pos, Expr * fn, Expr * arg) {
+static Expr * makeCall(Exprs & exprs, PosIdx pos, Expr * fn, Expr * arg) {
if (auto e2 = dynamic_cast(fn)) {
e2->args.push_back(arg);
return fn;
}
- return new ExprCall(pos, fn, {arg});
+ return exprs.add(pos, fn, {arg});
}
%}
-%union {
- // !!! We're probably leaking stuff here.
- nix::Expr * e;
- nix::ExprList * list;
- nix::ExprAttrs * attrs;
- nix::Formals * formals;
- nix::Formal * formal;
- nix::NixInt n;
- nix::NixFloat nf;
- nix::StringToken id; // !!! -> Symbol
- nix::StringToken path;
- nix::StringToken uri;
- nix::StringToken str;
- std::vector * attrNames;
- std::vector> * inheritAttrs;
- std::vector> * string_parts;
- std::variant * to_be_string;
- std::vector>> * ind_string_parts;
-}
+%define api.value.type variant
-%type start expr expr_function expr_if expr_op
-%type expr_select expr_simple expr_app
-%type expr_pipe_from expr_pipe_into
-%type expr_list
-%type binds binds1
-%type formals formal_set
-%type formal
-%type attrpath
-%type attrs
-%type string_parts_interpolated
-%type ind_string_parts
-%type path_start
-%type string_parts string_attr
-%type attr
-%token ID
-%token STR IND_STR
-%token INT_LIT
-%token FLOAT_LIT
-%token PATH HPATH SPATH PATH_END
-%token URI
+%type start expr expr_function expr_if expr_op
+%type expr_select expr_simple expr_app
+%type expr_pipe_from expr_pipe_into
+%type > list
+%type binds binds1
+%type formals formal_set
+%type formal
+%type > attrpath
+%type >> attrs
+%type >> string_parts_interpolated
+%type >>> ind_string_parts
+%type path_start
+%type > string_parts string_attr
+%type attr
+%token ID
+%token STR IND_STR
+%token INT_LIT
+%token FLOAT_LIT
+%token PATH HPATH SPATH PATH_END
+%token URI
%token IF THEN ELSE ASSERT WITH LET IN_KW REC INHERIT EQ NEQ AND OR IMPL OR_KW
%token PIPE_FROM PIPE_INTO /* <| and |> */
%token DOLLAR_CURLY /* == ${ */
@@ -193,86 +179,90 @@ expr: expr_function;
expr_function
: ID ':' expr_function
- { auto me = new ExprLambda(CUR_POS, state->symbols.create($1), 0, $3);
+ { auto me = state->exprs.add(CUR_POS, state->symbols.create($1), $3);
$$ = me;
SET_DOC_POS(me, @1);
}
| formal_set ':' expr_function[body]
- { auto me = new ExprLambda(CUR_POS, state->validateFormals($formal_set), $body);
+ {
+ state->validateFormals($formal_set);
+ auto me = state->exprs.add(state->positions, state->exprs.alloc, CUR_POS, std::move($formal_set), $body);
$$ = me;
SET_DOC_POS(me, @1);
}
| formal_set '@' ID ':' expr_function[body]
{
auto arg = state->symbols.create($ID);
- auto me = new ExprLambda(CUR_POS, arg, state->validateFormals($formal_set, CUR_POS, arg), $body);
+ state->validateFormals($formal_set, CUR_POS, arg);
+ auto me = state->exprs.add(state->positions, state->exprs.alloc, CUR_POS, arg, std::move($formal_set), $body);
$$ = me;
SET_DOC_POS(me, @1);
}
| ID '@' formal_set ':' expr_function[body]
{
auto arg = state->symbols.create($ID);
- auto me = new ExprLambda(CUR_POS, arg, state->validateFormals($formal_set, CUR_POS, arg), $body);
+ state->validateFormals($formal_set, CUR_POS, arg);
+ auto me = state->exprs.add(state->positions, state->exprs.alloc, CUR_POS, arg, std::move($formal_set), $body);
$$ = me;
SET_DOC_POS(me, @1);
}
| ASSERT expr ';' expr_function
- { $$ = new ExprAssert(CUR_POS, $2, $4); }
+ { $$ = state->exprs.add(CUR_POS, $2, $4); }
| WITH expr ';' expr_function
- { $$ = new ExprWith(CUR_POS, $2, $4); }
+ { $$ = state->exprs.add(CUR_POS, $2, $4); }
| LET binds IN_KW expr_function
{ if (!$2->dynamicAttrs.empty())
throw ParseError({
.msg = HintFmt("dynamic attributes not allowed in let"),
.pos = state->positions[CUR_POS]
});
- $$ = new ExprLet($2, $4);
+ $$ = state->exprs.add($2, $4);
}
| expr_if
;
expr_if
- : IF expr THEN expr ELSE expr { $$ = new ExprIf(CUR_POS, $2, $4, $6); }
+ : IF expr THEN expr ELSE expr { $$ = state->exprs.add(CUR_POS, $2, $4, $6); }
| expr_pipe_from
| expr_pipe_into
| expr_op
;
expr_pipe_from
- : expr_op PIPE_FROM expr_pipe_from { $$ = makeCall(state->at(@2), $1, $3); }
- | expr_op PIPE_FROM expr_op { $$ = makeCall(state->at(@2), $1, $3); }
+ : expr_op PIPE_FROM expr_pipe_from { $$ = makeCall(state->exprs, state->at(@2), $1, $3); }
+ | expr_op PIPE_FROM expr_op { $$ = makeCall(state->exprs, state->at(@2), $1, $3); }
;
expr_pipe_into
- : expr_pipe_into PIPE_INTO expr_op { $$ = makeCall(state->at(@2), $3, $1); }
- | expr_op PIPE_INTO expr_op { $$ = makeCall(state->at(@2), $3, $1); }
+ : expr_pipe_into PIPE_INTO expr_op { $$ = makeCall(state->exprs, state->at(@2), $3, $1); }
+ | expr_op PIPE_INTO expr_op { $$ = makeCall(state->exprs, state->at(@2), $3, $1); }
;
expr_op
- : '!' expr_op %prec NOT { $$ = new ExprOpNot($2); }
- | '-' expr_op %prec NEGATE { $$ = new ExprCall(CUR_POS, new ExprVar(state->s.sub), {new ExprInt(0), $2}); }
- | expr_op EQ expr_op { $$ = new ExprOpEq($1, $3); }
- | expr_op NEQ expr_op { $$ = new ExprOpNEq($1, $3); }
- | expr_op '<' expr_op { $$ = new ExprCall(state->at(@2), new ExprVar(state->s.lessThan), {$1, $3}); }
- | expr_op LEQ expr_op { $$ = new ExprOpNot(new ExprCall(state->at(@2), new ExprVar(state->s.lessThan), {$3, $1})); }
- | expr_op '>' expr_op { $$ = new ExprCall(state->at(@2), new ExprVar(state->s.lessThan), {$3, $1}); }
- | expr_op GEQ expr_op { $$ = new ExprOpNot(new ExprCall(state->at(@2), new ExprVar(state->s.lessThan), {$1, $3})); }
- | expr_op AND expr_op { $$ = new ExprOpAnd(state->at(@2), $1, $3); }
- | expr_op OR expr_op { $$ = new ExprOpOr(state->at(@2), $1, $3); }
- | expr_op IMPL expr_op { $$ = new ExprOpImpl(state->at(@2), $1, $3); }
- | expr_op UPDATE expr_op { $$ = new ExprOpUpdate(state->at(@2), $1, $3); }
- | expr_op '?' attrpath { $$ = new ExprOpHasAttr(state->alloc, $1, std::move(*$3)); delete $3; }
+ : '!' expr_op %prec NOT { $$ = state->exprs.add($2); }
+ | '-' expr_op %prec NEGATE { $$ = state->exprs.add(CUR_POS, state->exprs.add(state->s.sub), {state->exprs.add(0), $2}); }
+ | expr_op EQ expr_op { $$ = state->exprs.add($1, $3); }
+ | expr_op NEQ expr_op { $$ = state->exprs.add($1, $3); }
+ | expr_op '<' expr_op { $$ = state->exprs.add(state->at(@2), state->exprs.add(state->s.lessThan), {$1, $3}); }
+ | expr_op LEQ expr_op { $$ = state->exprs.add(state->exprs.add(state->at(@2), state->exprs.add(state->s.lessThan), {$3, $1})); }
+ | expr_op '>' expr_op { $$ = state->exprs.add(state->at(@2), state->exprs.add(state->s.lessThan), {$3, $1}); }
+ | expr_op GEQ expr_op { $$ = state->exprs.add(state->exprs.add(state->at(@2), state->exprs.add(state->s.lessThan), {$1, $3})); }
+ | expr_op AND expr_op { $$ = state->exprs.add(state->at(@2), $1, $3); }
+ | expr_op OR expr_op { $$ = state->exprs.add