On FreeBSD, sysctl(KERN_PROC_PATHNAME) returns a null-terminated
string with pathLen including the terminator. This causes Nix to
fail during manual generation with:
error:
… while calling the 'concatStringsSep' builtin
at /nix/var/nix/builds/nix-63232-402489527/source/doc/manual/generate-settings.nix:99:1:
98| in
99| concatStrings (attrValues (mapAttrs (showSetting prefix) settingsInfo))
| ^
100|
error: input string '/nix/store/gq89cj02b5zs67cbd85vzg5cgsgnd8mj-nix-2.31.2/bin/nix␀'
cannot be represented as Nix string because it contains null bytes
The issue occurs because generate-settings.nix reads the nix binary
path from JSON and evaluates it as a Nix string, which cannot contain
null bytes. Normal C++ string operations don't trigger this since they
handle null-terminated strings correctly.
Strip the null terminator on FreeBSD to match other platforms (Linux
uses /proc/self/exe, macOS uses _NSGetExecutablePath).
Credit: @wahjava (FreeBSD ports and Nixpkgs contributor)
The use of sourceToSink is an unnecessary serialization bottleneck.
While we are at it, fix the copyRecursive implementation to actually copy
the whole directory. It wasn't used for anything prior, but now it has a use
and accompanying tests for flake clone.
Also do a better JSON and testing for deep and shallow NAR listings.
As documented, this for file system objects themselves, since
`MemorySourceAccessor` is an implementation detail.
`listNar` did the not-so-pretty thing of going straight to JSON. Now it
uses `MemorySourceAccessor::File`, or rather variations of it, to go to
a C++ data type first, and only JSON second.
To accomplish this we add some type parameters to the `File` data type.
Actually, we need to do two rounds of this, because shallow NAR
listings. There is `FileT` and `DirectoryT` accordingly.
This matches the "NAR Listing" JSON format, and also helps distinguish
from regular file contents.
Why we want to match that will become clear in the next comments, when
we will in fact use (variations of) this data type for NAR listings.
This is necessary to ban symlink following. It can be considered
a defense in depth against issues similar to CVE-2024-45593. By
slightly changing the API in a follow-up commit we will be able
to mitigate the symlink following issue for good.
This is an example of "Parse, don't validate" principle [1].
Before, we had a number of `StringSet`s in `DerivationOptions` that
were not *actually* allowed to be arbitrary sets of strings. Instead,
each set member had to be one of:
- a store path
- a CA "downstream placeholder"
- an output name
Only later, in the code that checks outputs, would these strings be
further parsed to match these cases. (Actually, only 2 by that point,
because the placeholders must be rewritten away by then.)
Now, we fully parse everything up front, and have an "honest" data type
that reflects these invariants:
- store paths are parsed, stored as (opaque) deriving paths
- CA "downstream placeholders" are rewritten to the output deriving
paths they denote
- output names are the only arbitrary strings left
Since the first two cases both become deriving paths, that leaves us
with a `std::variant<SingleDerivedPath, String>` data type, which we use
in our sets instead.
Getting rid of placeholders is especially nice because we are replacing
them with something much more internally-structured / transparent.
[1]: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Several bugs to squash:
- Apparently DELETE is an already used macro with Win32. We can avoid it
by using Camel case instead (slightly hacky but also fits the naming
convention better)
- Gets rid of the raw usage of isatty. Added an isTTY impl to abstract over
the raw API.
Typically PosixSourceAccessor can be used from multiple threads,
but mtime is not updated atomically (i.e. with compare_exchange_weak),
so mtime gets raced. It's only needed in dumpPathAndGetMtime and mtime
tracking can be gated behind that.
Also start using getLastModified interface instead of dynamic casts.
The variant has on the left-hand side the topologically sorted vector
and the right-hand side is a pair showing the path and its parent that
represent a cycle in the graph making the sort impossible.
This change prepares for enhanced cycle error messages that can provide
more context about the cycle. The variant approach allows callers to
handle cycles more flexibly, enabling better error reporting that shows
the full cycle path and which files are involved.
Adapted from Lix commit f7871fcb5.
Change-Id: I70a987f470437df8beb3b1cc203ff88701d0aa1b
Co-Authored-By: Maximilian Bosch <maximilian@mbosch.me>
Otherwise PosTable grows indefinitely for each reload. Since
the total input size is limited to 4GB (uint32_t for byte offset PosIdx)
it can get exhausted pretty. This ensures that we don't waste memory
on reloads as well.
It turns out this code path is only used for unit tests (to ensure our
JSON formats are possible to parse by other code, elsewhere). No
user-facing functionality consumes this format.
Therefore, let's drop the old version parsing support.
This was already dropped in `inputFromURL()`, but not in
`inputFromAttrs()`. Now it's done in `fixGitURL()`, which is used by
both.
In principle, `git+` shouldn't be used in the `url` attribute, since
we already know that it's a Git URL. But since it currently works, we
don't want to break it.
Fixes#14429.
Introduce a `CompressedSource` class in libutil's `serialise.hh` that
compresses a `RestartableSource` and owns the compressed data. This is a
general-purpose utility that can be used anywhere compressed data needs
to be treated as a source.
The `showBytes()` function was redundant with `renderSize()` as the
latter automatically selects the appropriate unit (KiB, MiB, GiB, etc.)
based on the value, whereas `showBytes()` always formatted as MiB
regardless of size.
Co-authored-by: Bernardo Meurer Costa <beme@anthropic.com>
In particular
- Remove `get`, it is redundant with `valueAt` and the `get` in
`util.hh`.
- Remove `nullableValueAt`. It is morally just the function composition
`getNullable . valueAt`, not an orthogonal combinator like the others.
- `optionalValueAt` return a pointer, not `std::optional`. This also
expresses optionality, but without creating a needless copy. This
brings it in line with the other combinators which also return
references.
- Delete `valueAt` and `optionalValueAt` taking the map by value, as we
did for `get` in 408c09a120, which
prevents bugs / unnecessary copies.
`adl_serializer<DerivationOptions::OutputChecks>::from_json` was the one
use of `getNullable`. I give it a little static function for the
ultimate creation of a `std::optional` it does need to do (after
switching it to using `getNullable . valueAt`. That could go in
`json-utils.hh` eventually, but I didn't bother for now since only one
things needs it.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
addToStore(): Don't parse the NAR
* StringSource: Implement skip()
This is slightly faster than doing a read() into a buffer just to
discard the data.
* LocalStore::addToStore(): Skip unnecessary NARs rather than parsing them
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Since 2.32, nix now needs boost 1.87 or later to build,
due to using unordered::concurrent_flat_map try_emplace_and_cvisit
../src/libexpr/eval.cc: In member function ‘void nix::EvalState::evalFile(const nix::SourcePath&, nix::Value&, bool)’:
../src/libexpr/eval.cc:1096:20: error: ‘class boost::unordered::concurrent_flat_map<nix::SourcePath, nix::Value*, std::hash<nix::SourcePath>, std::equal_to<nix::SourcePath>, traceable_allocator<std::pair<const nix::SourcePath, nix::Value*> > >’ has no member named ‘try_emplace_and_cvisit’; did you mean ‘try_emplace_or_cvisit’?
1096 | fileEvalCache->try_emplace_and_cvisit(
| ^~~~~~~~~~~~~~~~~~~~~~
| try_emplace_or_cvisit
See 834580b539
