Similar to the deepSeq fix, toJSON on deeply nested structures caused
an uncontrolled OS-level stack overflow.
Fix by adding call depth tracking to printValueAsJSON.
When deepSeq encounters an error while evaluating a list element, the
error trace now includes the list index, making it easier to locate
the problematic element.
builtins.deepSeq on deeply nested structures (e.g., a linked list with
100,000 elements) caused an uncontrolled OS-level stack overflow with
no Nix stack trace.
Fix by adding call depth tracking to forceValueDeep, integrating with
Nix's existing max-call-depth mechanism. Now produces a controlled
"stack overflow; max-call-depth exceeded" error with a proper stack
trace.
Closes: https://github.com/NixOS/nix/issues/7816
this is a painful change. we should really add EvalState or EvalMemory
as an argument to various functions as we need it, but because we want
to preserve the stablity API, we hack it in as a field of nix_value.
The use of sourceToSink is an unnecessary serialization bottleneck.
While we are at it, fix the copyRecursive implementation to actually copy
the whole directory. It wasn't used for anything prior, but now it has a use
and accompanying tests for flake clone.
Also do a better JSON and testing for deep and shallow NAR listings.
As documented, this for file system objects themselves, since
`MemorySourceAccessor` is an implementation detail.
`listNar` did the not-so-pretty thing of going straight to JSON. Now it
uses `MemorySourceAccessor::File`, or rather variations of it, to go to
a C++ data type first, and only JSON second.
To accomplish this we add some type parameters to the `File` data type.
Actually, we need to do two rounds of this, because shallow NAR
listings. There is `FileT` and `DirectoryT` accordingly.
This matches the "NAR Listing" JSON format, and also helps distinguish
from regular file contents.
Why we want to match that will become clear in the next comments, when
we will in fact use (variations of) this data type for NAR listings.
`nix derivation add`, and its C API counterpart, now works a bit closer
to `builtins.derivation` in that they don't require the user to fill-in
input addressed paths correctly ahead of time.
The logic for this is carefully deduplicated, between all 3 entry
points, and also between the existing `checkInvariants` function. There
are some more functional tests, and there are also many more unit tests.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Co-authored-by: edef <edef@edef.eu>
These do a read/write test in the middles of some computation. They are
an imperative way to test intermediate values rather than functionally
testing end outputs.
We don't use the various set<string_view>s that we construct,
and all we really care about is ensuring that all outputs are
of a single, consistent type.
This is necessary to ban symlink following. It can be considered
a defense in depth against issues similar to CVE-2024-45593. By
slightly changing the API in a follow-up commit we will be able
to mitigate the symlink following issue for good.
This is an example of "Parse, don't validate" principle [1].
Before, we had a number of `StringSet`s in `DerivationOptions` that
were not *actually* allowed to be arbitrary sets of strings. Instead,
each set member had to be one of:
- a store path
- a CA "downstream placeholder"
- an output name
Only later, in the code that checks outputs, would these strings be
further parsed to match these cases. (Actually, only 2 by that point,
because the placeholders must be rewritten away by then.)
Now, we fully parse everything up front, and have an "honest" data type
that reflects these invariants:
- store paths are parsed, stored as (opaque) deriving paths
- CA "downstream placeholders" are rewritten to the output deriving
paths they denote
- output names are the only arbitrary strings left
Since the first two cases both become deriving paths, that leaves us
with a `std::variant<SingleDerivedPath, String>` data type, which we use
in our sets instead.
Getting rid of placeholders is especially nice because we are replacing
them with something much more internally-structured / transparent.
[1]: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Now the error message looks something like:
error:
… during upload of 'file:///tmp/storeabc/4yxrw9flcvca7f3fs7c5igl2ica39zaw.narinfo'
error: blah blah
Also makes fail and failEx themselves noexcept, since all the operations they
do are noexcept and we don't want exceptions escaping from them.
The indentation level of the code is already high enough. We can just
wrap the whole function in a try/catch and mark it noexcept.
Partially cherry-picked from https://gerrit.lix.systems/c/lix/+/2133
Co-authored-by: eldritch horrors <pennae@lix.systems>
Makes the cross-x86_64-w64-mingw32 devshell slightly less
broken. It still needs a bit of massaging to function, but
that's much less cumbersome now that the generic machinery
with genericClosure that evaluates drvPath doesn't barf on
unavailable packages.
