Since the root cause (the lack of backpressure control) has
been fixed in the previous commit we can revert the change from
8ffea0a018 and make the default size much
smaller.
Instead of naively stalling the download thread we can instead stop the transfer.
This allows the other multiplexed connections to continue downloading (and unpacking),
if the result of the download gets piped into a GitFileSystemObjectSink.
Prior art in lix project:
- 4ae6fb5a8f
- 12156d3beb
This patch is very different from the lix one, since we are using a decompression sink
in the middle of the pipeline but the co-authored-by is there since I was motivated to
implement this by looking at the lix side of things.
Co-authored-by: eldritch horrors <pennae@lix.systems>
The use of sourceToSink is an unnecessary serialization bottleneck.
While we are at it, fix the copyRecursive implementation to actually copy
the whole directory. It wasn't used for anything prior, but now it has a use
and accompanying tests for flake clone.
Also do a better JSON and testing for deep and shallow NAR listings.
As documented, this for file system objects themselves, since
`MemorySourceAccessor` is an implementation detail.
`listNar` did the not-so-pretty thing of going straight to JSON. Now it
uses `MemorySourceAccessor::File`, or rather variations of it, to go to
a C++ data type first, and only JSON second.
To accomplish this we add some type parameters to the `File` data type.
Actually, we need to do two rounds of this, because shallow NAR
listings. There is `FileT` and `DirectoryT` accordingly.
This matches the "NAR Listing" JSON format, and also helps distinguish
from regular file contents.
Why we want to match that will become clear in the next comments, when
we will in fact use (variations of) this data type for NAR listings.
We don't use the various set<string_view>s that we construct,
and all we really care about is ensuring that all outputs are
of a single, consistent type.
This is necessary to ban symlink following. It can be considered
a defense in depth against issues similar to CVE-2024-45593. By
slightly changing the API in a follow-up commit we will be able
to mitigate the symlink following issue for good.
This is an example of "Parse, don't validate" principle [1].
Before, we had a number of `StringSet`s in `DerivationOptions` that
were not *actually* allowed to be arbitrary sets of strings. Instead,
each set member had to be one of:
- a store path
- a CA "downstream placeholder"
- an output name
Only later, in the code that checks outputs, would these strings be
further parsed to match these cases. (Actually, only 2 by that point,
because the placeholders must be rewritten away by then.)
Now, we fully parse everything up front, and have an "honest" data type
that reflects these invariants:
- store paths are parsed, stored as (opaque) deriving paths
- CA "downstream placeholders" are rewritten to the output deriving
paths they denote
- output names are the only arbitrary strings left
Since the first two cases both become deriving paths, that leaves us
with a `std::variant<SingleDerivedPath, String>` data type, which we use
in our sets instead.
Getting rid of placeholders is especially nice because we are replacing
them with something much more internally-structured / transparent.
[1]: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Now the error message looks something like:
error:
… during upload of 'file:///tmp/storeabc/4yxrw9flcvca7f3fs7c5igl2ica39zaw.narinfo'
error: blah blah
Also makes fail and failEx themselves noexcept, since all the operations they
do are noexcept and we don't want exceptions escaping from them.
The indentation level of the code is already high enough. We can just
wrap the whole function in a try/catch and mark it noexcept.
Partially cherry-picked from https://gerrit.lix.systems/c/lix/+/2133
Co-authored-by: eldritch horrors <pennae@lix.systems>
Makes the cross-x86_64-w64-mingw32 devshell slightly less
broken. It still needs a bit of massaging to function, but
that's much less cumbersome now that the generic machinery
with genericClosure that evaluates drvPath doesn't barf on
unavailable packages.
