The single-string syntax '>=8.16.0 <8.17.0' only applied the lower
bound, causing curl 8.17.0 to be incorrectly rejected. Split into two
separate version_compare() calls for compatibility with Meson 1.1,
since multi-argument syntax requires Meson 1.8+.
Since the root cause (the lack of backpressure control) has
been fixed in the previous commit we can revert the change from
8ffea0a018 and make the default size much
smaller.
Instead of naively stalling the download thread we can instead stop the transfer.
This allows the other multiplexed connections to continue downloading (and unpacking),
if the result of the download gets piped into a GitFileSystemObjectSink.
Prior art in lix project:
- 4ae6fb5a8f
- 12156d3beb
This patch is very different from the lix one, since we are using a decompression sink
in the middle of the pipeline but the co-authored-by is there since I was motivated to
implement this by looking at the lix side of things.
Co-authored-by: eldritch horrors <pennae@lix.systems>
`listNar` did the not-so-pretty thing of going straight to JSON. Now it
uses `MemorySourceAccessor::File`, or rather variations of it, to go to
a C++ data type first, and only JSON second.
To accomplish this we add some type parameters to the `File` data type.
Actually, we need to do two rounds of this, because shallow NAR
listings. There is `FileT` and `DirectoryT` accordingly.
`nix derivation add`, and its C API counterpart, now works a bit closer
to `builtins.derivation` in that they don't require the user to fill-in
input addressed paths correctly ahead of time.
The logic for this is carefully deduplicated, between all 3 entry
points, and also between the existing `checkInvariants` function. There
are some more functional tests, and there are also many more unit tests.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Co-authored-by: edef <edef@edef.eu>
We don't use the various set<string_view>s that we construct,
and all we really care about is ensuring that all outputs are
of a single, consistent type.
This is an example of "Parse, don't validate" principle [1].
Before, we had a number of `StringSet`s in `DerivationOptions` that
were not *actually* allowed to be arbitrary sets of strings. Instead,
each set member had to be one of:
- a store path
- a CA "downstream placeholder"
- an output name
Only later, in the code that checks outputs, would these strings be
further parsed to match these cases. (Actually, only 2 by that point,
because the placeholders must be rewritten away by then.)
Now, we fully parse everything up front, and have an "honest" data type
that reflects these invariants:
- store paths are parsed, stored as (opaque) deriving paths
- CA "downstream placeholders" are rewritten to the output deriving
paths they denote
- output names are the only arbitrary strings left
Since the first two cases both become deriving paths, that leaves us
with a `std::variant<SingleDerivedPath, String>` data type, which we use
in our sets instead.
Getting rid of placeholders is especially nice because we are replacing
them with something much more internally-structured / transparent.
[1]: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Now the error message looks something like:
error:
… during upload of 'file:///tmp/storeabc/4yxrw9flcvca7f3fs7c5igl2ica39zaw.narinfo'
error: blah blah
Also makes fail and failEx themselves noexcept, since all the operations they
do are noexcept and we don't want exceptions escaping from them.
The indentation level of the code is already high enough. We can just
wrap the whole function in a try/catch and mark it noexcept.
Partially cherry-picked from https://gerrit.lix.systems/c/lix/+/2133
Co-authored-by: eldritch horrors <pennae@lix.systems>
Several bugs to squash:
- Apparently DELETE is an already used macro with Win32. We can avoid it
by using Camel case instead (slightly hacky but also fits the naming
convention better)
- Gets rid of the raw usage of isatty. Added an isTTY impl to abstract over
the raw API.
Replaces the usage of createAtRoot, which goes as far up the
directory tree as possible with rooted variant makeFSSourceAccessor.
The changes in this patch should be safe wrt to not asserting on relative
paths. Arguments passed to makeFSSourceAccessor here should already be using
absolute paths.
This makes the proto serializer characterisation test data be
accompanied by JSON data.
This is arguably useful for a reasons:
- The JSON data is human-readable while the binary data is not, so it
provides some indication of what the test data means beyond the C++
literals.
- The JSON data is language-agnostic, and so can be used to quickly rig
up tests for implementation in other languages, without having source
code literals at all (just go back and forth between the JSON and the
binary).
- Even though we have no concrete plans to place the binary protocol 1-1
or with JSON, it is still nice to ensure that the JSON serializers and
binary protocols have (near) equal coverage over data types, to help
ensure we didn't forget a JSON (de)serializer.
Make instances for them that share code with `nix path-info`, but do a
slightly different format without store paths containing store dirs
(matching the other latest JSON formats).
Progress on #13570.
If we depend on the store dir, our JSON serializers/deserializers take
extra arguements, and that interfaces with the likes of various
frameworks for associating these with types (e.g. nlohmann in C++, Serde
in Rust, and Aeson in Haskell).
For now, `nix path-info` still uses the previous format, with store
dirs. We may yet decide to "rip of the band-aid", and just switch it
over, but that is left as a future PR.
The variant has on the left-hand side the topologically sorted vector
and the right-hand side is a pair showing the path and its parent that
represent a cycle in the graph making the sort impossible.
This change prepares for enhanced cycle error messages that can provide
more context about the cycle. The variant approach allows callers to
handle cycles more flexibly, enabling better error reporting that shows
the full cycle path and which files are involved.
Adapted from Lix commit f7871fcb5.
Change-Id: I70a987f470437df8beb3b1cc203ff88701d0aa1b
Co-Authored-By: Maximilian Bosch <maximilian@mbosch.me>
Add support for configuring S3 storage class via the storage-class
parameter for S3BinaryCacheStore. This allows users to optimize costs
by selecting appropriate storage tiers (STANDARD, GLACIER,
INTELLIGENT_TIERING, etc.) based on access patterns.
The storage class is applied via the x-amz-storage-class header for
both regular PUT uploads and multipart upload initiation.
Since we haven't released v2 yet (2.32 has v1) we can just update this
in-place and avoid version churn.
Note that as a nice side effect of using the standard `Hash` JSON impl,
we don't neeed this `hashFormat` parameter anymore.
It turns out this code path is only used for unit tests (to ensure our
JSON formats are possible to parse by other code, elsewhere). No
user-facing functionality consumes this format.
Therefore, let's drop the old version parsing support.
The old string format is a holdover from the pre JSON days. It is not
friendly to users who need to get the information out of it.
Also introduce the sort of versioning we have for derivation for this
format too.
- Use canonical content address JSON format for floating content
addressed derivation outputs
This keeps it more consistent.
- Reorganize inputs into nested structure (`inputs.srcs` and
`inputs.drvs`)
This will allow for an easier to use, but less compact, alternative
where `srcs` is just a list of derived paths.
It also allows for other experiments for derivations with a different
input structure, as I suspect will be needed for secure build traces.
Have one to that instead of one to `Derivation`. `DerivationBuilder`
doesn't need `inputDrvs`, so `BasicDerivation` suffices.
(In fact, it doesn't need `inputSrcs` either, but we don't yet hve a
type to exclude that.)
