Introduce a `CompressedSource` class in libutil's `serialise.hh` that
compresses a `RestartableSource` and owns the compressed data. This is a
general-purpose utility that can be used anywhere compressed data needs
to be treated as a source.
Make uploads run in constant memory. Also change the callbacks to be
noexcept, since we really don't want to be unwinding the stack in the
curl thread. That will definitely corrupt that stack and make nix/curl
crash in very bad ways.
Implement `uploadPart()` for uploading individual parts in S3 multipart
uploads:
- Constructs URL with `?partNumber=N&uploadId=ID` query parameters
- Uploads chunk data with `application/octet-stream` mime type
- Extracts and returns `ETag` from response
This is a good default (the methods that allow for an arbitrary choice
of source accessor are generally preferable both to implement and to
use). And it also pays its way by allowing us to delete *both* the
`DummyStore` and `LocalStore` implementations.
Introduces `scanForReferencesDeep` to provide per-file granularity when
scanning for store path references, enabling better diagnostics for
cycle detection and `nix why-depends --precise`.
Implement `abortMultipartUpload()` for cleaning up incomplete multipart
uploads on error:
- Constructs URL with `?uploadId=ID` query parameter
- Issues `DELETE` request to abort the multipart upload
The `showBytes()` function was redundant with `renderSize()` as the
latter automatically selects the appropriate unit (KiB, MiB, GiB, etc.)
based on the value, whereas `showBytes()` always formatted as MiB
regardless of size.
Co-authored-by: Bernardo Meurer Costa <beme@anthropic.com>
In particular
- Remove `get`, it is redundant with `valueAt` and the `get` in
`util.hh`.
- Remove `nullableValueAt`. It is morally just the function composition
`getNullable . valueAt`, not an orthogonal combinator like the others.
- `optionalValueAt` return a pointer, not `std::optional`. This also
expresses optionality, but without creating a needless copy. This
brings it in line with the other combinators which also return
references.
- Delete `valueAt` and `optionalValueAt` taking the map by value, as we
did for `get` in 408c09a120, which
prevents bugs / unnecessary copies.
`adl_serializer<DerivationOptions::OutputChecks>::from_json` was the one
use of `getNullable`. I give it a little static function for the
ultimate creation of a `std::optional` it does need to do (after
switching it to using `getNullable . valueAt`. That could go in
`json-utils.hh` eventually, but I didn't bother for now since only one
things needs it.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
S3 buckets support object versioning to prevent unexpected changes,
but Nix previously lacked the ability to fetch specific versions of
S3 objects. This adds support for a `versionId` query parameter in S3
URLs, enabling users to pin to specific object versions:
```
s3://bucket/key?region=us-east-1&versionId=abc123
```
Move HttpBinaryCacheStore class from .cc file to header to enable
inheritance by S3BinaryCacheStore. Create S3BinaryCacheStore class that
overrides upsertFile() to implement multipart upload logic.
Add a sizeHint parameter to BinaryCacheStore::upsertFile() to enable
size-based upload decisions in implementations. This lays the groundwork
for reintroducing S3 multipart upload support.
Add support for HTTP DELETE requests to FileTransfer infrastructure:
This enables S3 multipart upload abort functionality via DELETE requests
to S3 endpoints.
