Add support for configuring S3 storage class via the storage-class
parameter for S3BinaryCacheStore. This allows users to optimize costs
by selecting appropriate storage tiers (STANDARD, GLACIER,
INTELLIGENT_TIERING, etc.) based on access patterns.
The storage class is applied via the x-amz-storage-class header for
both regular PUT uploads and multipart upload initiation.
The old string format is a holdover from the pre JSON days. It is not
friendly to users who need to get the information out of it.
Also introduce the sort of versioning we have for derivation for this
format too.
- Use canonical content address JSON format for floating content
addressed derivation outputs
This keeps it more consistent.
- Reorganize inputs into nested structure (`inputs.srcs` and
`inputs.drvs`)
This will allow for an easier to use, but less compact, alternative
where `srcs` is just a list of derived paths.
It also allows for other experiments for derivations with a different
input structure, as I suspect will be needed for secure build traces.
Have one to that instead of one to `Derivation`. `DerivationBuilder`
doesn't need `inputDrvs`, so `BasicDerivation` suffices.
(In fact, it doesn't need `inputSrcs` either, but we don't yet hve a
type to exclude that.)
Fix#14480
This method is not well-defined for arbitrary stores, which do not have
a notion of a "real path" -- it is only well-defined for local file
systems stores, which do have exactly that notion, and so it is moved to
that sub-interface instead.
Some call-sites had to be fixed up for this, but in all cases the
changes are positive. Using `getFSSourceAccessor` allows for more other
stores to work properly. `nix-channel` was straight-up wrong in the case
of redirected local stores. And the building logic with remote building
and a non-local store is also fixed, properly gating some deletions on
store type.
Co-authored-by: Robert Hensing <robert@roberthensing.nl>
Previously, only shared memory segments were cleaned up.
This could lead to leaked message queues and semaphore sets when builds use System V IPC, exhausting kernel IPC limits over time.
This commit extends the cleanup to all three System V IPC types:
1. Shared memory segments
2. Message queues
3. Semaphores
Additionally, we stop removing IPC objects during iteration, as it could corrupt the kernel's iterator state and cause some objects to be skipped. The new implementation uses a two-pass approach where we list first and then remove them in a separate pass.
The IPC IDs are now extracted during iteration using actual system calls (shmget, msgget, semget) rather than being looked up later, ensuring the objects exist when we capture their IDs.
In Linux, IPC objects are automatically cleaned up when the IPC namespace is destroyed.
On Darwin, since there are no IPC namespaces, the IPC objects may sometimes persist after the build user's processes are killed.
This patch modifies the cleanup logic to use sysctl calls to identify and remove left over shm segments associated with the build user.
Fixes: #12548
Progress on #13405, which asks for an explicit characterisation of the
equivalence relation like the one given here.
Also progress on #11895, because we're using the term "build trace
entry" instead of "realisation".
Mention #9259, a future work item.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Add three configuration settings to `S3BinaryCacheStoreConfig` to control
multipart upload behavior:
- `bool multipart-upload` (default `false`): Enable/disable multipart uploads
- `uint64_t multipart-chunk-size` (default 5 MiB): Size of each upload part
- `uint64_t multipart-threshold` (default 100 MiB): Minimum file size for multipart
The feature is disabled by default.
Stop delegating to `HttpBinaryCacheStore::upsertFile` and instead
handle compression in the S3 store's `upsertFile` override, then call
our own `upload()` method. This separation is necessary for future
multipart upload support.
Introduce protected `upload` method overloads in `HttpBinaryCacheStore`
that handle the actual upload after compression has been applied. This
separates compression concerns (in `upsertFile`) from upload mechanics
(in `upload`).
Two overloads are provided:
1. `upload(path, RestartableSource &, sizeHint, mimeType, contentEncoding)`
2. `upload(path, CompressedSource &, mimeType)`
Introduce a `CompressedSource` class in libutil's `serialise.hh` that
compresses a `RestartableSource` and owns the compressed data. This is a
general-purpose utility that can be used anywhere compressed data needs
to be treated as a source.
Make uploads run in constant memory. Also change the callbacks to be
noexcept, since we really don't want to be unwinding the stack in the
curl thread. That will definitely corrupt that stack and make nix/curl
crash in very bad ways.
Fix a race condition where interrupting a download (via Ctrl-C) during a
retry attempt could cause a crash. When `enqueueItem()` throws because the
download thread is shutting down, the exception would propagate without
setting `done=true`, causing the `TransferItem` destructor to invoke the
callback a second time.
This triggered an assertion failure in `Callback::rethrow()` with:
`Assertion '!prev' failed` and the error message `cannot enqueue download
request because the download thread is shutting down`.
The fix catches the exception from `enqueueItem()` and calls `fail()` to
properly complete the transfer, ensuring the callback is invoked exactly
once.
