Introduce protected `upload` method overloads in `HttpBinaryCacheStore`
that handle the actual upload after compression has been applied. This
separates compression concerns (in `upsertFile`) from upload mechanics
(in `upload`).
Two overloads are provided:
1. `upload(path, RestartableSource &, sizeHint, mimeType, contentEncoding)`
2. `upload(path, CompressedSource &, mimeType)`
Introduce a `CompressedSource` class in libutil's `serialise.hh` that
compresses a `RestartableSource` and owns the compressed data. This is a
general-purpose utility that can be used anywhere compressed data needs
to be treated as a source.
We were getting this flex lexer warning during build:
```
../src/libexpr/lexer.l:333: warning, -s option given but default rule can be matched
```
The lexer uses `%option nodefault` but the `PATH_START` state only had
rules for specific patterns (`PATH_SEG` and `HPATH_START`) without a
catch-all rule to handle unexpected input.
Added a catch-all rule with `unreachable()`. This code path should never
be reached in normal operation since `PATH_START` is only entered after
matching `PATH_SEG` or `HPATH_START`, and we immediately rewind to
re-parse those same patterns. The catch-all exists solely to satisfy
flex's `%option nodefault` requirement.
Make uploads run in constant memory. Also change the callbacks to be
noexcept, since we really don't want to be unwinding the stack in the
curl thread. That will definitely corrupt that stack and make nix/curl
crash in very bad ways.
Fix a race condition where interrupting a download (via Ctrl-C) during a
retry attempt could cause a crash. When `enqueueItem()` throws because the
download thread is shutting down, the exception would propagate without
setting `done=true`, causing the `TransferItem` destructor to invoke the
callback a second time.
This triggered an assertion failure in `Callback::rethrow()` with:
`Assertion '!prev' failed` and the error message `cannot enqueue download
request because the download thread is shutting down`.
The fix catches the exception from `enqueueItem()` and calls `fail()` to
properly complete the transfer, ensuring the callback is invoked exactly
once.
Some zsh setups (including mine) do not load the
completion if `#compdef` is not on the first line.
So we move the `# shellcheck` comment to the
second line to avoid this issue.
This continues the work for formalizing our current JSON docs. Note that
in the process, a few bugs were caught:
- `closureSize` was repeated twice, forgot `closureDownloadSize`
- `file*` fields should be `download*`. They are in fact called that in
the line-oriented `.narinfo` file, but were renamed in the JSON
format.
We immediately use this in the JSON schemas for Derivation and Deriving
Path, but we cannot yet use it in Store Object Info because those paths
*do* include the store dir currently.
- Uses the more explicit `@ingroup` most of the time, to avoid problems
with nested groups, and to make group membership more explicit.
The division into headers is not great for documentation purposes,
so this helps.
- More attention for memory management details
- Various other improvements to doc comments
Per #7591, the `nix-store --gc --print-dead` command does not provide
any feedback about the amount of disk space that is used by dead store
paths. It looks like this has been the case since 7ab68961e (* Garbage
collector: added an option `--use-atime' to delete paths in...,
2008-09-17).
Update the nix-store documentation to remove the claim that this is
function that `nix-store --gc --print-dead` performs.
