I started getting these warnings `warning: download buffer is full; consider increasing the 'download-buffer-size' setting` but the documentation does not make it obvious what unit of measurement it accepts.
This allows RemoteStore::addMultipleToStore() to free the Source
objects early (and in particular the associated sinkToSource()
buffers). This should fix#7359. For example, memory consumption of
nix copy --derivation --to ssh-ng://localhost?remote-store=/tmp/nix --derivation --no-check-sigs \
/nix/store/4p9xmfgnvclqpii8pxqcwcvl9bxqy2xf-nixos-system-...drv
went from 353 MB to 74 MB.
During garbage collection we cache several things -- a set of known-dead
paths, a set of known-alive paths, and a map of paths to their derivers.
Currently they use STL maps and sets, which are ordered structures that
typically are backed by binary trees. Since we are putting pseudorandom
paths into these and looking them up by exact key, we don't need the
ordering, and we're paying a nontrivial cost per insertion.
The existing maps require O(n log n) memory and have O(log n) insertion
and lookup time.
We could instead use unordered maps, which are typically backed by
hashmaps. These require O(n) memory and have O(1) insertion and lookup
time.
On my system this appears to result in a dramatic speedup -- prior to
this patch I was able to delete 400k paths out of 9.5 million over the
course of 34.5 hours. After this patch the same result took 89 minutes.
This result should NOT be taken at face value because the two runs
aren't really comparable; in particular the first started when I had 9.5
million store paths and the seconcd started with 7.8 million, so we are
deleting a different set of paths starting from a much cleaner
filesystem. But I do think it's indicative.
Related: https://github.com/NixOS/nix/issues/9581
This fixes segfaults with nix copy when there was an error processing
addMultipleToStore.
Running with ASAN/TSAN pointed at an use-after-free with threads from
the pool accessing the graph declared in processGraph after the function
was exiting and destructing the variables.
It turns out that if there is an error before pool.process() is called,
for example while we are still enqueuing tasks, then pool.process()
isn't called and threads are still left to run.
By creating the pool last we ensure that it is stopped first before
running other destructors even if an exception happens early.
[ lix porting note: nix does not name threads so the patch has been
adapted to not pass thread name ]
Link: https://git.lix.systems/lix-project/lix/issues/618
Link: https://gerrit.lix.systems/c/lix/+/2355
Fix a footgun. In my case, I had a couple of build ("output")
directories sitting around.
rm -rf build-*
Was confused for a bit why a meson.build file was missing.
Probably also helps with autocompletion.
I tried meson-build-support first, but I had to add something like
a nix- prefix, in order to make meson happy. They've reserved the
meson- prefix.
Instead of the unhelpful
warning: 'https://cache.flakehub.com' does not appear to be a binary cache
you now get
warning: unable to download 'https://cache.flakehub.com/nix-cache-info': HTTP error 401
response body:
{"code":401,"error":"Unauthorized","message":"Unauthorized."}
This gets rid of unnecessary copies in range-based-for loops and
local variables, when they are used solely as `const &`.
Also added a fixme comment about a suspicious move out of const,
which might not be intended.
Looks like some cruft has been left over from previous refactorings.
This removes dead variables, which should not have side effects in their
constructors. In cases where the variable initialization has a purpose
[[maybe_unused]] is inserted to silence compiler warnings.
This reduces the amount of boilerplate. More importantly, it provides
a place to add compiler flags (such as -O3) without having to add it
to every subproject (and the risk of forgetting to include it).
First the motivation: I recently faced a bug that I assume is coming
from the topoSortPaths function where the GC was trying to delete a
path having some alive referrers. I resolved this by manually deleting
the faulty path referrers using nix-store --query --referrers. I sadly
did not manage to reproduce this bug.
This bug alone is not a big deal. However, this bug is
triggering a cascading failure: invalidatePathChecked is throwing a
PathInUse exception. This exception is not catched and fails the whole GC
run. From there, the machine (a builder machine) was unable to GC its
Nix store, which led to an almost full disk with no way to
automatically delete the dead Nix paths.
Instead, I think we should log the error for the specific store path
we're trying to delete, specifying we can't delete this path because
it still has referrers. Once we're done with logging that, the GC run
should continue to delete the dead store paths it can delete.
This is the first part of rewriteDerivation() factored out into its
own method. It's not used anywhere else at the moment, but it's useful
on lazy-trees for rewriting virtual paths.
In these trivial cases the final vector size (or lower bound on the size) is known,
so we can avoid some vector reallocations. This is not very important, but is just
good practice and general hygiene.
This is good practice to avoid pessimisations.
Left comments for the reasoning why ctors should be noexcept.
There are some tricky cases where we intentionally want throwing move ctors/assignments.
But those cases should really be reviewed, since some of those can be replaced
with more idiomatic copy/move-and-swap.
