fetchToStore() caching was broken because it uses the fingerprint of
the accessor, but now that the accessor (typically storeFS) is a
composite (like MountedSourceAccessor or AllowListSourceAccessor),
there was no fingerprint anymore. So fetchToStore now uses the new
getFingerprint() method to get the specific fingerprint for the
subpath.
Previously, Nix would not create a cache entry for substituted/cached
inputs
This led to severe slowdowns in some scenarios where a large input (like
Nixpkgs) had already been unpacked to the store but didn't exist in a
users cache, as described in https://github.com/NixOS/nix/issues/11228
Using the same method as https://github.com/NixOS/nix/pull/12911, we can
create a cache entry for the fingerprint of substituted/cached inputs
and avoid this problem entirely
This is sometimes easier / more performant to implement, and
independently it is also a more convenient interface for many callers.
The existing store-wide `getFSAccessor` is only used for
- `nix why-depends`
- the evaluator
I hope we can get rid of it for those, too, and then we have the option
of getting rid of the store-wide method.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
* It is tough to contribute to a project that doesn't use a formatter,
* It is extra hard to contribute to a project which has configured the formatter, but ignores it for some files
* Code formatting makes it harder to hide obscure / weird bugs by accident or on purpose,
Let's rip the bandaid off?
Note that PRs currently in flight should be able to be merged relatively easily by applying `clang-format` to their tip prior to merge.
For example, instead of doing
#include "nix/store-config.hh"
#include "nix/derived-path.hh"
Now do
#include "nix/store/config.hh"
#include "nix/store/derived-path.hh"
This was originally planned in the issue, and also recent requested by
Eelco.
Most of the change is purely mechanical. There is just one small
additional issue. See how, in the example above, we took this
opportunity to also turn `<comp>-config.hh` into `<comp>/config.hh`.
Well, there was already a `nix/util/config.{cc,hh}`. Even though there
is not a public configuration header for libutil (which also would be
called `nix/util/config.{cc,hh}`) that's still confusing, To avoid any
such confusion, we renamed that to `nix/util/configuration.{cc,hh}`.
Finally, note that the libflake headers already did this, so we didn't
need to do anything to them. We wouldn't want to mistakenly get
`nix/flake/flake/flake.hh`!
Progress on #7876
The short answer for why we need to do this is so we can consistently do
`#include "nix/..."`. Without this change, there are ways to still make
that work, but they are hacky, and they have downsides such as making it
harder to make sure headers from the wrong Nix library (e..g.
`libnixexpr` headers in `libnixutil`) aren't being used.
The C API alraedy used `nix_api_*`, so its headers are *not* put in
subdirectories accordingly.
Progress on #7876
We resisted doing this for a while because it would be annoying to not
have the header source file pairs close by / easy to change file
path/name from one to the other. But I am ameliorating that with
symlinks in the next commit.
This allows writing lock files with dirty inputs, so long as they have
a NAR hash. (Currently they always have a NAR hash, but with lazy
trees that may not always be the case.)
Generally dirty locks are bad for reproducibility (we can detect if
the dirty input has changed, but we have no way to fetch it except
substitution). Hence we don't allow them by default.
Fixes#11181.
We now just check that the fetcher doesn't change any attributes in
the input, and return all the original attributes (i.e. discarding any
new attributes and keeping any attributes that the fetcher didn't
keep).
This fixes the error
'{"__final":true,"lastModified":1686592866,"narHash":"sha256-riGg89eWhXJcPNrQGcSwTEEm7CGxWC06oSX44hajeMw","owner":"nixos","repo":"nixpkgs","rev":"0eeebd64de89e4163f4d3cf34ffe925a5cf67a05","type":"github"}' resulted in different input
'{"__final":true,"lastModified":1686592866,"narHash":"sha256-riGg89eWhXJcPNrQGcSwTEEm7CGxWC06oSX44hajeMw=","owner":"nixos","repo":"nixpkgs","rev":"0eeebd64de89e4163f4d3cf34ffe925a5cf67a05","type":"github"}'
in flake-regressions/tests/nix-community/patsh/0.2.1 (note the lack of
a trailing '=' in the NAR hash in the lock file).
We haven't added the narHash attribute yet at that point. And if the
caller uses getAccesor() instead of fetchToStore() (e.g. in `nix
registry pin`), the narHash attribute will never be added. This could
lead to a mismatch.
The ability to substitute inputs was removed in #10612 because it was
broken: with user-specified inputs containing a `narHash` attribute,
substitution resulted in an input that lacked the attributes returned
by the real fetcher (such as `lastModified`).
To fix this, we introduce a new input attribute `final`. If `final =
true`, fetching the input cannot add or change any attributes.
We only attempt to substitute inputs that have `final = true`. This is
implied by lock file entries; we only write a lock file if all its
entries are "final".
The user can specified `final = true` in `fetchTree`, in which case it
is their responsibility to ensure that all attributes returned by the
fetcher are included in the `fetchTree` call. For example,
nix eval --impure --expr 'builtins.fetchTree { type = "github"; owner = "NixOS"; repo = "patchelf"; final = true; narHash = "sha256-FSoxTcRZMGHNJh8dNtKOkcUtjhmhU6yQXcZZfUPLhQM="; }'
succeeds in a store path with the specified NAR hash exists or is
substitutable, but fails with
error: fetching final input '{"final":true,"narHash":"sha256-FSoxTcRZMGHNJh8dNtKOkcUtjhmhU6yQXcZZfUPLhQM=","owner":"NixOS","repo":"patchelf","type":"github"}' resulted in different input '{"final":true,"lastModified":1718457448,"narHash":"sha256-FSoxTcRZMGHNJh8dNtKOkcUtjhmhU6yQXcZZfUPLhQM=","owner":"NixOS","repo":"patchelf","rev":"a0f54334df36770b335c051e540ba40afcbf8378","type":"github"}'
Known behavior changes:
- `MemorySourceAccessor`'s comparison operators no longer forget to
compare the `SourceAccessor` base class.
Progress on #10832
What remains for that issue is hopefully much easier!
Progress on #5638
There are still a global fetcher and eval settings, but they are pushed
down into `libnixcmd`, which is a lot less bad a place for this sort of
thing.
Continuing process pioneered in
52bfccf8d8.
Subflakes are flakes in the same tree, accessed in flake inputs via
relative paths (e.g. `inputs.foo.url = "path:./subdir"`). Previously
these didn't work very well because they would be separately copied to
the store, which is inefficient and makes references to parent
directories tricky or impossible. Furthermore, they had their own NAR
hash in the lock file, which is superfluous since the parent is
already locked.
Now subflakes are accessed via the accessor of the calling flake. This
avoids the unnecessary copy and makes it possible for subflakes to
depend on flakes in a parent directory (so long as they're in the same
tree).
Lock file nodes for relative flake inputs now have a new `parent` field:
{
"locked": {
"path": "./subdir",
"type": "path"
},
"original": {
"path": "./subdir",
"type": "path"
},
"parent": [
"foo",
"bar"
]
}
which denotes that `./subdir` is to be interpreted relative to the
directory of the `bar` input of the `foo` input of the root flake.
Extracted from the lazy-trees branch.
After the removal of the InputAccessor::fetchToStore() method, the
only remaining functionality in InputAccessor was `fingerprint` and
`getLastModified()`, and there is no reason to keep those in a
separate class.
Having a narHash doesn't mean that we have the other attributes
returned by the fetcher (such as lastModified or rev). For instance,
$ nix flake metadata github:NixOS/patchelf/7c2f768bf9601268a4e71c2ebe91e2011918a70f
Last modified: 2024-01-15 10:51:22
but
$ nix flake metadata github:NixOS/patchelf/7c2f768bf9601268a4e71c2ebe91e2011918a70f?narHash=sha256-PPXqKY2hJng4DBVE0I4xshv/vGLUskL7jl53roB8UdU%3D
(does not print a "Last modified")
The latter only happens if the store path already exists or is
substitutable, which made this impure behaviour unpredictable.
Fixes#10601.
* Convert all InputScheme::fetch() methods to getAccessor().
* Add checkLocks() method for checking lock attributes.
* Rename fetch() to fetchToStore().
It's better to just check whether the input has all the attributes
needed to consider itself locked (e.g. whether a Git input has an
'rev' attribute).
Also, the 'locked' field was actually incorrect for Git inputs: it
would be set to true even for dirty worktrees. As a result, we got
away with using fetchTree() internally even though fetchTree()
requires a locked input in pure mode. In particular, this allowed
'--override-input' to work by accident.
The fix is to pass a set of "overrides" to call-flake.nix for all the
unlocked inputs (i.e. the top-level flake and any --override-inputs).
Also move `SourcePath` into `libutil`.
These changes allow `error.hh` and `error.cc` to access source path and
position information, which we can use to produce better error messages
(for example, we could consider omitting filenames when two or more
consecutive stack frames originate from the same file).
