we combine closures built by different users, the resulting set may
contain multiple paths from the same output path equivalence class.
For instance, if we do
$ NIX_USER_ID=foo nix-env -i libXext
$ NIX_USER_ID=root nix-env -i libXt
$ NIX_USER_ID=foo nix-env -i libXmu
(where libXmu depends on libXext and libXt, who both depend on
libX11), then the following will happen:
* User foo builds libX11 and libXext because they don't exist
yet.
* User root builds libX11 and libXt because the latter doesn't
exist yet, while the former *does* exist but cannot be trusted.
The instance of libX11 built by root will almost certainly
differ from the one built by foo, so they are stored in separate
locations.
* User foo builds libXmu, which requires libXext and libXt. Foo
has trusted copies of both (libXext was built by himself, while
libXt was built by root, who is trusted by foo). So libXmu is
built with foo's libXext and root's libXt as inputs.
* The resulting libXmu will link against two copies of libX11,
namely the one used by foo's libXext and the one used by root's
libXt. This is bad semantically (it's observable behaviour, and
might well lead to build time or runtime failure (e.g.,
duplicate definitions of symbols)) and in terms of efficiency
(the closure of libXmu contains two copies of libX11, so both
must be deployed).
The problem is to apply hash rewriting to "consolidate" the set of
input paths to a build. The invariant we wish to maintain is that
any closure may contain at most one path from each equivalence
class.
So in the case of a collision, we select one path from each class,
and *rewrite* all paths in that set to point only to paths in that
set. For instance, in the example above, we can rewrite foo's
libXext to link against root's libX11. That is, the hash part of
foo's libX11 is replaced by the hash part of root's libX11.
The hard part is to figure out which path to select from each
class. Some selections may be cheaper than others (i.e., require
fewer rewrites). The current implementation is rather dumb: it
tries all possible selections, and picks the cheapest. This is an
exponential time algorithm.
There certainly are more efficient common-case (heuristical)
approaches. But I don't know yet if there is a worst-case
polynomial time algorithm.
* Only build a derivation if there are no trusted output paths in the
equivalence classes for that derivation's outputs.
* Set the trust ID to the current user name, or use the value of the
NIX_USER_ID environment variable.
paths (e.g., `/nix/store/...random-hash...-aterm'), which are
subsequently rewritten to actual content-addressable store paths
(i.e., the hash part of the store path equals the hash of the
contents).
A complication is that the temporary output paths have to be passed
to the builder (e.g., in $out). Likewise, other environment
variables and command-line arguments cannot contain fixed store
paths because their names are no longer known in advance.
Therefore, we now put placeholder store paths in environment
variables and command-line arguments, which we *rewrite* to the
actual paths prior to running the builder.
TODO: maintain the mapping of derivation placeholder outputs
("output path equivalence classes") to actual output paths in the
database. Right now the first build succeeds and all its
dependencies fail because they cannot find the output of the first.
TODO: locking is no longer an issue with random temporary paths, but
at the cost of having no blocking if we build the same thing twice
in parallel. Maybe the "random" path should actually be a hash of
the placeholder and the name of the user who started the build.
content hashes. This is to prevent a rewrite of
...HASH...HASH...
and
...HASH...0000...
(where HASH is the randomly generated prefix) from hashing to the
same value. This would happen because they would both resolve to
...0000...0000... Exploiting this into a security hole is left as
an exercise to the reader ;-)
idea is that any component in the Nix store resides has a store path
name that has a hash component equal to the hash of the contents of
that component, i.e.,
hashPartOf(path) = hashOf(contentsAt(path))
E.g., a path /nix/store/nc35k7yr8...-foo would have content hash
nc35k7yr8...
Of course, when building components in the Nix store, we don't know
the content hash until after the component has been built. We
will handle this by building the component at some randomly
generated prefix in the Nix store, and then afterwards *rewriting*
the random prefix to the hash of the actual contents.
The tricky part is components that reference themselves, such as ELF
executables that contain themselves in their RPATH. We can support
this by computing content hashes "modulo" the original prefix, i.e.,
we zero out every occurence of the randomly generated prefix,
compute the content hash, then rewrite the random prefix to the
final location.
`removeAttrs attrs ["x", "y"]' returns the set `attrs' with the
attributes named `x' and `y' removed. It is not an error for the
named attributes to be missing from the input set.
* Make the `derivation' primitive much more lazy. The expression
`derivation attrs' now evaluates to (essentially)
attrs // {
type = "derivation";
outPath = derivation! attrs;
drvPath = derivation! attrs;
}
where `derivation!' is a primop that does the actual derivation
instantiation (i.e., it does what `derivation' used to do). The
advantage is that it allows commands such as `nix-env -qa' and
`nix-env -i' to be much faster since they no longer need to
instantiate all derivations, just the `name' attribute. (However,
`nix-env' doesn't yet take advantage of this since it still always
evaluates the `outPath' and `drvPath' attributes).
Also, this allows derivations to cyclically reference each other,
for example,
webServer = derivation {
...
hostName = "svn.cs.uu.nl";
services = [svnService];
};
svnService = derivation {
...
hostName = webServer.hostName;
};
Previously, this would yield a black hole (infinite recursion).
derivations. This is mostly to simplify the implementation of
nix-prefetch-{url, svn}, which now work properly in setuid
installations.
* Enforce valid store names in `nix-store --add / --add-fixed'.
continue building when one fails unless `--keep-going' is
specified.
* When `--keep-going' is specified, print out the set of failing
derivations at the end (otherwise it can be hard to find out which
failed).
