nixos/rfcs
1
0
Fork 0
mirror of https://github.com/NixOS/rfcs.git synced 2025-12-20 07:51:18 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions

Rewrite most of RFC after 1st meeting

Browse source
This commit is contained in:
Doron Behar 2021-02-24 20:08:37 +02:00
parent 7b0c9e11fa
commit 034a13e50e
1 changed files with 110 additions and 118 deletions
Download patch file Download diff file Expand all files Collapse all files

230
rfcs/0075-declarative-wrappers.md
View file

@ -12,11 +12,12 @@ related-issues: POC Implementation at [#85103](https://github.com/NixOS/nixpkgs/
[summary]: #summary [summary]: #summary
Manage the environment of wrappers declaratively and deprecate shell based Manage the environment of wrappers declaratively and deprecate shell based
methods for calculating runtime environment of packages. Make wrappers a methods for calculating runtime environment of packages. Make it easier to
separate derivation so that mere changes to the environment will not trigger a debug why env vars are added to an executable, by storing this information
rebuild. Make it easier to debug why env vars are added to an executable, by inside `/nix/store/.../nix-support/` of the dependencies that using them
using Nix as the language to evaluate what env vars are needed, instead of not requires require runtime environment. Create a new `makeWrapperAuto` hook that
documented good enough and not easily debug-able shell hooks. will make the `fixupPhase` read all of the deps environment that's needed and
automatically wrap the executables with the proper environment.
# Motivation # Motivation
[motivation]: #motivation [motivation]: #motivation
@ -218,120 +219,132 @@ and get rid of these global environment variables.
# Detailed design # Detailed design
[design]: #detailed-design [design]: #detailed-design
The current design is roughly implemented at The end goal is to make the experience of getting a derivation wrapped as
[pull 85103](https://github.com/NixOS/nixpkgs/pull/85103) . automatic as possible. A derivation that needs a certain environment to run,
will put the dependency that is linked to this environment variable in
`envInputs` and `mkDerivation`'s standard `fixupPhase`. As a start, we'll (me
probably) will introduce a new `makeWrapperAuto` setup hook that will take care
in the way described as follows.
The idea is to have a Nix function, let us call it `wrapGeneric`, with an As a start, we'll need to think about all packages in Nixpkgs that using them
interface similar to requires some environment variables to be set. Every such package will put in
[`wrapMpv`](https://github.com/NixOS/nixpkgs/blob/a5985162e31587ae04ddc65c4e06146c2aff104c/pkgs/applications/video/mpv/wrapper.nix#L9-L23) `$dev/nix-support/wrappers.json` a list of environment variables that are
and "linked" to this package. "linked" means that using this package requires these
[`wrapNeovim`](https://github.com/NixOS/nixpkgs/blob/a5985162e31587ae04ddc65c4e06146c2aff104c/pkgs/applications/editors/neovim/wrapper.nix#L11-L24) environment variables to be set in runtime.
which will accept a single derivation or an array of them and will wrap all of
their executables with the proper environment, based on their inputs.
`wrapGeneric` should iterate recursively all `buildInputs` and `makeWrapperAuto` will traverse all `buildInputs` and `propagatedBuildInputs`
`propagatedBuildInputs` of the input derivation(s), and construct an attrset with of a derivation, and look for a `wrappers.json` file in these inputs. It will
which it'll calculate the necessary environment of the executables. Then either collect all the environment variables that need to be set in the resulting
via `wrapProgram` or a better method, it'll create the wrappers. executables, by merging all of the values of the environment variables in all
of the inputs' `wrappers.json` files. `wrappers.json` might look like this for
a given package:
A contributor using `wrapGeneric` shouldn't _care_ what type of wrapping needs ```json
to be performed on his derivation's executables - whether these are Qt related {
wrappings or a Gtk / gobject related. `wrapGeneric` should know all there is to "GI_TYPELIB_PATH": [
know about environment variables every library / input may need during runtime, "/nix/...gobject-introspection.../...",
and with this information at hand, construct the necessary wrapper. "/nix/...librsvg.../...",
],
"GIO_EXTRA_MODULES": [
"/nix/...dconf.../lib/gio/modules"
],
"XDG_DATA_DIRS": [
"/nix/...gtk+3.../...",
"/nix/...gsettings-desktop-schemas.../..."
],
"GDK_PIXBUF_MODULE_FILE": "/nix/...librsvg.../lib/gdk.../loaders.cache",
}
```
In order for `wrapGenric` to know all of this information about our packaged The information found inside an input's `wrappers.json` will include all the
libraries - the information about runtime env, we need to write in the information about wrappers found in the input's inputs, and so on. Thus in
`passthru`s of these libraries, what env vars they need. Such information was contrast to the [POC Nixpkgs PR](https://github.com/NixOS/nixpkgs/pull/85103)
added in the POC pull at [commit and the [original design of the
@6283f15](https://github.com/NixOS/nixpkgs/pull/85103/commits/6283f15bb9b65af64571a78b039115807dcc2958). RFC](https://github.com/doronbehar/rfcs/blob/60d3825fdd4e6574b7e5d70264445d1c801368c6/rfcs/0075-declarative-wrappers.md#L251),
prior to [the 1st
meeting](https://github.com/NixOS/rfcs/pull/75#issuecomment-760942876),
traversing all the inputs and the inputs' inputs, will not happen during eval
time and only partly, during build time - every package already built will be
able to give another package all the information there needs to know about the
related environment variables.
Additional features / improvements are [already Most of the work to do will be:
available](https://github.com/NixOS/nixpkgs/pull/85103#issuecomment-614195666)
in the POC pull. For example:
- It should be **impossible** for multi-value env vars to have duplicates, as 1. Gather information about what environment variables are "linked" to each
that's guaranteed by Nix' behavior when constructing arrays / attrsets. package, and edit these derivations to include a `wrappers.json`, with
- Asking the wrapper creator to use more links and less colon-separated values `makeWrapperAuto`.
in env vars - should help avoid what [pull 2. Design the `makeWrapperAuto` shell hook:
84689](https://github.com/NixOS/nixpkgs/pull/84689) fixed. - It should introduce a shell function (to be called `wrappersInfo`) that
will allow piping a JSON string from `builtins.toJSON` and spit a
`wrappers.json` that will include both what was piped into it, and the
content from the package's inputs' `wrappers.json`.
- It should make the executables in `$out/bin/` get wrapped according to
what's currently in this package's `wrappers.json`, during `fixupPhase`.
- The above should be also possible to do manually for executables outside
`$out/bin/` with say adding to a derivation a Nix variable:
```nix
wrapExtraPrograms = [ "/libexec/" "/share/scripts" ];
```
3. Most of the packages with linked environment variables, have lots of reverse
dependencies, so once `makeWrapperAuto` is ready, it'd nice to have a hydra
job that will build all of these packages with the `wrappers.json` file in
them. For instance these packages include:
- `gdk-pixbuf`
- `gsettings-desktop-schemas`
- `pango`
- `gtk3`
# Examples and Interactions # Examples and Interactions
[examples-and-interactions]: #examples-and-interactions [examples-and-interactions]: #examples-and-interactions
All examples are copied from and based on the [POC When switching to `makeWrapperAuto` from `makeWrapper` there shouldn't be
pull](https://github.com/NixOS/nixpkgs/pull/85103). manual usage of `wrapProgram` for most cases. A package that uses `wrapProgram`
should be able to switch to `wrappersInfo` and declare any nontrivial
environment variables with it to get propagated to reverse dependencies and to
it's executables automatically.
Here's a new method of creating a python environment: Currently I imagine the usage of `wrappersInfo` (the name can be debated) as
so:
```nix ```nix
my-python-env = wrapGeneric python3 { # Propagate GST plugins' path
linkByEnv = { postInstall = ''
# you can use "link" here and that will link only the necessary files echo "${builtins.toJSON {
# to the runtime environment. GST_PLUGIN_SYSTEM_PATH_1_0 = [
PYTHONPATH = "linkPkg"; # @out@ should be expanded by `wrappersInfo` to what's in `$out`, see:
}; # https://github.com/NixOS/nixpkgs/pull/85103#issuecomment-613071343
extraPkgs = with python3.pkgs; [ "@out@/lib/gstreamer-1.0"
matplotlib
]; ];
wrapOut = { }}" | wrappersInfo
# tells wrapGeneric to add to the wrappers this value for PYTHONPATH.
# Naturally, this should play along with the values given in
# linkByEnv.PYTHONPATH.
PYTHONPATH = "$out/${python3.sitePackages}";
};
};
```
Consider a package is wrapped without directly making accessible the unwrapped
derivation. Meaning, say `all-packages.nix` has:
```nix
my-awesome-pkg = wrapGeneric (callPackage ../applications/my-awesome-pkg { }) { };
```
Assuming the user knows `my-awesome-pkg` is wrapped with `wrapGeneric`, they would
need to use an overlay like this, to override the unwrapped derivation:
```nix
self: super:
{
my-awesome-pkg = super.wrapGeneric (
super.my-awesome-pkg.unwrapped.overrideAttrs(oldAttrs: {
preFixup = ''
overriding preFixup from an overlay!!
''; '';
})
) {};
}
``` ```
And to override the wrapper derivation, e.g to add new optional features not `wrapQtAppsHook` and `wrapGAppsHook` should be replaced with `makeWrapperAuto`
strictly necessary (as in [pull while enable derivations to get rid of well known workarounds such as:
83482](https://github.com/NixOS/nixpkgs/pull/83482)), it should be possible
using:
```nix ```nix
self: super: # hook for gobject-introspection doesn't like strictDeps
# https://github.com/NixOS/nixpkgs/issues/56943
strictDeps = false;
```
{ And:
my-awesome-pkg = super.wrapGeneric super.my-awesome-pkg.unwrapped {
extraPkgs = [ ```nix
super.qt5.certain-qt-plugin preFixup = ''
]; makeWrapperArgs+=("''${qtWrapperArgs[@]}")
}; '';
}
``` ```
# Drawbacks # Drawbacks
[drawbacks]: #drawbacks [drawbacks]: #drawbacks
The current design is heavily based on Nix, and knowing how to write and debug Using `wrapProgram` will be simpler then using `wrappersInfo` and it might be
Nix expressions is a skill not everyone are akin to learn. Also, overriding a hard to explain why is there no `wrapProgramAuto`. However, this interface
wrapped derivation is somewhat more awkward, due to this. Perhaps this might get improved in design through this RFC or in the future and in any case
interface could be improved, and for sure proper documentation written should proper documentation should help.
help.
# Alternatives # Alternatives
[alternatives]: #alternatives [alternatives]: #alternatives
@ -339,36 +352,15 @@ help.
Perhaps our shell hooks _can_ be fixed / improved, and we could help make it Perhaps our shell hooks _can_ be fixed / improved, and we could help make it
easier to debug them via `NIX_DEBUG`. Then it might help us track down e.g why easier to debug them via `NIX_DEBUG`. Then it might help us track down e.g why
environment variables are added twice etc. Still though, this wouldn't solve environment variables are added twice etc. Still though, this wouldn't solve
half of the other issues presented here. Most importantly, the shell hooks rely many issues presented above.
upon being in the inputs during build of the original derivation. Hence, mere
requests for changes to an environment a wrapper sets, trigger rebuilds that
take a lot of time and resources from average users. See [this
comment](https://github.com/NixOS/nixpkgs/pull/88136#issuecomment-632674653).
# Unresolved questions # Unresolved questions
[unresolved]: #unresolved-questions [unresolved]: #unresolved-questions
The POC implementation does 1 thing which I'm most sure could be done better, Discussing the design I guess, here or in the Nixpkgs PR that will follow this
and that's iterating **recursively** all `buildInputs` and RFC.
`propagatedBuildInputs` of the given derivations. This is currently implemented
with a recursive (Nix) function, prone to reach a state of infinite recursion.
This risk is currently mitigated using an array of packages we know don't need
any env vars at runtime, and for sure are very much at the bottom of the list
of all Nixpkgs' dependency graph. This part is implemented
[here](https://github.com/NixOS/nixpkgs/pull/85103/files#diff-44c2102a355f50131eb8f69fb7e7c18bR75-R131).
There are other methods of doing this recursive search, but I haven't yet
investigated all of them. For reference and hopefully for an advice, this need
was requested by others and discussed at:
- [nix issue 1245](https://github.com/NixOS/nix/issues/1245).
- [Interesting idea by @aszlig at nix issue
1245](https://github.com/NixOS/nix/issues/1245#issuecomment-401642781).
- [@nmattia's
post](https://www.nmattia.com/posts/2019-10-08-runtime-dependencies.html).
- [Discourse thread](https://discourse.nixos.org/t/any-way-to-get-a-derivations-inputdrvs-from-within-nix/7212/3).
# Future work # Future work
[future]: #future-work [future]: #future-work
Not that I can think of. Fix all wrapper related issues declaratively!