nixos/rfcs
1
0
Fork 0
mirror of https://github.com/NixOS/rfcs.git synced 2025-11-08 19:46:12 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge 1026104d8f into c655bdaab4

Browse source
This commit is contained in:
piegames 2025-05-26 11:51:26 -04:00 • committed by GitHub
commit e5529749f9
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 330 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

330
rfcs/0148-pipe-operator.md Normal file
View file

@ -0,0 +1,330 @@
---
feature: pipe-operator
start-date: 2023-05-23
author: @piegamesde
shepherd-team: @roberth @rhendric @illustris @adrian-gierakowski
shepherd-leader: @rhendric
related-issues: (will contain links to implementation PRs)
---
# Summary
[summary]: #summary
Introduce a new "pipe" operator, `|>`, to the Nix language, defined as `f a` = `a |> f`.
Additionally, elevate `lib.pipe` to a built-in function.
As a reminder, `pipe a [ f g h ]` is defined as `h (g (f a))`.
# Motivation
[motivation]: #motivation
Creating advanced data processing like transforming a list is a thing commonly done in nixpkgs.
Yet the language has no support for function concatentation/composition,
which results in such constructs looking unwieldy and difficult to format well.
`lib.pipe` may be the most powerful library function with that regard,
but it is unknown and overlooked by many because it is not easily discoverable:
Despite its great usefulness, it is currently used in less than 30 files in Nixpkgs
(`rg '[\. ]pipe .* \['`).
Additionally, it is not accessible to Nix code outside of nixpkgs,
and due to Nix's lazy evaluation debugging type errors is really difficult.
Let's have a look at an arbitrarily chosen snippet of Nixpkgs code:
```nix
defaultPrefsFile = pkgs.writeText "nixos-default-prefs.js" (lib.concatStringsSep "\n" (lib.mapAttrsToList (key: value: ''
// ${value.reason}
pref("${key}", ${builtins.toJSON value.value});
'') defaultPrefs));
```
It is arguably pretty hard to read and reason about. Even when applying some more whitespace-generous formatting:
```nix
defaultPrefsFile = pkgs.writeText "nixos-default-prefs.js" (
lib.concatStringsSep "\n" (
lib.mapAttrsToList
(
key: value: ''
// ${value.reason}
pref("${key}", ${builtins.toJSON value.value});
''
)
defaultPrefs
)
);
```
One can observe the following issues:
- If you want to follow the data flow, you must read it from bottom to top,
from the inside to the outside (the input here is `defaultPrefs`).
- Adding a function call to the output would require wrapping the entire
expression in parentheses and increasing its indentation.
Compare this to the equivalent call with `lib.pipe`:
```nix
defaultPrefsFile = pipe defaultPrefs [
(lib.mapAttrsToList (
key: value: ''
// ${value.reason}
pref("${key}", ${builtins.toJSON value.value});
''
))
(lib.concatStringsSep "\n")
(pkgs.writeText "nixos-default-prefs.js")
];
```
The code now clearly reads from top to bottom in the order the data is processed,
it is easy to add and remove processing steps at any point.
With a dedicated pipe operator, it would look like this:
```nix
defaultPrefsFile = defaultPrefs
|> lib.mapAttrsToList (
key: value: ''
// ${value.reason}
pref("${key}", ${builtins.toJSON value.value});
''
)
|> lib.concatStringsSep "\n"
|> pkgs.writeText "nixos-default-prefs.js";
```
The artificial distinction between the first input and the functions via the list now is gone,
and so are the parentheses around the functions.
With the lower character overhead, using the operator becomes attractive in more situations,
whereas a `pipe` pays for its overhead only in more complex scenarios (usually three functions or more).
Having a dedicated operator also increases visibility and discoverability of the feature.
# Detailed design
[design]: #detailed-design
## `|>` operator
A new operator `|>` is introduced into the Nix language.
It is defined as function application with the order of arguments swapped: `f a` = `a |> f`.
It is left-associative and has a binding strength weaker than function application:
`a |> f |> g b |> h` = `h ((g b) (f a))`.
# Examples and Interactions
[examples-and-interactions]: #examples-and-interactions
## Tooling support
Like any language extension, this will require the available Nix tooling to be updated.
Updating parsers should be pretty easy, as the syntax changes to the language are fairly minimal.
Tooling that evaluates Nix code in some way or does static code analysis should be easy to support too,
since one may treat the operator as syntactic sugar for function application.
No fundamentally new semantics are introduced to the language.
## Nixpkgs interaction
As soon as the Nixpkgs minimum version contains `|>`, using it will be allowed and encouraged in the documentation.
There might be efforts to automatically convert existing `lib.pipe` usage or even discourage/deprecate using that,
see future work.
### Existing lib functions
Nixpkgs `lib` contains a couple of functions that are concatenated versions of other lib functions,
for example `concatMapStringsSep` being a fuse of `map` and `concatStringsSep`.
This is not unusual in many programming languages,
nevertheless the existence of easy to use piping functionality would reduce the need for some of them.
Of course removing existing lib functions is not an option, but in the future,
newly added functions should meet stronger criteria than being purely convenience helpers replacing two function calls with one.
To keep with that example, is the function called `concatMapStringsSep` or `concatMapStringSep`?
In which order do you provide the mapper or the separator first?
Using `map (…) |> concatStringsSep` requires to memorize less information.
Some example with different alternatives:
```nix
lib.concatMapStringsSep "\n" (test: writeTest "success" test.name "${test}/bin/${test.name}") (lib.attrValues bin)
lib.concatStringsSep "\n" (map (test: writeTest "success" test.name "${test}/bin/${test.name}") (lib.attrValues bin))
lib.attrValues bin |> map (test: writeTest "success" test.name "${test}/bin/${test.name}") |> lib.concatStringsSep "\n"
lib.concatStringsSep "\n" <| map (test: writeTest "success" test.name "${test}/bin/${test.name}") <| lib.attrValues bin
```
# Prior art
Nickel has `|>` too, with the same name and semantics.
F# has `|>`, called "pipe-forward" operator, with the same semantics.
Additionally, it also has "pipe-backward" `<|` and `>>`/`<<` for forwards and backwards function composition.
`<|` is equivalent to function application, however its lower binding order allows removing parentheses:
`g (f a)` = `g <| f a`. All these operators have the same precedence and are left-associative.
F#'s `<|` being left-associative strongly reduces its power of usage,
this can be considered a mistake/compromise/collateral in the language design.
All other discussed variants of `<|` in other languages are right-associative.
Elm has the same operators as F#.
Haskell has the (backwards) function composition operator `.` in its prelude: `(g . f) a` = `g (f a)`.
It also has "reverse application" `&`, which is roughly equivalent to `|>`,
and `$`, which is function application again but right-associative and very weakly binding.
`.` binds stronger than both.
`|>` is definable as an infix function in several other programming languages,
and in even more languages as macro or higher-order function (including Nix, that's `lib.pipe`).
Notable, the Haskell package `flow` provides some common operators like `|>` and `<|`,
with the usual associativity and same binding strength (unlike Haskell's `$` and `&` discussed above).
Languages that allow for custom operators with custom associativity and precedence like Haskell and Scala
(but unlike F#) usually forbid mixing same-strengh operators with different associativity without using parentheses
as a syntax/compile error.
# Alternatives
[alternatives]: #alternatives
For each change this RFC proposes, there is always the trivial alternative of not doing it. See #drawbacks.
We could use the occasion and introduce more operators like those mentioned above.
## Function composition operators
Function composition is mostly interesting for the so-called "point-free" programming style,
where partially applied compositions of functions are preferred over the introduction of lambda terms.
However, Nix is not well suited for that programming style for various reasons,
nor would that point-free style have nearly as many applications in typical Nixpkgs code.
Take for example this library function, written in a point-free style by using `flip pipe` as function concatenation operator:
```nix
concatMapAttrs = f: flip pipe [ (mapAttrs f) attrValues (foldl' mergeAttrs { }) ];
```
When reading this code, one has to manually do the headwork of inferring the types to understand what this function does.
In Haskell, its powerful type system and type inference would quickly spot any mistakes made.
But in Nix, this can lead to very confusing runtime errors instead
(even ignoring the additional stack trace noise of using `flip pipe`).
Compare this to the fully specifified version of the same function:
```nix
concatMapAttrs = f: v: pipe v [ (mapAttrs f) attrValues (foldl' mergeAttrs { }) ];
```
Would you have guessed correctly from the first code example whether it's `f: v:` or `v: f:`?
## Pipe-forward vs pipe-backward
We could use `<|` instead of `|>` instead:
```nix
defaultPrefsFile =
pkgs.writeText "nixos-default-prefs.js" <|
lib.concatStringsSep "\n" <|
lib.mapAttrsToList (
key: value: ''
// ${value.reason}
pref("${key}", ${builtins.toJSON value.value});
''
) <| # the '<|' here is optional/redundant
defaultPrefs
;
```
`<|` also opens up to other scenarios in which `|>` might be less well suited
(examples inspired by https://github.com/NixOS/nix/issues/1845):
```nix
lib.makeOverridable <|
{ foo, bar }:
builtins.trace "my debug stuff" <|
# some more code here
```
While only one of them would probably be sufficient for most use cases, we could also have both `|>` and `<|`.
Given that we want to call them `|>` and `<|`, users should assume both having equal binding strength.
Therefore mixing them without parentheses should be forbidden like in other languages,
having `<|` weaker than `|>` like Haskell's `$` and `&` would be a bad idea.
## Change the `pipe` function signature
There are many equivalent ways to declare this function, instead of just using the current design.
For example, one could flip its arguments so that it becomes function composition on list of functions.
Not only can this function trivially replace pipe; it can also be readily used where a function is expected, such as in `map`.
The current design of `pipe` has the advantage that its asymmetry points at its operating direction, which is quite valuable.
## `apply` keyword
As suggested in https://github.com/NixOS/rfcs/pull/148#discussion_r1206966546,
one could introduce a keyword (tentatively called `apply`) for piping,
which syntactically similar to `with` and `assert` statements:
```nix
apply f;
apply g;
x
# The same as
f (g x)
```
The biggest disadvantage with it is backwards compatibility of adding a new keyword into the language,
which would require solving language versioning first (see RFC #137).
This approach would be roughly equivalent to introducing a `<|` operator.
See the above for a discussion on the overall design space of that.
## `builtins.pipe`
`lib.pipe`'s functionality could be implemented as a built-in function.
The main motivation for this is that it allows to give better error messages
like line numbers when some part of the pipeline fails:
Currently `lib.pipe` internally uses a fold over the list,
therefore any type mismatches will give a trace which points into `lib.fold`,
leaving the user without the information at which stage of the pipeline it failed.
(This is less of a problem when used in packages, but significant enough that currently,
`lib.pipe` unfortunately should not be used in the implementation of any library functions.)
This could probably be fixed within Nixpkgs alone,
however not without incurring a significant performance penalty for using "reflection".
A built-in operator would be able to provide this more detailed error information basically for free.
Additionally, it allows easy usage outside of Nixpkgs and increases discoverability.
While Nixpkgs is bound to minimum Nix versions and thus `|>` won't be available until
several years after its initial implementation,
it can directly benefit from `builtins.pipe` and its better error diagnostic by overriding `lib.pipe`.
Elevating a Nixpkgs library function to a builtin has been done several times before,
for example `bitAnd`, `splitVersion` and `concatStringsSep`.
The main drawback is that once `|>` is available, there is little use for `builtins.pipe` anymore,
so the main purpose of that would be as a stop-gap for Nixpkgs
until the minimum Nix version is sufficiently high to allow using `|>`.
# Drawbacks
[drawbacks]: #drawbacks
- Introducing `|>` has the drawback of adding complexity to the language, and it will break older tooling.
# Unresolved questions
[unresolved]: #unresolved-questions
- What is the precise binding strength of the operator?
- Who is going to implement this in Nix?
- How difficult will the implementation be?
- Will this affect evaluation performance in some way?
- There is reason to expect that replacing `lib.pipe` with a builtin will reduce its overhead,
and that the builtin should have little to no overhead compared to regular function application.
In order to decide which operators to add to the language (see Alternatives),
a larger survey across the Nixpkgs code will be conducted.
This will give us quantitative information to better make any decisions involving tradeoffs.
# Future work
[future]: #future-work
Once introduced and usable in Nixpkgs, existing code may benefit from being migrated to using these features.
Automatically transforming nested function calls into pipelines is unlikely,
as doing so is not guaranteed to always be a subjective improvement to the code.
It might be possible to write a lint which detects opportunities for piping, for example in nixpkgs-hammering.
On the other hand, the migration from `pipe` to `|>` should be a straightforward transformation on the syntax tree.