nixos/rfcs
1
0
Fork 0
mirror of https://github.com/NixOS/rfcs.git synced 2025-11-09 12:06:11 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions

Review

Browse source
This commit is contained in:
Marcus Bointon 2025-08-27 16:57:00 +02:00 • committed by Pierre Penninckx
parent f81fecc0ec
commit e9e966aea0
1 changed files with 158 additions and 208 deletions
Download patch file Download diff file Expand all files Collapse all files

364
rfcs/0189-contracts.md
View file

@ -11,79 +11,71 @@ related-issues: https://github.com/NixOS/nixpkgs/pull/432529
# Summary # Summary
[summary]: #summary [summary]: #summary
In Nixpkgs, modules duplicate a lot of code to setup their dependencies. In nixpkgs, modules include a lot of duplicate code to set up their dependencies.
We introduce a pattern that allows to move We introduce a pattern for moving this custom code out of the modules and making it shareable
this custom code out of the modules and make it shareable in an incremental, backwards-compatible, extensible, and testable way.
in an incremental, backwards compatible, extensible and testable way.
# Motivation # Motivation
[motivation]: #motivation [motivation]: #motivation
As a motivating example, let's take a module As a motivating example, let's take a module
that sets up a service that needs a database that sets up a service that needs a database
and this database can be PostgreSQL or MySQL. which can be PostgreSQL or MySQL.
Letting the user choose which database they want Letting the user choose which database they want to use
is a great feature to have for a module but it is is a great feature to have for a module, but it requires
a lot of code and must be thought out a lot of code that has nothing to do with the module's core functionality,
thoroughly to get it right and to test correctly. and is difficult to get right and to test thoroughly.
Having this code live in each module separately Having this code live in each separate module is a waste for the whole community.
is a waste for the whole community. We see many disadvantages:
We see at least those disadvantages:
- It's more code to review and maintain for everybody. - It's more code to review and maintain for everybody.
- More burden on maintainers of a module implementing this feature: - Increased burden on maintainers for every module implementing this feature:
they must know how to setup their dependencies at a low-level they must know how to set up their dependencies at a low level,
and must keep the code up to date. and must keep that code up to date.
- Leads to difference in interface: - Setting up the same dependency across different modules can use an entirely different interface.
options to setup a same dependency are different across modules. - Every maintainer has their own style and knowledge,
- Leads to difference in implementation: leading to large variations in quality and reliability across implementations.
every maintainer has their own style and knowledge, - As a consequence of maintainer burden, very few modules allow you to choose from multiple dependencies (e.g. PostgreSQL, MySQL or other).
leading to not every implementation being of the same quality - Dependencies can't be changed or extended without changing the module's source code:
and being tested equally. a user cannot easily choose to use a dependency the maintainer didn't add code for (e.g. SQLite).
- Leads to difference in features:
some implementations are more featureful than others,
very few modules allow you to choose from multiple dependencies (e.g. PostgreSQL, MySQL or other).
- Not extendable without changing the source code:
a user cannot easily choose to use a dependency the maintainer didn't add code for.
What we propose answers to all those issues This proposal resolves all those issues, as well as allowing a few things that are not currently possible:
as well as allows a few things that's not possible currently:
- interfacing with dependencies and services outside of NixOS, - Interfacing with dependencies and services outside of NixOS,
- using stubs in NixOS tests. - Using stubs in NixOS tests.
# Detailed design # Detailed design
[design]: #detailed-design [design]: #detailed-design
The core idea is to decouple the usage of a feature and its implementation. The core idea is to decouple the use of a feature from its implementation.
Let's first introduce some nomenclature: Let's first introduce some nomenclature:
- _consumer_: The module using or needing a feature. - _consumer_: The module using or needing a feature.
Example: Nextcloud, Vaultwarden and others consume a database. Example: Nextcloud, Vaultwarden and others require a database.
- _provider_: The module implementing a feature. - _provider_: The module implementing a feature.
Example: PostgreSQL, MySQL or SQlite provide a database. Example: PostgreSQL, MySQL, or SQLite provide database services.
- _inputs_: The set of options the consumer uses to communicate with the provider. - _inputs_: The set of options the consumer uses to communicate with the provider.
- _outputs_: The set of options the provider uses to communicate back to the consumer. - _outputs_: The set of options the provider uses to communicate back to the consumer.
- _contract_: The concept sitting in-between a consumer and provider - _contract_: The concept sitting between a consumer and provider
and making them agree on the `inputs` and `output`. defining the `inputs` and `outputs`.
The _contract_ is a submodule with imposed options The _contract_ is a submodule with imposed options
associated with a behavior which every _provider_ must respect associated with a behavior which every _provider_ must respect
and which is enforced through generic NixOS tests. and which is enforced through generic NixOS tests.
A _consumer_ and _provider_ fit then together thanks to structural typing A _consumer_ and _provider_ can then fit together thanks to structural typing
thanks to the contract enforcing the same `inputs` and `outputs` on both sides. in the contract, enforcing matching `inputs` and `outputs` on each side.
Structural typing was chosen because it fits nicely with Structural typing was chosen because it fits nicely with
the existing module system. This follows the self-imposed constraint the existing module system. This follows the self-imposed constraint
of being as much backwards compatible as possible. of maintaining as much backwards compatibility as possible.
Indeed, this design can be added to existing modules incrementally Indeed, this design can be added to existing modules incrementally
and in a backwards compatible way ,and in a backwards-compatible way,
by adding a new option with the contract name by adding a new option with the contract name
which will translate options from the contract which will translate options from the contract
to options already defined by the existing module. into options already defined by the existing module.
Identified possible contracts are: Some examples of possible contracts:
- File backup - File backup
- Streaming backup (for databases) - Streaming backup (for databases)
- Secrets (out of store values) provisioning - Secrets (out of store values) provisioning
@ -94,11 +86,11 @@ Identified possible contracts are:
- LDAP user and group management - LDAP user and group management
- OIDC provider integration - OIDC provider integration
- Forward auth setup - Forward auth setup
- Any implicit convention in nixpkgs can be encoded this way
This RFC's goal is _not_ to define all those contracts Any implicit convention in nixpkgs can be encoded this way.
nor to identify the exhaustive list of existing contracts.
It's goal is to define a pattern, taking as example a few diverse examples. This RFC's goal is _not_ to define all these contracts
nor to identify an exhaustive list of existing contracts, but to define a pattern derived from a few diverse examples.
These contracts will live under a new option path `contracts` These contracts will live under a new option path `contracts`
like `contracts.fileBackup` and `contracts.streamingBackup`. like `contracts.fileBackup` and `contracts.streamingBackup`.
@ -108,12 +100,11 @@ See [prior-art][] for some useful comparisons that can help you get a better pic
# Implementation # Implementation
[implementation]: #implementation [implementation]: #implementation
The implementation has been worked out initially in the [SelfHostBlocks][] repo and perfected in the [module interfaces][] repo. The implementation was worked out initially in the [SelfHostBlocks][] repo and perfected in the [module interfaces][] repo.
There are some slight variations proposed in this RFC compared to the module interfaces repo to get it out sooner rather than later. See the [corresponding unresolved section][unresolved-duallink] section. There are some slight variations proposed in this RFC relative to the module interfaces repo to get it out sooner rather than later. See the [corresponding unresolved section][unresolved-duallink].
It is important to keep in mind the following implementation comes from It is important to keep in mind that the proposed implementation comes from
seeing this pattern emerge "in the wild". The implementation came naturally seeing this pattern emerge naturally "in the wild" from trying to increase code reuse, providing solid evidence on the utility of this approach.
out of trying to increase code reuse. This somewhat legitimizes the implementation.
[SelfHostBlocks]: https://github.com/ibizaman/selfhostblocks/tree/main/modules/contracts [SelfHostBlocks]: https://github.com/ibizaman/selfhostblocks/tree/main/modules/contracts
[module interfaces]: https://github.com/fricklerhandwerk/module-interfaces [module interfaces]: https://github.com/fricklerhandwerk/module-interfaces
@ -126,28 +117,27 @@ There are up to 4 different individuals or teams involved for one contract:
```mermaid ```mermaid
flowchart TD flowchart TD
ProviderTeam(["Provider Team"]) -. Maintains .-> Provider["Provider"] ProviderTeam(Provider Team) -. Maintains .-> Provider
Contract["Contract"] --> Provider & Consumer["Consumer"] Contract["Contract"] --> Provider & Consumer
ContractTeam(["Contract Team"]) -. Maintains .-> Contract ContractTeam("Contract Team") -. Maintains .-> Contract
EndUser["End User"] -.-> Provider & Consumer EndUser["End User"] -.-> Provider & Consumer
ConsumerTeam(["Consumer Team"]) -. Maintains .-> Consumer ConsumerTeam("Consumer Team") -. Maintains .-> Consumer
Provider@{ shape: rect} Provider["Provider"]
Consumer@{ shape: rect} Consumer["Consumer"]
EndUser@{ shape: rounded}
``` ```
1. `contract team`: The team maintaining a contract. 1. `Contract Team`: The team maintaining a contract.
2. `provider team`: The team maintaining one module provider of that contract. Each provider of a same contract can have its own team. 2. `Provider Team`: The team maintaining one module provider of that contract. Each provider of a same contract can have its own team.
3. `consumer team`: The team maintaining one module consumer of that contract. Each consumer of a same contract can have its own team. 3. `Consumer Team`: The team maintaining one module consumer of that contract. Each consumer of a same contract can have its own team.
4. `end user`: The end user linking one consumer of their choice with one provider of their choice for that contract. 4. `End User`: The end user linking one consumer of their choice with one provider of their choice for that contract.
Note that the contract is the central component here. Note that the `Contract` is the central component here.
The provider and the consumer teams do not need to know what the other team is doing, The provider and the consumer teams do not need to know what the other team is doing,
they can simply follow the contract and it will guarantee interoperability. they can simply follow the contract, and it will guarantee interoperability.
One nice property here is the `end user` can themselves add a new provider or consumer. One nice property here is the `End User` can add a new provider or consumer themselves.
One more property is a module can consume or provide one or multiple times the same contract or different contracts. A module can consume or provide multiple instances of the same or different contracts, for example a single HTTP server module might provide `Web Server` and `Reverse Proxy` contracts.
## Data Flow ## Data Flow
[dataflow]: #dataflow [dataflow]: #dataflow
@ -156,11 +146,11 @@ Another consideration before looking at the code is how data flows through a con
```mermaid ```mermaid
sequenceDiagram sequenceDiagram
participant Consumer as Consumer participant Consumer
participant Contract as Contract participant Contract
participant Provider as Provider participant Provider
participant EndUser as End User participant EndUser as End User
participant Config as Config participant Config
autonumber autonumber
Consumer ->> Contract: set input Consumer ->> Contract: set input
Contract ->> Provider: read input Contract ->> Provider: read input
@ -169,42 +159,38 @@ sequenceDiagram
end end
Provider ->> Config: do side effect Provider ->> Config: do side effect
opt opt
Provider ->> Contract: set ouput Provider ->> Contract: set output
end end
opt opt
Contract ->> Consumer: read output Contract ->> Consumer: read output
end end
``` ```
1. A `consumer` sets the `input` option of the contract. 1. A `Consumer` sets the `input` option of the contract.
2. The `provider` reads from that `input` option. 2. The `Provider` reads from that `input` option.
3. The `provider` optionally accepts provider-specific options set by the `end user`. 3. The `Provider` optionally accepts provider-specific options set by the `End User`.
4. The `provider` does some side effect (otherwise, there's no point). 4. The `Provider` does some side effect (otherwise, there's no point).
5. The `provider` optionally writes to the `output` of the contract. 5. The `Provider` optionally writes to the `output` of the contract.
6. The `consumer` optionally reads from the `output` of the contract. 6. The `Consumer` optionally reads from the `output` of the contract.
If you squint, this looks just like function application, only applied at the module level. If you squint, this looks just like a functional application, only applied at the module level.
## Contract Interface ## Contract Interface
[Contract Interface]: #contract-interface [Contract Interface]: #contract-interface
_The draft PR from which the following snippets are taken can be found [here][draftPR]._ _The following snippets are taken from the [draft PR][draftPR]._
_The intended reading order is first this document then going to the PR afterwards._ _The intended reading order is first this document, then the PR._
[draftPR]: https://github.com/NixOS/nixpkgs/pull/432529 [draftPR]: https://github.com/NixOS/nixpkgs/pull/432529
Links to relevant commits: Links to relevant commits:
- [contracts: init underlying module][] - [contracts: init underlying module](https://github.com/NixOS/nixpkgs/pull/432529/commits/bb561e9927ff73be12122644362ec3a1af61fd20)
- [contracts: add option to declare behavior tests][] - [contracts: add option to declare behavior tests](https://github.com/NixOS/nixpkgs/pull/432529/commits/75be2ddbc5b260a2a2e7f03c0103af803f54879b)
- [contracts: allow consumer to be unset][] - [contracts: allow consumer to be unset](https://github.com/NixOS/nixpkgs/pull/432529/commits/891ef82cf57bf31f7f4c02fae6d9739147af1753)
[contracts: init underlying module]: https://github.com/NixOS/nixpkgs/pull/432529/commits/bb561e9927ff73be12122644362ec3a1af61fd20
[contracts: add option to declare behavior tests]: https://github.com/NixOS/nixpkgs/pull/432529/commits/75be2ddbc5b260a2a2e7f03c0103af803f54879b
[contracts: allow consumer to be unset]: https://github.com/NixOS/nixpkgs/pull/432529/commits/891ef82cf57bf31f7f4c02fae6d9739147af1753
We declare a new top-level option `contracts` of type `attrsOf (submodule ...)`. We declare a new top-level option `contracts` of type `attrsOf (submodule ...)`.
Each contract will be a new value of this option. Each contract will define a new value for this option.
With the `description` fields removed for brevity, the option is declared like so: With the `description` fields removed for brevity, the option is declared like so:
@ -293,35 +279,35 @@ Let's review this submodule option by option.
The following two options are only used when defining a new contract. The following two options are only used when defining a new contract.
- `input`: Input options for the contract. The `deferredModule` allows for the options to be declared independently in each contract. - `input`: Input options for the contract. `deferredModule` in the inherited types allows for the options to be declared independently in each contract.
- `output`: Output options for the contract. Same remark about `deferredModule`. - `output`: Output options for the contract, with the same use of `deferredModule`.
Now that we have the options to declare the `input` and `output` of a contract, Now that we have the ability to declare the `input` and `output` options of a contract,
we can declare matching `consumer` and `provider` options using dependent types. we can declare matching `consumer` and `provider` options using dependent types.
- `consumer`: Submodule option with 3 nested options: - `consumer`: Submodule option with 3 nested options:
- `provider`: The linked `provider` for this consumer. - `provider`: The linked `provider` for this consumer.
This has to be set by the `end user` as they choose which consumer and provider to link. This has to be set by the `end user` as they choose which consumer and provider to link.
- `input`: Option whose type comes from the top-level `input` `deferredModule`. - `input`: An option whose type comes from the top-level `input` `deferredModule`.
This option is made writable because the `consumer` is expected to write to it. This option is made writable because the `consumer` is expected to write to it.
- `output`: Option whose type comes from the top-level `output` `deferredModule`. - `output`: An option whose type comes from the top-level `output` `deferredModule`.
This option is made `readOnly` because the `consumer` should only read from it. This option is made `readOnly` because the `consumer` should only read from it.
Its default value comes from the linked `provider`'s `output`. Its default value comes from the linked `provider`'s `output`.
- `provider`: Submodule option with 3 nested options: - `provider`: Submodule option with 3 nested options:
- `consumer`: The linked `consumer` for this provider. - `consumer`: The linked `consumer` for this provider.
This has to be set by the `end user` as they choose which consumer and provider to link. This has to be set by the `end user` as they choose which consumer and provider to link.
This option is made nullable because the end user is not required to always use a contract. This option is made nullable because the end user is not necessarily required to use a contract.
- `input`: Option whose type comes from the top-level `input` `deferredModule`. - `input`: An option whose type comes from the top-level `input` `deferredModule`.
This option is made `readOnly` because the `provider` should only read from it. This option is made `readOnly` because the `provider` should only read from it.
Its default value comes from the linked `consumer`'s `input`. Its default value comes from the linked `consumer`'s `input`.
- `output`: Option whose type comes from the top-level `output` `deferredModule`. - `output`: An option whose type comes from the top-level `output` `deferredModule`.
This option is made writable because the `provider` is expected to write to it. This option is made writable because the `provider` is expected to write to it.
- `behaviorTest`: A full NixOS VM test which enforces similar side effects - `behaviorTest`: A full NixOS VM test which enforces similar side effects
for all providers of a given contract. The test is generic on the provider for all providers of a given contract. The test is generic on the provider,
and each provider must instantiate this generic test to verify they do indeed and each provider must instantiate this generic test to verify they do indeed
implement a contract. It is used to enforce any behavior not captured by the types. implement the declared contract. It is used to enforce any behavior not captured by the types.
The `end user` would then combine a consumer and provider like so: The `end user` would then combine a consumer and provider like so:
@ -340,49 +326,44 @@ config = {
}; };
``` ```
Notice the `end user` must link the consumer and provider both ways. Notice the `end user` must link the consumer and provider in both directions.
This is discussed in [the unresolved section][unresolved]. This is discussed in [the unresolved section][unresolved].
# Examples and Interactions # Examples and Interactions
[examples-and-interactions]: #examples-and-interactions [examples-and-interactions]: #examples-and-interactions
In this section we will explain, for each contract implemented in the PR, In this section we will explain, for each contract implemented in the PR,
why they are useful and their interesting properties. For actual code, why they are useful, and their interesting properties. See the PR for actual code.
instead of simply copying the code here, see the PR.
## File Backup Contract ## File Backup Contract
[fileBackupContract]: #file-backup-contract [fileBackupContract]: #file-backup-contract
Links to relevant commits: Links to relevant commits:
- [file backup contract: init][] - [file backup contract: init](https://github.com/NixOS/nixpkgs/pull/432529/commits/a59b42345c64e5d9f793fad779dcfbc02d1918a0)
- [restic: implement file backup contract provider][] - [restic: implement file backup contract provider](https://github.com/NixOS/nixpkgs/pull/432529/commits/762a7318e3cd47f02743b46227595acf250a3084)
- [restic: define file backup contract behavior test][] - [restic: define file backup contract behavior test](https://github.com/NixOS/nixpkgs/pull/432529/commits/ad5751c854c0effb2a4c5bfbb993288f755c659e)
- [nextcloud: use file backup contract][] - [nextcloud: use file backup contract](https://github.com/NixOS/nixpkgs/pull/432529/commits/6b7a87adc0b6c3d476ca6caa5d9ce4f1846049c1)
[file backup contract: init]: https://github.com/NixOS/nixpkgs/pull/432529/commits/a59b42345c64e5d9f793fad779dcfbc02d1918a0
[restic: implement file backup contract provider]: https://github.com/NixOS/nixpkgs/pull/432529/commits/762a7318e3cd47f02743b46227595acf250a3084
[restic: define file backup contract behavior test]: https://github.com/NixOS/nixpkgs/pull/432529/commits/ad5751c854c0effb2a4c5bfbb993288f755c659e
[nextcloud: use file backup contract]: https://github.com/NixOS/nixpkgs/pull/432529/commits/6b7a87adc0b6c3d476ca6caa5d9ce4f1846049c1
This contract is for modules that have files to be backed up. This contract is for modules that have files to be backed up.
Without this contract, a user wanting to back up a service Without this contract, a user wanting to back up a service
must know the layout of the service on the file system. must know the layout of the service on the file system.
Usually there is a `dataDir` option or similar, so one Usually there is a `dataDir` option or similar, so one
can suspect backing this up is enough. But maybe not? might suspect that backing this up is enough. But what if this isn't true,
There is no way to know apart from reading the upstream documentation. and you end up making backups that can't be restored?
There is no way to know except by reading the upstream documentation.
But even then, one must remember to use the correct user But even then, one must also remember to use the correct user
to run the backup. If not, the backup will fail on first run. to run the backup. If not, the backup will likely fail on first run.
And more pernicious, some files should sometimes be excluded from the backup Often, some files should be excluded from the backup (e.g. env files or keys)
and that's usually only found out by experience. and that's usually only found out by experience, which may happen too late.
The contract allows the maintainer of the service to encode all this, Defining a contract allows the maintainer of the service to encode all of these subtleties,
hiding this complexity from the end user. hiding this complexity from the end user.
Having a contract here means also we have a lot of freedom on the organization of the backups. Embedding this information in a contract means also we have a lot of freedom in how backups are organized.
It becomes easy to backup multiple services to multiple locations with multiple different programs like shown in this pseudo-code snippet: It becomes easy to back up multiple services to multiple locations using multiple different programs, as shown in this pseudocode snippet:
```nix ```nix
let let
@ -433,163 +414,132 @@ in
} }
``` ```
This user-defined matrix of combination is not possible now. This user-defined matrix of combinations is not currently possible;
It would require at least some heavy work it would require at least some heavy work
by the maintainers of Nextcloud and Vaultwarden. by the maintainers of Nextcloud and Vaultwarden.
The behavior test creates some files somewhere, backs them up, deletes them, restores them The behavior test creates some files somewhere, backs them up, deletes them, restores them
and finally verifies the files are correctly restored. and finally verifies the files have been restored correctly.
To do this generically, we need a way to start the backup To do this generically, we need a way to perform a backup and restore from it that is standardised across all providers.
and to restore from a backup which is standard across all providers. This is where the idea for the `output.backupService` and `output.restoreScript` options comes from.
This is where the idea for the `output.backupService` and `output.restoreScript` comes from.
Although the `consumer` does not care about those two options Although the `consumer` does not care about those two options
they can be useful to the `end user`. they can be useful to the `end user`.
They also allow to create automated backups on deploys They also allow creating automated backups on deploys,
and restoration from backups on rollbacks too. and restoring from backups on rollbacks too.
## Streaming Backup Contract ## Streaming Backup Contract
[streamingBackupContract]: #streaming-backup-contract [streamingBackupContract]: #streaming-backup-contract
Links to relevant commits: Links to relevant commits:
- [streaming backup contract: init][] - [streaming backup contract: init](https://github.com/NixOS/nixpkgs/pull/432529/commits/700919f0c121ef500b3ec31d5126bd677434c19d)
- [restic: implement streaming backup contract provider][] - [restic: implement streaming backup contract provider](https://github.com/NixOS/nixpkgs/pull/432529/commits/1d92450136106c25f1affb70817cef4bdae00c83)
- [postgresql: implement streaming backup contract consumer][] - [postgresql: implement streaming backup contract consumer](https://github.com/NixOS/nixpkgs/pull/432529/commits/2e02b68087fa36f274695911789db2d10579cc3c)
- [restic: define streaming backup contract behavior test][] - [restic: define streaming backup contract behavior test](https://github.com/NixOS/nixpkgs/pull/432529/commits/d360b941b45e5bacf0eb5b8a58825e7a51e53d4f)
[streaming backup contract: init]: https://github.com/NixOS/nixpkgs/pull/432529/commits/700919f0c121ef500b3ec31d5126bd677434c19d For databases, and possibly other use cases, there may not be files that can be backed up.
[restic: implement streaming backup contract provider]: https://github.com/NixOS/nixpkgs/pull/432529/commits/1d92450136106c25f1affb70817cef4bdae00c83
[postgresql: implement streaming backup contract consumer]: https://github.com/NixOS/nixpkgs/pull/432529/commits/2e02b68087fa36f274695911789db2d10579cc3c
[restic: define streaming backup contract behavior test]: https://github.com/NixOS/nixpkgs/pull/432529/commits/d360b941b45e5bacf0eb5b8a58825e7a51e53d4f
For databases and possibly other use cases,
there are no files laying around that can be backed up.
Instead, the backup can be read from a stream, usually on stdout of some program. Instead, the backup can be read from a stream, usually on stdout of some program.
Creating files from those streams would allow to use the `fileBackup` contract directly Creating files from those streams, and then backing them up would allow using the `fileBackup` contract directly,
but it would be incredibly wasteful in resources, if even possible. but it would be incredibly wasteful of resources, if it's even possible (e.g. it may consume excessive amounts of disk space).
This is why another contract has been created which require a different backup tactic and thus different `input` and `output` options. To address this, we can define another contract that takes a different backup approach and thus has different `input` and `output` options.
Like for the `fileBackup` contract, the test backs up a stream, As for the `fileBackup` contract, the test backs up a stream,
deletes the original resource and restores it, making sure it is correctly restored. deletes the original resource and restores it, making sure it is correctly restored.
Here though, instead of engineering a stub for a stream, we directly use Here though, instead of engineering a stub for a stream, we use
the new `streamingBackup consumer` added to `services.postgresql`. the `streamingBackup consumer` added to `services.postgresql` directly.
## Secrets Contract ## Secrets Contract
[secretsContract]: #secrets-contract [secretsContract]: #secrets-contract
Links to relevant commits: Links to relevant commits:
- [secret contract: init][] - [secret contract: init](https://github.com/NixOS/nixpkgs/pull/432529/commits/1bedf2dcf0960a4f33b7b7394aad51c4a3e436ae)
- [secret contract: declare behavior test][] - [secret contract: declare behavior test](https://github.com/NixOS/nixpkgs/pull/432529/commits/a14ec6ee6cb2205d7125dfa38f305838f8ce11ac)
[secret contract: init]: https://github.com/NixOS/nixpkgs/pull/432529/commits/1bedf2dcf0960a4f33b7b7394aad51c4a3e436ae To pass credentials to a target host for deployment,
[secret contract: declare behavior test]: https://github.com/NixOS/nixpkgs/pull/432529/commits/a14ec6ee6cb2205d7125dfa38f305838f8ce11ac the most common (as far as the author of this RFC knows) way to do this
To pass credentials to a target host we deploy to,
the most common (as far as the author of this RCF knows) way to do this
is to encrypt the secret (possibly in the nix store) is to encrypt the secret (possibly in the nix store)
and on activation decrypt it in an agreed upon location on the file system. and on activation decrypt it to an agreed-upon location on the file system.
Currently in nixpkgs, most of the modules that require one or more secrets Currently in nixpkgs, most of the modules that require one or more secrets
define a global option that accepts a file containing all the secrets define a global option that accepts a file containing all the secrets
in a given format. Usually the module uses under the hood in a given format. Usually the module uses the `systemd.services.<name>.serviceConfig.EnvironmentFile` option under the hood, using [dotenv](https://www.dotenv.org/docs/security/env.html) format. Failure to provide the file in the correct format
the `systemd.services.<name>.serviceConfig.EnvironmentFile` option
and the format is [dotenv][]. Failure to provide the file in the correct format
will result in an error at deploy time. will result in an error at deploy time.
[dotenv]: https://www.dotenv.org/docs/security/env.html
Some services go the extra mile and provide one option per secret Some services go the extra mile and provide one option per secret
and accept a path to a file that contains the raw secret like [kadmin][]'s and accept a path to a file that contains the raw secret like [kadmin](https://github.com/NixOS/nixpkgs/blob/nixos-25.05/nixos/modules/services/security/kanidm.nix)'s
`adminPasswordFile` option. They implement some machinery to transform this file `adminPasswordFile` option. They implement some machinery to transform this file
in the expected format by the upstream service. in the expected format by the upstream service.
This moves the possible failure at evaluation time which is a very nice property. This moves the possible failure at evaluation time which is a very nice property.
[kadmin]: https://github.com/NixOS/nixpkgs/blob/nixos-25.05/nixos/modules/services/security/kanidm.nix
_Aside: This is such big step forward in user experience that we would like_ _Aside: This is such big step forward in user experience that we would like_
_to see this more available. This will be tackled though in the [vars][] proposal_ _to see this more readily available. This will be tackled in the [vars](https://discourse.nixos.org/t/vars-a-framework-for-managing-secrets-and-computed-values/62411) proposal_
_and **not** in this RFC. The `vars` proposal will use the secrets contract_ _and **not** in this RFC. The `vars` proposal will use the secrets contract_
_as presented here or a slightly modified if deemed necessary._ _as presented here, or in a slightly modified if deemed necessary._
[vars]: https://discourse.nixos.org/t/vars-a-framework-for-managing-secrets-and-computed-values/62411
One problem encountered by those modules providing one option per secret One problem encountered by those modules providing one option per secret
is the file must be readable by the user of the service. is that the file must be readable by the user of the service.
This is often solved by relying on [systemd's credentials][] system This is often solved by relying on [systemd's credentials](https://systemd.io/CREDENTIALS/) system
or less securely by using the `root` user in the service startup to read from the file. or less securely by using the `root` user in the service startup to read from the file.
[systemd's credentials]: https://systemd.io/CREDENTIALS/ This contract provides an alternative where the `consumer` of the contract — the module requiring a secret — imposes a `user` on the secret `provider`, which here would be [agenix](https://github.com/ryantm/agenix) or [sops-nix](https://github.com/Mic92/sops-nix) for example.
This contract provides an alternative where the `consumer` of the contract - the module requiring a secret - imposes a `user` to the secret `provider`, which here would be [agenix][] or [sops-nix][] for example. In contrast to the previous contracts we covered, the `consumer` here needs to read the `output` of the `provider`
[agenix]: https://github.com/ryantm/agenix
[sops-nix]: https://github.com/Mic92/sops-nix
Contrary to the previous contracts we covered, the `consumer` here needs to read the `output` of the `provider`
because it contains the path to the file containing the secret. because it contains the path to the file containing the secret.
When testing a module that expects a file containing a raw secret, When testing a module that expects a file containing a raw secret,
the ubiquitous method to provide the file is using `pkgs.writeText`. the ubiquitous method to provide the file is by using `pkgs.writeText`.
This works but has the issue the created file is world readable This works, but has the issue the created file is world-readable
and we thus do not test the file is accessible with the correct user. so we do not test whether the file is accessible with the correct user.
To avoid this pitfall going forwards, we created the [`testing.hardcodedSecret` To avoid this pitfall going forwards, we created the [`testing.hardcodedSecret`
`provider`][hardcodedSecret: new secret contract consumer] `provider`](https://github.com/NixOS/nixpkgs/pull/432529/commits/6fbd099aa306d2cce337b8fa7ed7e0c8a255aebf)
which is an improved version of `pkgs.writeText` which is an improved version of `pkgs.writeText`
where the resulting file is created with the requested `owner`, `mode`, etc. where the resulting file is created with the requested `owner`, `mode`, etc.
as described by the contract `consumer`. as described by the contract `consumer`.
[hardcodedSecret: new secret contract consumer]: https://github.com/NixOS/nixpkgs/pull/432529/commits/6fbd099aa306d2cce337b8fa7ed7e0c8a255aebf This new provider has been tested using the [contract's behavior test](https://github.com/NixOS/nixpkgs/pull/432529/commits/448410a520225bc71e1616611cef7ad086c64cd1)
and has been used in [`services.stash`'s module](https://github.com/NixOS/nixpkgs/pull/432529/commits/19419ad95913fbed4636d0b24d95c80517c18340) as an example.
This new provider has been tested using the [contract's behavior test][hardcodedSecret: define behavior test for secret contract]
and has been used in [`services.stash`'s module][stash: use secret contract for passwords] as an example.
[hardcodedSecret: define behavior test for secret contract]: https://github.com/NixOS/nixpkgs/pull/432529/commits/448410a520225bc71e1616611cef7ad086c64cd1
[stash: use secret contract for passwords]: https://github.com/NixOS/nixpkgs/pull/432529/commits/19419ad95913fbed4636d0b24d95c80517c18340
# Drawbacks # Drawbacks
[drawbacks]: #drawbacks [drawbacks]: #drawbacks
We are not aware of any because this solution is fully backwards compatible, We are not aware of any because this solution is fully backwards compatible,
incremental and has a lot advantages. It also arose from a real practical need. incremental, and has many advantages. It also arose from a real practical need.
Care should be taken to not abuse this pattern though. It should be reserved Care should be taken to not abuse this pattern though. It should be reserved
for contracts where abstracting away a `consumer` and `provider` makes sense. for contracts where abstracting away a `consumer` and `provider` makes sense.
We didn't find a general rule for that but a good indication that the pattern gets abused We didn't find a general rule for that but a good indicator of an unnecessary contract is where we only find one instance of a `consumer` and `provider` pair in the whole of nixpkgs.
is if we only find one `consumer` and `provider` pair in the whole of nixpkgs.
# Alternatives # Alternatives
[alternatives]: #alternatives [alternatives]: #alternatives
This design arose from trying to maximize code reuse. This design arose from trying to maximize code reuse.
We started by fiddling with nix code and the implementation came up naturally. We started by fiddling with nix code and the implementation emerged naturally.
We are not aware of any alternatives to do this, We are not aware of any alternative ways to do this,
mostly because our attempts to tweak the code often led us often to infinite recursion or other module issues mostly because our attempts to tweak the code often led us often to infinite recursion or other module issues
so we couldn't stray too far from the way it is written now. so we couldn't stray too far from the way it already works.
# Prior art # Prior art
[prior-art]: #prior-art [prior-art]: #prior-art
We did not find any discussion about any of this by the nix community. We did not find any discussion about any of this by the nix community.
It is a bit self-centered but the two talks I (ibizaman) gave on this subject in nixpkgs can be considered prior art. It is a bit self-centered, but the two talks I (`ibizaman`) gave on this subject in nixpkgs can be considered prior art.
Note the syntax presented is a bit outdated now but the message is still relevant: Note the syntax in this presentation is outdated, but the underlying message remains the same:
- 04/2024: Scale21x in Pasadena: [Easier NixOS self-hosting with module contracts](https://www.youtube.com/watch?v=lw7PgphB9qM) - 04/2024: Scale21x in Pasadena: [Easier NixOS self-hosting with module contracts](https://www.youtube.com/watch?v=lw7PgphB9qM)
- 11/2024 at NixCon2024 in Berlin: [Enabling incremental adoption of NixOS with module contracts](https://www.youtube.com/watch?v=CP0hR6w1csc) - 11/2024 at NixCon2024 in Berlin: [Enabling incremental adoption of NixOS with module contracts](https://www.youtube.com/watch?v=CP0hR6w1csc)
A pre-RFC has been opened [on discourse][prerfc]. A pre-RFC has been opened [on discourse](https://discourse.nixos.org/t/pre-rfc-decouple-services-using-structured-typing/58257).
[prerfc]: https://discourse.nixos.org/t/pre-rfc-decouple-services-using-structured-typing/58257 A few useful comparisons beyond nixpkgs:
A few useful comparisons outside of nixpkgs are:
- Contracts are closely related to Golang interfaces with options being methods and input and output options the inputs and outputs of the methods. - Contracts are closely related to Golang interfaces with options being methods and input and output options the inputs and outputs of the methods.
The important bit is that in Golang, the saying goes "the bigger the interface, the weaker the abstraction". The important bit is that in Golang, the saying goes "the bigger the interface, the weaker the abstraction".
We should strive to keep the number of options to a minimum to make the contracts more general. We should strive to keep the number of options to a minimum to make the contracts more general.
- Contracts are reminiscent of the [reverse dependency principle](https://en.wikipedia.org/wiki/Dependency_inversion_principle) which is used in a lot of places. - Contracts are reminiscent of the [reverse dependency principle](https://en.wikipedia.org/wiki/Dependency_inversion_principle) which is used in many places.
# Unresolved questions # Unresolved questions
[unresolved]: #unresolved-questions [unresolved]: #unresolved-questions
@ -610,28 +560,28 @@ config = {
}; };
``` ```
It would be so much nicer if we could somehow require only to give the `consumer` to the `provider` It would be so much nicer if we could somehow require specifying only the `consumer` to the `provider`,
and it managed to make the link backwards automatically. and it managed to make the reciprocal link automatically.
In the snippet above, this means removing the need for the `provider to consumer` line. In the snippet above, this would remove the need for the `provider to consumer` line.
The issue comes from the `consumer` and `provider` option in the top-level `contracts` definition to be of type `optionType`. The issue comes from the `consumer` and `provider` option in the top-level `contracts` definition to be of type `optionType`.
They don't have access to the actual `input` and `output` values of an instantiated contract. They don't have access to the actual `input` and `output` values of an instantiated contract.
Experimenting on this has been done in the [module interfaces][] repo. There are some experiments on this in the [module interfaces][] repo.
There, we set the `provider` option as a function which takes an argument There, we set the `provider` option as a function which takes an argument
which is the instantiated `consumer`, so it is not of type `optionType` but of type `submodule` and has access to the real input and output values. which is the instantiated `consumer`, so it is not of type `optionType` but of type `submodule`, and has access to the real input and output values.
Unfortunately, this has two downsides: Unfortunately, this has two downsides:
1. It requires one more line in each provider definition. This would be okay if there wasn't the following downside. 1. It requires one more line in each provider definition. This would be okay except for the following downside:
2. There's no way to write side effects. This means the `provider` can only write to its own `output`, which misses the whole point of having contracts in the first place. 2. There's no way to write side effects. This means the `provider` can only write to its own `output`, which misses the whole point of having contracts in the first place.
There's maybe a way to solve this but we didn't figure it out. Help is appreciated! There may be a way to solve this, but we have not yet figured it out. Help would be appreciated!
Beware though you will be crossing the edge of the module system and entering the land of infinite recursion. Beware though; you will be crossing the edge of the module system and entering the land of infinite recursion.
## Documentation ## Documentation
[unresolved-documentation]: #unresolved-questions-documentation [unresolved-documentation]: #unresolved-questions-documentation
It is not possible to build the manual right now. Doing so results in an error. It is not currently possible to build the manual; doing so results in an error:
```bash ```bash
$ (cd nixos/; nix-build release.nix -A manual.x86_64-linux) $ (cd nixos/; nix-build release.nix -A manual.x86_64-linux)
@ -647,12 +597,12 @@ $ (cd nixos/; nix-build release.nix -A manual.x86_64-linux)
``` ```
Comments in the [draft PR][draftPR] have been added to indicate what has been tried. Comments in the [draft PR][draftPR] have been added to indicate what has been tried.
Any help is appreciated to solve this. We would appreciate help in solving this.
# Future work # Future work
[future]: #future-work [future]: #future-work
- Solve the [documentation][unresolved-documentation] issue. - Solve the [documentation][unresolved-documentation] issue.
- Identify contracts and their inputs, outputs and behavior tests. - Identify useful contracts and their inputs, outputs, and behavior tests.
- Identify services that would benefit from being consumers and providers of contracts and add the necessary options. - Identify services that would benefit from being consumers and providers of contracts and add the necessary options.
- Optionally solve the [dual-link][unresolved-duallink] issue. - Optionally solve the [dual-link][unresolved-duallink] issue.