mirror of
https://github.com/NixOS/rfcs.git
synced 2025-11-08 19:46:12 +01:00
[RFC 0146] Meta.Categories, not Filesystem Directory Trees (#146)
* Meta.Categories, not Filesystem Directory Trees * Whitespace cleanup * Add a short answer to the bikeshedding problem * Add a short line on "Do nothing" alternative * Extend an answer for the "Ignore/nuke" alternative * Add "update ci" to future work * Add a repl-like interaction example * Add more arguments for categorization and against its nuking * small rewording * Add an option of category data structure * reorder arguments against nuking * add argument for usefulness of categorization * add drawback * rework nuke argument * update metainfo - shepherd team and leader * Add prior art section Also, add extra links * typo * Categorization Team * Remove the optional data structure * typo * reword the creation of a team * Debtags FAQ * Update rules and duties of categorization team * The team shall have authority to carry out their duties * A section for the team * Appstream as prior art * Section for code implementation * Move categorization team to implementation * Update future work * A hybrid approach to be considered by the future team * extra duties for the team * reword duties from team * typo * A semantic detail: treat the first element of meta.categories as most important * Move hybrid approach to alternatives section * identify AndersonTorres' tag * Suggestions from FCP * remove infinisil from shepherd team * add an extra reference to the categorization team
This commit is contained in:
Anderson Torres
GitHub
parent
6499fd85b4
commit
62d1245ec1
1 changed files with 356 additions and 0 deletions
356
rfcs/0146-meta-categories.md
Executable file
356
rfcs/0146-meta-categories.md
Executable file
|
|
@ -0,0 +1,356 @@
|
||||||
|
---
|
||||||
|
feature: Decouple filesystem from categorization
|
||||||
|
start-date: 2023-04-23
|
||||||
|
author: Anderson Torres (@AndersonTorres)
|
||||||
|
co-authors:
|
||||||
|
shepherd-team: @7c6f434c @natsukium @fgaz
|
||||||
|
shepherd-leader: @7c6f434c
|
||||||
|
related-issues: (will contain links to implementation PRs)
|
||||||
|
---
|
||||||
|
|
||||||
|
# Summary
|
||||||
|
[summary]: #summary
|
||||||
|
|
||||||
|
Deploy a new method of categorization for the packages maintained by Nixpkgs,
|
||||||
|
not relying on filesystem idiosyncrasies.
|
||||||
|
|
||||||
|
# Motivation
|
||||||
|
[motivation]: #motivation
|
||||||
|
|
||||||
|
Currently, Nixpkgs uses the filesystem, or more accurately, the directory tree
|
||||||
|
layout in order to informally categorize the softwares it packages, as described
|
||||||
|
in the [Hierarchy](https://nixos.org/manual/nixpkgs/stable/#sec-hierarchy)
|
||||||
|
section of Nixpkgs manual.
|
||||||
|
|
||||||
|
This is a simple, easy to understand and consecrated-by-use method of
|
||||||
|
categorization, partially employed by many other package managers like GNU Guix
|
||||||
|
and NetBSD pkgsrc.
|
||||||
|
|
||||||
|
However this system of categorization has serious problems:
|
||||||
|
|
||||||
|
1. It is bounded by the constraints imposed by the filesystem.
|
||||||
|
|
||||||
|
- Restrictions on filenames, subdirectory tree depth, permissions, inodes,
|
||||||
|
quotas, and many other things.
|
||||||
|
- Some of these restrictions are not well documented and are found simply
|
||||||
|
by "bumping" on them.
|
||||||
|
- The restrictions can vary on an implementation basis.
|
||||||
|
- Some filesystems have more restrictions or less features than others,
|
||||||
|
forcing an uncomfortable lowest common denominator.
|
||||||
|
- Some operating systems can impose additional constraints over otherwise
|
||||||
|
full-featured filesystems because of backwards compatibility (8 dot
|
||||||
|
3, anyone?).
|
||||||
|
|
||||||
|
2. It requires a local checkout of the tree.
|
||||||
|
|
||||||
|
Certainly this checkout can be "cached" using some form of `find . >
|
||||||
|
/tmp/pkgs-listing.txt`, or more sophisticated solutions like `locate +
|
||||||
|
updatedb`. Nonetheless such solutions still require access to a fresh,
|
||||||
|
updated copy of the Nixpkgs tree.
|
||||||
|
|
||||||
|
3. The creation of a new category - and more generally the manipulation of
|
||||||
|
categories - requires an unpleaseant task of renaming and eventually patching
|
||||||
|
many seemingly unrelated files.
|
||||||
|
|
||||||
|
- Moving files around Nixpkgs codebase requires updating their forward and
|
||||||
|
backward references.
|
||||||
|
- Especially in some auxiliary tools like editor plugins, testing suites,
|
||||||
|
autoupdate scripts and so on.
|
||||||
|
- Rewriting `all-packages.nix` can be error-prone (even using Metapad) and it
|
||||||
|
can generate huge, noisy patches.
|
||||||
|
|
||||||
|
4. There is no convenient way to use multivalued categorization.
|
||||||
|
|
||||||
|
A piece of software can fulfill many categories; e.g.
|
||||||
|
- an educational game
|
||||||
|
- a console emulator (vs. a PC emulator)
|
||||||
|
- and a special-purpose programming language (say, a smart-contracts one).
|
||||||
|
|
||||||
|
The current one-size-fits-all restriction is artificial, imposes unreasonable
|
||||||
|
limitations and results in incomplete and confusing information.
|
||||||
|
|
||||||
|
- No, symlinks or hardlinks are not convenient for this purpose; not all
|
||||||
|
environments support them (falling on the "less features than others"
|
||||||
|
problem expressed before) and they convey nothing besides confusion - just
|
||||||
|
think about writing the corresponding entry in `all-packages.nix`.
|
||||||
|
|
||||||
|
5. It puts over the (possibly human) package writer the mental load of where to
|
||||||
|
put the files on the filesystem hierarchy, deviating them from the job of
|
||||||
|
really writing them.
|
||||||
|
|
||||||
|
- Or just taking the shortest path and throw it on a folder under `misc`.
|
||||||
|
|
||||||
|
6. It "locks" the filesystem, preventing its usage for other, more sensible
|
||||||
|
purposes.
|
||||||
|
|
||||||
|
7. The most important: the categorization is not discoverable via Nix language
|
||||||
|
infrastructure.
|
||||||
|
|
||||||
|
Indeed there is no higher level way to query about such categories besides
|
||||||
|
the one described in the bullet 2 above.
|
||||||
|
|
||||||
|
In light of such a bunch of problems, this RFC proposes a novel alternative to
|
||||||
|
the above mess: new `meta` attributes.
|
||||||
|
|
||||||
|
# Detailed design
|
||||||
|
[design]: #detailed-design
|
||||||
|
|
||||||
|
## Code Implementation
|
||||||
|
[code-implementation]: #code-implementation
|
||||||
|
|
||||||
|
A new attribute, `meta.categories`, will be included for every Nix expression
|
||||||
|
living inside Nixpkgs.
|
||||||
|
|
||||||
|
This attribute will be a list, whose elements are one of the possible elements
|
||||||
|
of the `lib.categories` set.
|
||||||
|
|
||||||
|
A typical snippet of `lib.categories` will be similar to:
|
||||||
|
|
||||||
|
```nix
|
||||||
|
{
|
||||||
|
assembler = {
|
||||||
|
name = "Assembler";
|
||||||
|
description = ''
|
||||||
|
A program that converts text written in assembly language to binary code.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
compiler = {
|
||||||
|
name = "Compiler";
|
||||||
|
description = ''
|
||||||
|
A program that converts a source from a language to another, usually from
|
||||||
|
a higher, human-readable level to a lower, machine level.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
font = {
|
||||||
|
name = "Font";
|
||||||
|
description = ''
|
||||||
|
A set of files that defines a set of graphically-related glyphs.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
game = {
|
||||||
|
name = "Game";
|
||||||
|
description = ''
|
||||||
|
A program developed with entertainment in mind.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
interpreter = {
|
||||||
|
name = "Interpreter";
|
||||||
|
description = ''
|
||||||
|
A program that directly executes instructions written in a programming
|
||||||
|
language, without requiring compilation into the native machine language.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
### Semantic Details
|
||||||
|
[semantic-details]: #semantic-details
|
||||||
|
|
||||||
|
Given that `meta.categories` is implemented as a list, it is interesting to
|
||||||
|
treat the first element of this list as the "most important" categorization, the
|
||||||
|
one that mostly identifies with the software being classified.
|
||||||
|
|
||||||
|
## Categorization Team
|
||||||
|
[categorization-team]: #categorization-team
|
||||||
|
|
||||||
|
Given the typical complexities that arise from categorization, and expecting
|
||||||
|
that regular maintainers are not expected to understand its minuteness
|
||||||
|
(according to the experience from [Debtags
|
||||||
|
Team](https://wiki.debian.org/Debtags/FAQ#Why_don.27t_you_just_ask_the_maintainers_to_tag_their_own_packages.3F)),
|
||||||
|
it is strongly recommended the creation of a team entrusted with authority to
|
||||||
|
manage issues related to categorization and carry their corresponding duties.
|
||||||
|
|
||||||
|
# Examples and Interactions
|
||||||
|
[examples-and-interactions]: #examples-and-interactions
|
||||||
|
|
||||||
|
In file bochs/default.nix:
|
||||||
|
|
||||||
|
```nix
|
||||||
|
stdenv.mkDerivation {
|
||||||
|
|
||||||
|
. . .
|
||||||
|
|
||||||
|
meta = {
|
||||||
|
. . .
|
||||||
|
categories = with lib.categories; [ emulator debugger ];
|
||||||
|
. . .
|
||||||
|
};
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
In a `nix repl`:
|
||||||
|
|
||||||
|
```
|
||||||
|
nix-repl> :l <nixpkgs>
|
||||||
|
Added XXXXXX variables.
|
||||||
|
|
||||||
|
nix-repl> pkgs.bochs.meta.categories
|
||||||
|
[ { ... } ]
|
||||||
|
|
||||||
|
nix-repl> map (z: z.name) pkgs.bochs.meta.categories
|
||||||
|
[ "debugger" "emulator" ]
|
||||||
|
```
|
||||||
|
|
||||||
|
# Drawbacks
|
||||||
|
[drawbacks]: #drawbacks
|
||||||
|
|
||||||
|
The most immediate drawbacks are:
|
||||||
|
|
||||||
|
1. A huge treewide edit of Nixpkgs
|
||||||
|
|
||||||
|
On the other hand, this is easily sprintable and amenable to automation.
|
||||||
|
|
||||||
|
2. Bikeshedding
|
||||||
|
|
||||||
|
How many and which categories we should create? Can we expand them later?
|
||||||
|
|
||||||
|
For start, we can follow/take inspiration from many of the already existing
|
||||||
|
categories sets and add extra ones when the needs arise. Indeed, it is way
|
||||||
|
easier to create such categories using Nix language when compared to other
|
||||||
|
software collections.
|
||||||
|
|
||||||
|
Further, the creation of a categorization team can resolve those litigations.
|
||||||
|
|
||||||
|
3. Superfluous
|
||||||
|
|
||||||
|
It can be argued that there are other ways to discover similar or related
|
||||||
|
package sets, like Repology.
|
||||||
|
|
||||||
|
However, this argument is a bit circular, because e.g. the classification
|
||||||
|
shown by Repology effectively replicates the classification done by the many
|
||||||
|
software collections in its catalog. Therefore, relying in Repology merely
|
||||||
|
transfers the question to external sources.
|
||||||
|
|
||||||
|
Further it becomes more pronounced when we take into account the fact Nixpkgs
|
||||||
|
is top 1 of most Repology statistics. The expected outcome, therefore, should
|
||||||
|
be precisely the opposite: Nixpkgs being _the_ source of structured metainfo
|
||||||
|
for other software collections.
|
||||||
|
|
||||||
|
# Alternatives
|
||||||
|
[alternatives]: #alternatives
|
||||||
|
|
||||||
|
1. Do nothing
|
||||||
|
|
||||||
|
This will exacerbate the problems already listed.
|
||||||
|
|
||||||
|
2. Ignore/nuke the categorization completely
|
||||||
|
|
||||||
|
This is an alternative worthy of some consideration. After all,
|
||||||
|
categorization is not without its problems, as shown above. Removing or
|
||||||
|
ignoring classification removes all problems.
|
||||||
|
|
||||||
|
However, there are good reasons to keep the categorization:
|
||||||
|
|
||||||
|
- The complete removal of categorization is too harsh. A solution that keeps
|
||||||
|
and enhances the categorization is way more preferrable than one that nukes
|
||||||
|
it completely.
|
||||||
|
|
||||||
|
- As said before, the categorization is already present; this RFC proposes to
|
||||||
|
expose it to a higher level, in a structured, more discoverable format.
|
||||||
|
|
||||||
|
- Categorization is very traditional among software collections. Many of them
|
||||||
|
are doing this just fine for years on end, and Nixpkgs can imitate them
|
||||||
|
easily - and even surpass them, given the benefits of Nix language
|
||||||
|
machinery.
|
||||||
|
|
||||||
|
- Categorization is useful in many scenarios and use cases - indeed they
|
||||||
|
are ubiquitous in software world:
|
||||||
|
- specialized search engines (from Repology to MELPA)
|
||||||
|
- code forges, from Sourceforge to Gitlab
|
||||||
|
- as said above, software collections from pkgsrc to slackbuilds
|
||||||
|
- organization and preservation (as Software Heritage)
|
||||||
|
|
||||||
|
3. Debtags/Appstream hybrid approach
|
||||||
|
|
||||||
|
A hybrid approach for code implementation would be implement two meta
|
||||||
|
attributes, namely
|
||||||
|
|
||||||
|
- `meta.categories` for Appstream-based categories
|
||||||
|
- the corresponding `lib.categories` should follow Appstream closely, with
|
||||||
|
few room to custom/extra categories
|
||||||
|
- `meta.tags` for Debtags-like tags
|
||||||
|
- while being inspired from the venerable Debtags work, the corresponding
|
||||||
|
`lib.tags` is completely free to modify and even divert from Debtags,
|
||||||
|
following its own way
|
||||||
|
- generally speaking, `lib.tags` should be less bureaucratic than
|
||||||
|
`lib.categories`
|
||||||
|
|
||||||
|
However, this approach arguably elevates the complexity of the whole work, and
|
||||||
|
adds too much redundancy.
|
||||||
|
|
||||||
|
# Prior art
|
||||||
|
[prior-art]: #prior-art
|
||||||
|
|
||||||
|
As said above, categorization is very traditional among software collections. It
|
||||||
|
is not hard to cite examples in this arena; the most interesting ones I have
|
||||||
|
found are listed below (linked at [references section](#references)):
|
||||||
|
|
||||||
|
- FreeBSD Ports;
|
||||||
|
- Debtags;
|
||||||
|
- Appstream Project;
|
||||||
|
|
||||||
|
# Unresolved questions
|
||||||
|
[unresolved]: #unresolved-questions
|
||||||
|
|
||||||
|
There are remaining issues to be solved by the categorization team:
|
||||||
|
|
||||||
|
- What data structure is suitable to represent a category?
|
||||||
|
- For now we stick to the most natural: a set `{ name, description }`.
|
||||||
|
|
||||||
|
- Should we have a set of primary, "most important" categories with mandatory
|
||||||
|
status, in the sense each package should set at least one of them?
|
||||||
|
- The answer is most certainly positive.
|
||||||
|
|
||||||
|
# Future work
|
||||||
|
[future]: #future-work
|
||||||
|
|
||||||
|
- Create the [categorization team](#categorization-team)
|
||||||
|
- Carry out the duties correlated to categorization, including but not limited
|
||||||
|
to:
|
||||||
|
|
||||||
|
- Decide between possibilities of implementation;
|
||||||
|
- Documentation updates;
|
||||||
|
- Category curation, integration and updates;
|
||||||
|
- Continuous Integration updates and adaptations;
|
||||||
|
- Coordinaton of efforts to import, integrate and update categorization of
|
||||||
|
packages;
|
||||||
|
- Litigations and disputations:
|
||||||
|
- Solve them, especially in corner cases;
|
||||||
|
- Enforce implementation issues
|
||||||
|
- Decide when a CI check should be converted to block
|
||||||
|
- Grace periods
|
||||||
|
|
||||||
|
# References
|
||||||
|
[references]: #references
|
||||||
|
|
||||||
|
- [Desktop Menu
|
||||||
|
Specification](https://specifications.freedesktop.org/menu-spec/latest/);
|
||||||
|
specifically,
|
||||||
|
- [Main
|
||||||
|
categories](https://specifications.freedesktop.org/menu-spec/latest/apa.html)
|
||||||
|
- [Additional
|
||||||
|
categories](https://specifications.freedesktop.org/menu-spec/latest/apas02.html)
|
||||||
|
- [Reserved
|
||||||
|
categories](https://specifications.freedesktop.org/menu-spec/latest/apas03.html)
|
||||||
|
|
||||||
|
- [Appstream](https://www.freedesktop.org/wiki/Distributions/AppStream/)
|
||||||
|
|
||||||
|
- [Debtags](https://wiki.debian.org/Debtags)
|
||||||
|
|
||||||
|
- [Debtags FAQ](https://wiki.debian.org/Debtags/FAQ)
|
||||||
|
|
||||||
|
- [NetBSD pkgsrc guide](https://www.netbsd.org/docs/pkgsrc/)
|
||||||
|
- Especially, [Chapter 12, Section
|
||||||
|
1](https://www.netbsd.org/docs/pkgsrc/components.html#components.Makefile)
|
||||||
|
contains a short list of CATEGORIES.
|
||||||
|
|
||||||
|
- [FreeBSD Porters
|
||||||
|
Handbook](https://docs.freebsd.org/en/books/porters-handbook/)
|
||||||
|
- Especially
|
||||||
|
[Categories](https://docs.freebsd.org/en/books/porters-handbook/makefiles/#porting-categories)
|
||||||
Loading…
Add table
Add a link
Reference in a new issue