1
1
Fork 0
mirror of https://github.com/NixOS/nix.git synced 2025-11-11 21:16:02 +01:00
Commit graph

13074 commits

Author SHA1 Message Date
John Ericson
af71a9dbd9 Fix JSON_IMPL macro to avoid extraneous copies
Should take the thing we're serializing by reference.
2025-09-23 13:05:12 -04:00
Jörg Thalheim
af82c847a7
Merge pull request #14048 from roberth/shellcheck
Shellcheck
2025-09-22 21:57:42 +02:00
John Ericson
5292b0e49e
Merge pull request #14038 from NixOS/thread-safe-dummy
libstore: Make writable dummy store thread-safe
2025-09-22 14:26:35 -04:00
Robert Hensing
5af644492b nix develop: Apply shellcheck 2025-09-22 19:31:22 +02:00
Sergei Zimmerman
5915fe3190
Revert "Use shared pointers in the memory source accessor"
This is no longer necessary.

This reverts commit 4df60e639b.
2025-09-22 20:22:17 +03:00
Sergei Zimmerman
c4c92c4c61
libstore: Make writable dummy store thread-safe
Tested by building with b_sanitize=thread and running:

nix flake prefetch-inputs --store "dummy://?read-only=false"

It might make sense to move this utility class out of dummy-store.cc,
but it seems fine for now.
2025-09-22 20:22:16 +03:00
Sergei Zimmerman
c71f80b6eb
libstore: Implement boost::hash for StorePath 2025-09-22 20:16:11 +03:00
John Ericson
91593a237f Convert Realisation JSON logic to standard style
No behavior is changed, just:

- Declare a canonical `nlohmnan::json::adl_serializer`

- Use `json-utils.hh` to shorten code without getting worse error
  messages.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2025-09-22 12:59:37 -04:00
Robert Hensing
169a368459
Merge pull request #14040 from NixOS/import-thunk
Ensure that files are parsed/evaluated only once (2nd attempt)
2025-09-22 18:33:00 +02:00
Eelco Dolstra
a0103fc302
Merge pull request #13852 from lovesegfault/warn-no-kvm
feat(libstore): warn when kvm is enabled but /dev/kvm isn't available
2025-09-22 18:31:57 +02:00
David McFarland
32d4ea8140 fix cross-build for cygwin 2025-09-22 12:27:04 -03:00
Eelco Dolstra
d32d77f4d4 Allocate ExprParseFile on the heap for now
https://github.com/NixOS/nix/pull/14013#issuecomment-3308085755
2025-09-22 12:06:51 +02:00
Eelco Dolstra
5f60602875 Reapply "Merge pull request #13938 from NixOS/import-thunk"
This reverts commit fd034814dc.
2025-09-22 11:48:58 +02:00
Robert Hensing
71b27774f0
libexpr: Document {eval,maybeThunk} methods 2025-09-22 01:41:02 +03:00
Robert Hensing
c121c65640 C API: Clarify valid use of bindings ordering 2025-09-21 21:16:46 +02:00
Sergei Zimmerman
b66c357b58
libstore: Implement DummyStore::narFromPath 2025-09-21 13:19:50 +03:00
Sergei Zimmerman
3a4c618483
libstore: Fix typo in description of dummy store 2025-09-21 13:19:49 +03:00
Sergei Zimmerman
ed9b377928
libstore: Disable path info cache for dummy store 2025-09-21 13:19:48 +03:00
Sergei Zimmerman
341878ce0f
libstore: Make dummy store also store path info 2025-09-21 13:19:47 +03:00
Sergei Zimmerman
02c9ac445f
libutil: Improve handling of non-directory root in MemorySourceAccessor 2025-09-21 13:19:45 +03:00
John Ericson
4df60e639b
Use shared pointers in the memory source accessor
This allows aliasing, like hard links.
2025-09-21 13:19:44 +03:00
Sergei Zimmerman
e04381edbd
libfetchers/github: Use getFSAccessor for downloadFile result
We should use proper abstractions for reading files from the store.
E.g. this caused errors when trying to download github flakes into
an in-memory store in #14023.
2025-09-21 01:12:42 +03:00
Robert Hensing
d0b1caf53a C API: Document and verify NIX_ERR_KEY behavior 2025-09-20 00:13:50 +02:00
Robert Hensing
2d1b412e5b libexpr-tests: Enable when test setup for building succeeds
Accidentally disabled by 9bc218ca3f
2025-09-19 23:39:00 +02:00
Robert Hensing
3d777eb37f C API: Add lazy attribute value and list item accessors 2025-09-19 23:39:00 +02:00
Robert Hensing
7c553a30a9 C API: Improve nix_get_attr_name_byidx() doc 2025-09-19 23:39:00 +02:00
Robert Hensing
0e74b25f62 C API: Fix bounds checking in _byidx functions
The docs weren't 100% clear about bounds checking, but suggested that
errors would be caught.
The bounds checks are cheap compared to the function calls they're in,
so we have no reason to omit them.
2025-09-19 23:39:00 +02:00
Bernardo Meurer Costa
b63d9fbc87
test(libstore): additional ParsedS3Url tests
Extracted from the work in #13752
2025-09-19 20:52:44 +00:00
John Ericson
936334638d
Merge pull request #14020 from xokdvium/dummy-store-fixes
libstore: Set display prefix for dummy store
2025-09-19 00:00:00 -04:00
Sergei Zimmerman
72e2b0efea
libstore: Set display prefix for dummy store
Otherwise the prefix is «unknown».
2025-09-19 02:28:59 +03:00
Sergei Zimmerman
94d37e62fc
treewide: Support builds with ASAN, enable in CI
Enables builds with ASAN to catch memory corruption
bugs faster and in CI. This is an incredibly valuable
instrument that must be used as much as possible.

Somewhat based on jade's work from Lix, though there's a lot that
we have to do differently:

19ae87e5ce

Co-authored-by: Jade Lovelace <lix@jade.fyi>
2025-09-19 01:33:57 +03:00
Jörg Thalheim
4808b0e24f
Merge pull request #14014 from xokdvium/fish delete
libexpr-c: Fix mismatched new/delete
2025-09-18 08:08:39 +02:00
Sergei Zimmerman
309d55807c
libexpr-c: Fix mismatched new/delete
This leads to ASAN errors:

==1137785==ERROR: AddressSanitizer: new-delete-type-mismatch on 0x523000001d00 in thread T0:
  object passed to delete has wrong type:
  size of the allocated type:   5968 bytes;
  size of the deallocated type: 5968 bytes.
  alignment of the allocated type:   8 bytes;
  alignment of the deallocated type: default-aligned.
2025-09-18 02:58:32 +03:00
Sergei Zimmerman
fd034814dc
Revert "Merge pull request #13938 from NixOS/import-thunk"
This has multiple dangling pointer issues that lead to segfaults in e.g.:

nix eval --expr '(builtins.getFlake "github:nixos/nixpkgs/25.05")' --impure

This reverts commit ad175727e4, reversing
changes made to d314750174.
2025-09-18 01:52:46 +03:00
Robert Hensing
b4fcb64276
Merge pull request #13980 from obsidiansystems/drv-json-issue-13570
Make the JSON format for derivation use basename store paths
2025-09-17 23:31:41 +02:00
Sergei Zimmerman
6138bc3de3
libexpr: Structural sharing of attrsets
This changes the implementation of Bindings to allow
for a more space-efficient implementation of attribute
set merges. This is accomplished by "layering" over the "base" Bindings.
The top "layer" is naturally the right-hand-side of the update operator //.

Such an implementation leads to significantly better memory usage on
something like nixpkgs:

nix-env --query --available --out-path --file ../nixpkgs --eval-system x86_64-linux > /dev/null

Comparison against 2b0fd88324 for x86_64-linux on nixpkgs f06c7c3b6f5074dbffcf02542fb86af3a5526afa:

| metric                 | mean_before     | mean_after      | mean_diff       | mean_%_change | p_value | t_stat  |
| -                      | -               | -               | -               | -             | -       | -       |
| cpuTime                | 21.1520         | 21.3414         | 0.1894          | 0.7784        | 0.3190  | 1.0219  |
| envs.bytes             | 461451951.6190  | 461451951.6190  | -               | -             | -       | -       |
| envs.elements          | 34344544.8571   | 34344544.8571   | -               | -             | -       | -       |
| envs.number            | 23336949.0952   | 23336949.0952   | -               | -             | -       | -       |
| gc.cycles              | 7.5238          | 7.2857          | -0.2381         | -4.6825       | 0.0565  | -2.0244 |
| gc.heapSize            | 1777848124.9524 | 1252162023.6190 | -525686101.3333 | -29.9472      | 0.0000  | -8.7041 |
| gc.totalBytes          | 3102787383.6190 | 2498431578.6667 | -604355804.9524 | -19.7704      | 0.0000  | -9.3502 |
| list.bytes             | 59928225.9048   | 59928225.9048   | -               | -             | -       | -       |
| list.concats           | 1240028.2857    | 1240028.2857    | -               | -             | -       | -       |
| list.elements          | 7491028.2381    | 7491028.2381    | -               | -             | -       | -       |
| nrAvoided              | 28165342.2381   | 28165342.2381   | -               | -             | -       | -       |
| nrExprs                | 1577412.9524    | 1577412.9524    | -               | -             | -       | -       |
| nrFunctionCalls        | 20970743.4286   | 20970743.4286   | -               | -             | -       | -       |
| nrLookups              | 10867306.0952   | 10867306.0952   | -               | -             | -       | -       |
| nrOpUpdateValuesCopied | 61206062.0000   | 25748169.5238   | -35457892.4762  | -58.8145      | 0.0000  | -8.9189 |
| nrOpUpdates            | 2167097.4286    | 2167097.4286    | -               | -             | -       | -       |
| nrPrimOpCalls          | 12337423.4286   | 12337423.4286   | -               | -             | -       | -       |
| nrThunks               | 29361806.7619   | 29361806.7619   | -               | -             | -       | -       |
| sets.bytes             | 1393822818.6667 | 897587655.2381  | -496235163.4286 | -36.7168      | 0.0000  | -9.1115 |
| sets.elements          | 84504465.3333   | 48270845.9524   | -36233619.3810  | -43.8698      | 0.0000  | -8.9181 |
| sets.number            | 5218921.6667    | 5218921.6667    | -               | -             | -       | -       |
| sizes.Attr             | 16.0000         | 16.0000         | -               | -             | -       | -       |
| sizes.Bindings         | 8.0000          | 24.0000         | 16.0000         | 200.0000      | -       | inf     |
| sizes.Env              | 8.0000          | 8.0000          | -               | -             | -       | -       |
| sizes.Value            | 16.0000         | 16.0000         | -               | -             | -       | -       |
| symbols.bytes          | 1368494.0952    | 1368494.0952    | -               | -             | -       | -       |
| symbols.number         | 109147.1905     | 109147.1905     | -               | -             | -       | -       |
| time.cpu               | 21.1520         | 21.3414         | 0.1894          | 0.7784        | 0.3190  | 1.0219  |
| time.gc                | 1.6011          | 0.8508          | -0.7503         | -37.1507      | 0.0017  | -3.6328 |
| time.gcFraction        | 0.0849          | 0.0399          | -0.0450         | -37.4504      | 0.0035  | -3.3116 |
| values.bytes           | 615968144.7619  | 615968144.7619  | -               | -             | -       | -       |
| values.number          | 38498009.0476   | 38498009.0476   | -               | -             | -       | -       |

Overall this does slow down the evaluator slightly (no more than ~10% in most cases),
but this seems like a very decent tradeoff for shaving off 33% of memory usage.
2025-09-17 23:54:45 +03:00
John Ericson
9d7229a2a4 Make the JSON format for derivation use basename store paths
See #13570 for details --- the idea is that included the store dir in
store paths makes systematic JSON parting with e.g. Serde, Aeson,
nlohmann, or similiar harder.

After talking to Eelco, we are changing the `Derivation` format right
away because not only is `nix derivation` technically experimental, we think it is
also less widely used in practice than, say, `nix path-info`.

Progress on #13570
2025-09-17 16:38:17 -04:00
John Ericson
d5ce8c3caa Use MemorySourceAccessor in DummyStore
Add `read-only` setting to `dummy://` store for back compat.

Test by changing an existing test to use this instead, fixing a TODO.

Co-Authored-By: HaeNoe <git@haenoe.party>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2025-09-17 16:09:53 -04:00
John Ericson
168c24b605 Declare DummyStoreConfig in a header
This will useful for unit tests.
2025-09-17 15:54:30 -04:00
John Ericson
613de9d9cc Add missing #pragma once 2025-09-17 15:21:21 -04:00
John Ericson
9bc218ca3f Add new C API for working with derivations
Also test the APIs we just added.
2025-09-16 13:25:36 -04:00
Jens Petersen
86bb7c958a meson: refactor nix_soversion into nix-meson-build-support/common
This is a follow-on to #13995 which added soversion to the libraries
2025-09-16 12:54:30 +08:00
Jörg Thalheim
5e17a3f81c
Merge pull request #13998 from Mic92/fast-flake-check
nix flake check: Skip substitutable derivations
2025-09-15 21:44:11 +02:00
Seth Flynn
ecdda5798c nix flake check: Skip substitutable derivations
Since `nix flake check` doesn't produce a `result` symlink, it doesn't
actually need to build/substitute derivations that are already known
to have succeeded, i.e. that are substitutable.

This can speed up CI jobs in cases where the derivations have already
been built by other jobs. For instance, a command like

  nix flake check github:NixOS/hydra/aa62c7f7db31753f0cde690f8654dd1907fc0ce2

should no longer build anything because the outputs are already in
cache.nixos.org.

Based-on: https://github.com/DeterminateSystems/nix-src/pull/134
Based-on: https://gerrit.lix.systems/c/lix/+/3841
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2025-09-15 21:31:03 +02:00
Jörg Thalheim
0b401e2199
Merge pull request #13995 from juhp/soversion
meson: add soversion with nix version to give SONAME to libs
2025-09-15 20:36:31 +02:00
Eelco Dolstra
ad175727e4
Merge pull request #13938 from NixOS/import-thunk
Ensure that files are parsed/evaluated only once
2025-09-15 19:03:18 +02:00
John Ericson
f3e3f75838 More get / getOr improvements
- Use `const K`, not `K`, otherwise we don't get auto referencing of
  rvalues.

- Generalized the deleted overloads, because we don't care what the key
  type is --- we want to get rid of anything that has an rvalue map
  type.
2025-09-15 12:25:37 -04:00
Jens Petersen
dd1a554aba meson: add soversion with nix version to give SONAME to libs (#13960, #13979)
remove 'pre' version suffix for non-releases (chokes Darwin ld)
2025-09-15 22:08:26 +08:00
Sergei Zimmerman
2b0fd88324
Merge pull request #13991 from xokdvium/bindings-remove-find
libexpr: Remove Bindings::find
2025-09-14 21:32:31 +00:00
John Ericson
ffc14ac91b
Merge pull request #13983 from xokdvium/bindings-fixes
libexpr: Make Bindings::iterator a proper strong type instead of pointer
2025-09-14 17:31:48 -04:00