1
1
Fork 0
mirror of https://github.com/NixOS/nix.git synced 2025-11-08 19:46:02 +01:00
Commit graph

21006 commits

Author SHA1 Message Date
Robert Hensing
aace1fb5d6 C API: test nix_store_get_fs_closure 2025-10-15 13:27:09 +02:00
Tristan Ross
9abcc68ad1
libstore-c: add nix_store_get_fs_closure 2025-10-06 09:12:02 -07:00
Jörg Thalheim
07b96c1d14
Merge pull request #14018 from NixOS/fix-soname-for-real
meson: Fix SONAME for unstable versions
2025-09-19 13:01:04 +02:00
John Ericson
936334638d
Merge pull request #14020 from xokdvium/dummy-store-fixes
libstore: Set display prefix for dummy store
2025-09-19 00:00:00 -04:00
John Ericson
8719c2e6b7
Merge pull request #14016 from xokdvium/asan
treewide: Support builds with ASAN, enable in CI
2025-09-18 23:48:23 -04:00
Sergei Zimmerman
72e2b0efea
libstore: Set display prefix for dummy store
Otherwise the prefix is «unknown».
2025-09-19 02:28:59 +03:00
Sergei Zimmerman
d5e84383d1
doc: Document building with sanitizers 2025-09-19 01:37:15 +03:00
Sergei Zimmerman
94d37e62fc
treewide: Support builds with ASAN, enable in CI
Enables builds with ASAN to catch memory corruption
bugs faster and in CI. This is an incredibly valuable
instrument that must be used as much as possible.

Somewhat based on jade's work from Lix, though there's a lot that
we have to do differently:

19ae87e5ce

Co-authored-by: Jade Lovelace <lix@jade.fyi>
2025-09-19 01:33:57 +03:00
Sergei Zimmerman
7dbfe9f576
meson: Fix SONAME for unstable versions
Replacing the string is not enough [^] for e.g. nixpkgs precise versions:

`2.31pre20250712_b1245123`

[^]: https://github.com/NixOS/nixpkgs/pull/444089
2025-09-18 23:34:27 +03:00
Jörg Thalheim
3a687eeb4f
Merge pull request #14015 from eclairevoyant/doc-global-builtins
doc: document global builtins
2025-09-18 08:11:18 +02:00
Jörg Thalheim
4808b0e24f
Merge pull request #14014 from xokdvium/fish delete
libexpr-c: Fix mismatched new/delete
2025-09-18 08:08:39 +02:00
éclairevoyant
40b47b9395
doc: document global builtins 2025-09-18 00:04:31 -04:00
Sergei Zimmerman
309d55807c
libexpr-c: Fix mismatched new/delete
This leads to ASAN errors:

==1137785==ERROR: AddressSanitizer: new-delete-type-mismatch on 0x523000001d00 in thread T0:
  object passed to delete has wrong type:
  size of the allocated type:   5968 bytes;
  size of the deallocated type: 5968 bytes.
  alignment of the allocated type:   8 bytes;
  alignment of the deallocated type: default-aligned.
2025-09-18 02:58:32 +03:00
Jörg Thalheim
9d8c6a6646
Merge pull request #14013 from xokdvium/revert-13938
Revert "Merge pull request #13938 from NixOS/import-thunk"
2025-09-18 01:23:14 +02:00
Sergei Zimmerman
fd034814dc
Revert "Merge pull request #13938 from NixOS/import-thunk"
This has multiple dangling pointer issues that lead to segfaults in e.g.:

nix eval --expr '(builtins.getFlake "github:nixos/nixpkgs/25.05")' --impure

This reverts commit ad175727e4, reversing
changes made to d314750174.
2025-09-18 01:52:46 +03:00
Robert Hensing
b4fcb64276
Merge pull request #13980 from obsidiansystems/drv-json-issue-13570
Make the JSON format for derivation use basename store paths
2025-09-17 23:31:41 +02:00
Robert Hensing
82315c3807
Merge pull request #13987 from xokdvium/bindings-bananza
libexpr: Structural sharing of attrsets
2025-09-17 23:15:05 +02:00
Sergei Zimmerman
0ccb00bb81
libexpr: Add release note for c-api-byidx change 2025-09-17 23:54:46 +03:00
Sergei Zimmerman
6138bc3de3
libexpr: Structural sharing of attrsets
This changes the implementation of Bindings to allow
for a more space-efficient implementation of attribute
set merges. This is accomplished by "layering" over the "base" Bindings.
The top "layer" is naturally the right-hand-side of the update operator //.

Such an implementation leads to significantly better memory usage on
something like nixpkgs:

nix-env --query --available --out-path --file ../nixpkgs --eval-system x86_64-linux > /dev/null

Comparison against 2b0fd88324 for x86_64-linux on nixpkgs f06c7c3b6f5074dbffcf02542fb86af3a5526afa:

| metric                 | mean_before     | mean_after      | mean_diff       | mean_%_change | p_value | t_stat  |
| -                      | -               | -               | -               | -             | -       | -       |
| cpuTime                | 21.1520         | 21.3414         | 0.1894          | 0.7784        | 0.3190  | 1.0219  |
| envs.bytes             | 461451951.6190  | 461451951.6190  | -               | -             | -       | -       |
| envs.elements          | 34344544.8571   | 34344544.8571   | -               | -             | -       | -       |
| envs.number            | 23336949.0952   | 23336949.0952   | -               | -             | -       | -       |
| gc.cycles              | 7.5238          | 7.2857          | -0.2381         | -4.6825       | 0.0565  | -2.0244 |
| gc.heapSize            | 1777848124.9524 | 1252162023.6190 | -525686101.3333 | -29.9472      | 0.0000  | -8.7041 |
| gc.totalBytes          | 3102787383.6190 | 2498431578.6667 | -604355804.9524 | -19.7704      | 0.0000  | -9.3502 |
| list.bytes             | 59928225.9048   | 59928225.9048   | -               | -             | -       | -       |
| list.concats           | 1240028.2857    | 1240028.2857    | -               | -             | -       | -       |
| list.elements          | 7491028.2381    | 7491028.2381    | -               | -             | -       | -       |
| nrAvoided              | 28165342.2381   | 28165342.2381   | -               | -             | -       | -       |
| nrExprs                | 1577412.9524    | 1577412.9524    | -               | -             | -       | -       |
| nrFunctionCalls        | 20970743.4286   | 20970743.4286   | -               | -             | -       | -       |
| nrLookups              | 10867306.0952   | 10867306.0952   | -               | -             | -       | -       |
| nrOpUpdateValuesCopied | 61206062.0000   | 25748169.5238   | -35457892.4762  | -58.8145      | 0.0000  | -8.9189 |
| nrOpUpdates            | 2167097.4286    | 2167097.4286    | -               | -             | -       | -       |
| nrPrimOpCalls          | 12337423.4286   | 12337423.4286   | -               | -             | -       | -       |
| nrThunks               | 29361806.7619   | 29361806.7619   | -               | -             | -       | -       |
| sets.bytes             | 1393822818.6667 | 897587655.2381  | -496235163.4286 | -36.7168      | 0.0000  | -9.1115 |
| sets.elements          | 84504465.3333   | 48270845.9524   | -36233619.3810  | -43.8698      | 0.0000  | -8.9181 |
| sets.number            | 5218921.6667    | 5218921.6667    | -               | -             | -       | -       |
| sizes.Attr             | 16.0000         | 16.0000         | -               | -             | -       | -       |
| sizes.Bindings         | 8.0000          | 24.0000         | 16.0000         | 200.0000      | -       | inf     |
| sizes.Env              | 8.0000          | 8.0000          | -               | -             | -       | -       |
| sizes.Value            | 16.0000         | 16.0000         | -               | -             | -       | -       |
| symbols.bytes          | 1368494.0952    | 1368494.0952    | -               | -             | -       | -       |
| symbols.number         | 109147.1905     | 109147.1905     | -               | -             | -       | -       |
| time.cpu               | 21.1520         | 21.3414         | 0.1894          | 0.7784        | 0.3190  | 1.0219  |
| time.gc                | 1.6011          | 0.8508          | -0.7503         | -37.1507      | 0.0017  | -3.6328 |
| time.gcFraction        | 0.0849          | 0.0399          | -0.0450         | -37.4504      | 0.0035  | -3.3116 |
| values.bytes           | 615968144.7619  | 615968144.7619  | -               | -             | -       | -       |
| values.number          | 38498009.0476   | 38498009.0476   | -               | -             | -       | -       |

Overall this does slow down the evaluator slightly (no more than ~10% in most cases),
but this seems like a very decent tradeoff for shaving off 33% of memory usage.
2025-09-17 23:54:45 +03:00
John Ericson
9d7229a2a4 Make the JSON format for derivation use basename store paths
See #13570 for details --- the idea is that included the store dir in
store paths makes systematic JSON parting with e.g. Serde, Aeson,
nlohmann, or similiar harder.

After talking to Eelco, we are changing the `Derivation` format right
away because not only is `nix derivation` technically experimental, we think it is
also less widely used in practice than, say, `nix path-info`.

Progress on #13570
2025-09-17 16:38:17 -04:00
Robert Hensing
3eb223f4bb
Merge pull request #10915 from NixOS/dummy-memory
Use `MemorySourceAccessor` in `DummyStore` so writes work
2025-09-17 22:33:17 +02:00
John Ericson
d5ce8c3caa Use MemorySourceAccessor in DummyStore
Add `read-only` setting to `dummy://` store for back compat.

Test by changing an existing test to use this instead, fixing a TODO.

Co-Authored-By: HaeNoe <git@haenoe.party>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2025-09-17 16:09:53 -04:00
Eelco Dolstra
7ff61576f2
Merge pull request #14009 from xokdvium/fix-authorization
Revert "tests/nixos: Fix daemon store reference in authorization test"
2025-09-17 21:58:44 +02:00
John Ericson
168c24b605 Declare DummyStoreConfig in a header
This will useful for unit tests.
2025-09-17 15:54:30 -04:00
John Ericson
613de9d9cc Add missing #pragma once 2025-09-17 15:21:21 -04:00
Sergei Zimmerman
86ad8d49f9
Revert "tests/nixos: Fix daemon store reference in authorization test"
This reverts commit 695f3bc7e3.
2025-09-17 22:05:26 +03:00
Jörg Thalheim
187520ce88
Merge pull request #14008 from juhp/update-COPYING
COPYING: update to latest lgpl-2.1.txt
2025-09-17 20:15:01 +02:00
Jens Petersen
a66ba324d7 COPYING: update to latest lgpl-2.1.txt (fixes #13758)
$ curl -I https://www.gnu.org/licenses/old-licenses/lgpl-2.1.txt
:
Last-Modified: Wed, 18 Sep 2024 14:34:04 GMT
ETag: "6733-62265b29fd1ee"
:
2025-09-18 00:45:01 +08:00
Jörg Thalheim
7822bd5692
Merge pull request #14005 from juhp/nix_soversion
nix-meson-build-support/common nix_soversion: fixup removal of 'pre'
2025-09-17 12:19:57 +02:00
Robert Hensing
0b19468368
Merge pull request #13907 from obsidiansystems/store-building-unit-test
C API for building
2025-09-16 19:45:51 +02:00
John Ericson
9bc218ca3f Add new C API for working with derivations
Also test the APIs we just added.
2025-09-16 13:25:36 -04:00
Jens Petersen
ca23c819e0 nix-meson-build-support/common nix_soversion: fixup removal of 'pre'
.strip() removes individual chars whereas .replace() affects whole substring

Thanks @keszybz
2025-09-16 18:31:40 +08:00
Jörg Thalheim
1a69fc6ab5
Merge pull request #14001 from juhp/common-soversion
meson: refactor nix_soversion into nix-meson-build-support/common
2025-09-16 08:50:05 +02:00
Jens Petersen
86bb7c958a meson: refactor nix_soversion into nix-meson-build-support/common
This is a follow-on to #13995 which added soversion to the libraries
2025-09-16 12:54:30 +08:00
Jörg Thalheim
5e17a3f81c
Merge pull request #13998 from Mic92/fast-flake-check
nix flake check: Skip substitutable derivations
2025-09-15 21:44:11 +02:00
Seth Flynn
ecdda5798c nix flake check: Skip substitutable derivations
Since `nix flake check` doesn't produce a `result` symlink, it doesn't
actually need to build/substitute derivations that are already known
to have succeeded, i.e. that are substitutable.

This can speed up CI jobs in cases where the derivations have already
been built by other jobs. For instance, a command like

  nix flake check github:NixOS/hydra/aa62c7f7db31753f0cde690f8654dd1907fc0ce2

should no longer build anything because the outputs are already in
cache.nixos.org.

Based-on: https://github.com/DeterminateSystems/nix-src/pull/134
Based-on: https://gerrit.lix.systems/c/lix/+/3841
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2025-09-15 21:31:03 +02:00
Jörg Thalheim
0b401e2199
Merge pull request #13995 from juhp/soversion
meson: add soversion with nix version to give SONAME to libs
2025-09-15 20:36:31 +02:00
Eelco Dolstra
ad175727e4
Merge pull request #13938 from NixOS/import-thunk
Ensure that files are parsed/evaluated only once
2025-09-15 19:03:18 +02:00
John Ericson
d314750174
Merge pull request #13999 from obsidiansystems/more-get-improvements
More `get` / `getOr` improvements
2025-09-15 12:55:40 -04:00
John Ericson
f3e3f75838 More get / getOr improvements
- Use `const K`, not `K`, otherwise we don't get auto referencing of
  rvalues.

- Generalized the deleted overloads, because we don't care what the key
  type is --- we want to get rid of anything that has an rvalue map
  type.
2025-09-15 12:25:37 -04:00
Jens Petersen
dd1a554aba meson: add soversion with nix version to give SONAME to libs (#13960, #13979)
remove 'pre' version suffix for non-releases (chokes Darwin ld)
2025-09-15 22:08:26 +08:00
Sergei Zimmerman
2b0fd88324
Merge pull request #13991 from xokdvium/bindings-remove-find
libexpr: Remove Bindings::find
2025-09-14 21:32:31 +00:00
John Ericson
ffc14ac91b
Merge pull request #13983 from xokdvium/bindings-fixes
libexpr: Make Bindings::iterator a proper strong type instead of pointer
2025-09-14 17:31:48 -04:00
Sergei Zimmerman
d830840433
libexpr: Remove Bindings::find
A follow-up optimization will make it impossible to make a find function
that returns an iterator in an efficient manner. All consumer code can
easily use the `get` variant.
2025-09-14 23:29:44 +03:00
Sergei Zimmerman
ddabd94f82
libexpr: Make Bindings::iterator a proper strong type instead of pointer
As evident from the number of tests that were holding this API completely
wrong (the end() iterator returned from find() is NEVER nullptr) we should
not have this footgun. A proper strong type guarantees that this confusion
will not happen again.

Also this will be helpful down the road when Bindings becomes something
smarter than an array of Attr.
2025-09-14 22:52:37 +03:00
Sergei Zimmerman
b974b7dc1e
Merge pull request #13985 from dramforever/master 2025-09-14 11:19:10 +00:00
dramforever
7295034362 libstore: Raise default connect-timeout to 15 secs
This allows the weird network or DNS server fallback mechanism inside
glibc to work, and prevents a "Resolving timed out after 5000
milliseconds" error. Read on for details.

The DNS request stuff (dns-hosts) in glibc uses this fallback procedure
to minimize network RTT in the ideal case while dealing with
ill-behaving networks and DNS servers gracefully (see resolv.conf(5)):

- Use sendmmsg() to send UDP DNS requests for IPv4 and IPv6 in parallel
- If that times out (meaning that none or only one of the responses have
  been received), send the requests one by one, waiting for the response
  before sending the next request ("single-request")
- If that still times out, try to use a different socket (hence
  different address) for each request ("single-request-reopen")

The default timeout inside glibc is 5 seconds. Therefore, setting
connect-timeout, and therefore CURLOPT_CONNECTTIMEOUT to 5 seconds
prevents the single-request fallback, and setting it to even 10 seconds
prevents the single-request-reopen fallback as well.

The fallback decision is saved by glibc, but only thread-locally, and
libcurl starts a new thread for getaddrinfo() for each connection.
Therefore for every connection the fallback starts from sendmmsg() all
over again. And since these are considered to have timed out by libcurl,
even though getaddrinfo() might return a successful result, it is not
cached in libcurl.

While a user could tweak these with resolv.conf(5) options (e.g. using
networking.resolvconf.extraOptions in NixOS), and indeed that is
probably needed to avoid annoying delays, it still means that the
default connect-timeout of 5 is too low. Raise it to give fallback a
chance.
2025-09-14 08:48:29 +08:00
John Ericson
5bc96798b1
Merge pull request #13982 from obsidiansystems/path-info-static-function
`ValidPathInfo`, `NarInfo`, turn funky constructor into static method
2025-09-13 20:10:45 -04:00
Sergei Zimmerman
e75501da3e
libexpr: Remove non-const iterators of Bindings 2025-09-13 23:21:24 +03:00
John Ericson
74be28820c ValidPathInfo, NarInfo, turn funky constructor into static method
This is more flexible, and needed for me to be able to reshuffle the
inheritance bureaucracy to make the JSON instances more precise.
2025-09-13 13:17:14 -04:00