5110 commits

Author	SHA1	Message	Date
Philip Wilk	aef431fbd1	bugfix/3514: do not throw on substituter errors if other substituters are still enabled (#13301) ... ## Motivation Nix currently hard fails if a substituter is inaccessible, even when they are other substituters available, unless `fallback = true`. This breaks nix build, run, shell et al entirely. This would modify the default behaviour so that nix would actually use the other available substituters and not hard error. Here is an example before vs after when using dotenv where I have manually stopped my own cache to trigger this issue, before and after the patch. The initial error is really frustrating because there is other caches available.   ## Context https://github.com/NixOS/nix/issues/3514#issuecomment-2905056198 is the earliest issue I could find, but there are many duplicates. There is an initial PR at https://github.com/NixOS/nix/pull/7188, but this appears to have been abandoned - over 2 years with no activity, then a no comment review in jan. There was a subsequent PR at https://github.com/NixOS/nix/pull/8983 but this was closed without merge - over a year without activity. <!-- Non-trivial change: Briefly outline the implementation strategy. --> I have visualised the current and proposed flows. I believe my logic flows line up with what is suggested in https://github.com/NixOS/nix/pull/7188#issuecomment-1375652870 but correct me if I am wrong. Current behaviour:  Proposed behaviour:  [Charts in lucid](https://lucid.app/lucidchart/1b51b08d-6c4f-40e0-bf54-480df322cccf/view) <!-- Invasive change: Discuss alternative designs or approaches you considered. --> Possible issues to think about: - I could not figure out where the curl error is created... I can't figure out how to swallow it and turn it into a warn or better yet, a debug log. - Unfortunately, in contrast with the previous point, I'm not sure how verbose we want to warns/traces to be - personally I think that the warn that a substituter has been disabled (when it happens) is sufficient, and that the next one is being used, but this is personal preference.	2025-09-12 17:29:34 -04:00
Sergei Zimmerman	0db2b8c8fe	Revert "meson: add soversion to libraries (#13960)" ... This reverts commit bdbc739d6e. Such a change needs more thought put into it. By versioning shared libraries we'd make a false impression that libraries themselves are actually versioned and have some sort of stable ABI, which is not the case. This will be useful when C bindings become stable, but as long as they are experimental it does not make sense to set SONAME. Also this change should not have been backported, since it's severely breaking.	2025-09-12 20:43:34 +03:00
Jörg Thalheim	377b60ee9b	Merge pull request #13926 from NaN-git/opt_boost-unordered ... replace all occurences of std::unordered_* by equivalents from boost	2025-09-12 11:46:42 +02:00
Jens Petersen	bdbc739d6e	meson: add soversion to libraries (#13960)	2025-09-12 14:06:39 +08:00
Philipp Otterbein	9dbc2cae4f	hashmaps with string keys: add transparent lookups	2025-09-10 23:04:44 +02:00
Philipp Otterbein	9f2b6a1b94	replace more std::unordered_* types by faster boost hash maps	2025-09-10 23:04:44 +02:00
Eelco Dolstra	fe5b669534	Move exportPaths() / importPaths() out of the Store class	2025-09-10 14:22:46 +02:00
Eelco Dolstra	9df99e0658	Remove ServeProto::Command::ExportPaths ... This seems to have been unused since the build-remote.pl removal in February 2017 (27dc76c1a5).	2025-09-10 10:57:15 +02:00
Eelco Dolstra	fa048e4383	Remove support for serve protocol < 5 ... This was introduced in August 2018 (2825e05d21).	2025-09-10 10:57:15 +02:00
Jörg Thalheim	8c789db05b	Merge pull request #13956 from NixOS/drop-unused-addMultipleToStoreLegacy ... Drop unused LegacySSHStore::addMultipleToStoreLegacy()	2025-09-10 10:52:05 +02:00
Eelco Dolstra	f8b15bfc7f	Merge pull request #13951 from NixOS/drop-old-daemon-protocol ... Remove support for worker protocol version < 18	2025-09-10 10:43:48 +02:00
Eelco Dolstra	5013b38df4	Drop unused LegacySSHStore::addMultipleToStoreLegacy()	2025-09-10 10:27:07 +02:00
Jörg Thalheim	7d26bf8cc7	Merge pull request #13906 from obsidiansystems/derivation-builder-simpler ... More `DerivationBuilder` simplifications	2025-09-10 09:58:07 +02:00
Jörg Thalheim	9c186c35fa	Merge pull request #13932 from NixOS/move-pathInfoCache ... Reduce false sharing between pathInfoCache and Store	2025-09-10 09:56:46 +02:00
Eelco Dolstra	86d19956f2	Remove WorkerProto::Op::ImportPaths ... This was obsoleted in May 2016 (538a64e8c3).	2025-09-09 15:41:17 +02:00
Eelco Dolstra	4fb61bc5af	Remove WorkerProto::Op::ExportPath ... This was obsoleted in May 2016 (538a64e8c3).	2025-09-09 15:41:12 +02:00
Eelco Dolstra	137a55122c	Remove support for daemon protocol version < 18 ... Version 18 was introduced in November 2016 (4b8f1b0ec0).	2025-09-09 15:41:01 +02:00
Sergei Zimmerman	7cc654afa9	libstore: Reallow unbracketed IPv6 addresses in store references ... This implements a special back-compat shim to specifically allow unbracketed IPv6 addresses in store references. This is something that is relied upon in the wild and the old parsing logic accepted both ways (brackets were optional). This patch restores this behavior. As always, we didn't have any tests for this. Addresses #13937.	2025-09-09 00:41:03 +03:00
Eelco Dolstra	9302ec5e0e	Add comment	2025-09-08 05:57:02 +02:00
Eelco Dolstra	e791ede495	LocalStore::State: Put behind a ref to reduce false sharing	2025-09-07 14:32:24 +02:00
Eelco Dolstra	a73cf447ac	Reduce false sharing between pathInfoCache and Store ... `perf c2c` shows a lot of cacheline conflicts between purely read-only Store methods (like `parseStorePath()`) and the Sync classes. So allocate pathInfoCache separately to avoid that.	2025-09-07 14:27:38 +02:00
Eelco Dolstra	9ff427d7ba	Merge pull request #13911 from xokdvium/store-uri-daemon-local ... libstore: Do not normalize daemon -> unix://, local -> local://	2025-09-07 14:10:45 +02:00
Eelco Dolstra	f363d958a7	Fix hang in enterChroot() draining userNamespaceSync ... Calling `drainFD()` will hang if another process has the write side open, since then the child won't get an EOF. This can happen if we have multiple threads doing a build, since in that case another thread may fork a child process that inherits the write side of the first thread. We could set O_CLOEXEC on the write side (using pipe2()) but it won't help here since we don't always do an exec() in the child, e.g. in the case of builtin builders. (We need a "close-on-fork", not a "close-on-exec".)	2025-09-07 01:12:44 +02:00
Sergei Zimmerman	3513ab13dc	libstore: Do not normalize daemon -> unix://, local -> local:// ... This is relied upon (specifically the `local` store) by existing tooling [1] and we broke this in 3e7879e6df (which was first released in 2.31). To lessen the scope of the breakage we should not normalize "auto" references and explicitly specified references like "local" or "daemon". It also makes sense to canonicalize local://,daemon:// to be more compatible with prior behavior. [1]: 05e1b3cba2/lib/NOM/Builds.hs (L60-L64)	2025-09-05 04:14:36 +03:00
John Ericson	49e9c14e2f	Merge pull request #13900 from NixOS/fix-mingw-windows-build ... Fix downstream MinGW build by not looking for Boost Regex	2025-09-04 21:05:12 -04:00
John Ericson	25d3c197b8	Merge pull request #13902 from NixOS/ssh-master-deadlock ... Fix deadlock in SSHMaster::addCommonSSHOpts()	2025-09-03 21:44:06 -04:00
John Ericson	2acb9559d5	Combine DerivationBuilder::{prepareBuild,startBuilder} ... After many other cleanups, it turns out there is no reason for these to be separate methods. We can combine them to simplify things.	2025-09-03 17:58:50 -04:00
John Ericson	14c206f05a	DerivationBuilder no more callback soup for logging ... `startBuilder` just returns the descriptor for the pipe now.	2025-09-03 17:34:45 -04:00
John Ericson	7f3314a68c	DerivationBuilder::initialOutputs make const ... At one point I remember it did mutatate `initialOutputs`, but not anymore!	2025-09-03 17:34:45 -04:00
John Ericson	b69576e2b3	Merge pull request #13905 from obsidiansystems/derivation-building-goal-simplify-0 ... Derivation building goal simplify -- no `goto`	2025-09-03 17:34:27 -04:00
John Ericson	7b22cd5105	Merge pull request #13839 from Mic92/infra ... don't include derivation name in temporary build directories	2025-09-03 17:15:03 -04:00
John Ericson	a30bf96349	DerivationBuildingGoal::initialOutputs make local variable ... Also inline `assertPathValidity` in the process.	2025-09-03 17:04:13 -04:00
John Ericson	c0c2a89f05	DerivationBuildingGoal::initialOutputs move initialization down to tryToBuild ... Will help us make this a local variable.	2025-09-03 17:04:08 -04:00
John Ericson	450633aa8c	Move machineName from DerivationBuildingGoal to HookInstance ... Exactly why is is correct is a little subtle, because sometimes the worker is owned by the worker. But the commit message in e437b08250 explained the situation well enough: I made that commit message part of the ABI docs, and now it should be understandable to the next person.	2025-09-03 17:03:56 -04:00
John Ericson	eb56b181ae	DerivationBuildingGoal: Make almost everything private	2025-09-03 16:25:12 -04:00
John Ericson	c6ba120000	DerivationBuildingGoal::started make local (lambda) variable	2025-09-03 16:19:35 -04:00
John Ericson	3b9c510ab1	DerivationBuildingGoal::outputLocks make local variable	2025-09-03 16:19:35 -04:00
John Ericson	a63ac8d98b	Inline DerivationBuildingGoal::hookDone	2025-09-03 16:19:35 -04:00
John Ericson	51dadaded4	Move up assert(!hook); ... We don't need to keep doing this every loop iteration, hook stuff it is only set above.	2025-09-03 16:19:35 -04:00
John Ericson	7c1e5b3345	In DerivationBuildingGoal Demote actLock to local variable ... It doesn't need to be a field any more, because we just use it with two loops.	2025-09-03 16:19:35 -04:00
John Ericson	4c44a213a3	Get rid of a tryToBuild tail recursive call with loop ... This will make it easier to convert somethings to RAII.	2025-09-03 16:19:35 -04:00
John Ericson	95c5779880	DerivationBuildingGoal::tryToBuild pull hook waiting out of switch ... Do this with a new `useHook` boolean we carefully make sure is set in all cases. This change isn't really worthwhile by itself, but it allows us to make further refactors (see later commits) which are well-motivated.	2025-09-03 16:19:35 -04:00
Eelco Dolstra	c7603c61c8	Mark tmpDir as const	2025-09-03 20:17:42 +02:00
Eelco Dolstra	2fe629c5d4	Fix deadlock in SSHMaster::addCommonSSHOpts() ... When useMaster is true, startMaster() acquires the state lock, then calls isMasterRunning(), which calls addCommonSSHOpts(), which tries to acquire the state lock again, causing a deadlock. The solution is to move tmpDir out of the state. It doesn't need to be there in the first place because it never changes.	2025-09-03 17:49:24 +02:00
John Ericson	6bdb5e8e09	Fix downstream MinGW build by not looking for Boost Regex	2025-09-02 10:41:39 -04:00
Jörg Thalheim	a0ce514769	Merge pull request #13866 from obsidiansystems/more-derivation-builder-cleanup ... Even more `DerivationBuilder` cleanup	2025-09-01 20:35:16 +02:00
Jörg Thalheim	0d300112fa	Merge pull request #13862 from obsidiansystems/build-failure-content-vs-presentation ... Properly separater builder failure content and presentation	2025-09-01 20:25:50 +02:00
Jörg Thalheim	de7f137f31	Merge pull request #13860 from obsidiansystems/derivation-building-resources-code-cleanup ... Derivation building resources code cleanup	2025-09-01 20:22:30 +02:00
Jörg Thalheim	dc29cdf66d	Merge pull request #13858 from obsidiansystems/no-more-defered-exception ... Get rid of `delayedException` in `DerivationBuilder`	2025-09-01 20:11:51 +02:00
Sergei Zimmerman	e548700010	lib{store,fetchers}: Pass URLs specified directly verbatim to FileTransferRequest ... The URL should not be normalized before handing it off to cURL, because builtin fetchers like fetchTarball/fetchurl are expected to work with arbitrary URLs, that might not be RFC3986 compliant. For those cases Nix should not normalize URLs, though validation is fine. ParseURL and cURL are supposed to match the set of acceptable URLs, since they implement the same RFC.	2025-09-01 02:22:23 +03:00