mirror of
https://github.com/NixOS/nix.git
synced 2025-11-08 19:46:02 +01:00
40 lines
1.9 KiB
Markdown
40 lines
1.9 KiB
Markdown
---
|
|
synopsis: "Improved S3 binary cache support via HTTP"
|
|
prs: [13752, 13823, 14026, 14120, 14131, 14135, 14144, 14170, 14190, 14198, 14206, 14209, 14222, 14223, 14330, 14333, 14335, 14336, 14337, 14350, 14356, 14357, 14374, 14375, 14376, 14377, 14391, 14393, 14420, 14421]
|
|
issues: [13084, 12671, 11748, 12403]
|
|
---
|
|
|
|
S3 binary cache operations now happen via HTTP, leveraging `libcurl`'s native
|
|
AWS SigV4 authentication instead of the AWS C++ SDK, providing significant
|
|
improvements:
|
|
|
|
- **Reduced memory usage**: Eliminates memory buffering issues that caused
|
|
segfaults with large files
|
|
- **Fixed upload reliability**: Resolves AWS SDK chunking errors
|
|
(`InvalidChunkSizeError`)
|
|
- **Lighter dependencies**: Uses lightweight `aws-crt-cpp` instead of full
|
|
`aws-cpp-sdk`, reducing build complexity
|
|
|
|
The new implementation requires curl >= 7.75.0 and `aws-crt-cpp` for credential
|
|
management.
|
|
|
|
All existing S3 URL formats and parameters remain supported, however the store
|
|
settings for configuring multipart uploads have changed:
|
|
|
|
- **`multipart-upload`** (default: `false`): Enable multipart uploads for large
|
|
files. When enabled, files exceeding the multipart threshold will be uploaded
|
|
in multiple parts.
|
|
|
|
- **`multipart-threshold`** (default: `100 MiB`): Minimum file size for using
|
|
multipart uploads. Files smaller than this will use regular PUT requests.
|
|
Only takes effect when `multipart-upload` is enabled.
|
|
|
|
- **`multipart-chunk-size`** (default: `5 MiB`): Size of each part in multipart
|
|
uploads. Must be at least 5 MiB (AWS S3 requirement). Larger chunk sizes
|
|
reduce the number of requests but use more memory.
|
|
|
|
- **`buffer-size`**: Has been replaced by `multipart-chunk-size` and is now an alias to it.
|
|
|
|
Note that this change also means Nix now supports S3 binary cache stores even
|
|
if built without `aws-crt-cpp`, but only for public buckets which do not
|
|
require authentication.
|