Skip to content

Commit

Permalink
Merge branch 'NixOS:master' into master
Browse files Browse the repository at this point in the history
  • Loading branch information
prinzdezibel authored Dec 9, 2024
2 parents 971d7ef + a44f454 commit a020629
Show file tree
Hide file tree
Showing 275 changed files with 5,678 additions and 17,122 deletions.
6 changes: 3 additions & 3 deletions doc/build-helpers/images/dockertools.section.md
Original file line number Diff line number Diff line change
Expand Up @@ -881,7 +881,7 @@ dockerTools.pullImage {
imageDigest = "sha256:b8ea88f763f33dfda2317b55eeda3b1a4006692ee29e60ee54ccf6d07348c598";
finalImageName = "nix";
finalImageTag = "2.19.3";
sha256 = "zRwlQs1FiKrvHPaf8vWOR/Tlp1C5eLn1d9pE4BZg3oA=";
hash = "sha256-zRwlQs1FiKrvHPaf8vWOR/Tlp1C5eLn1d9pE4BZg3oA=";
}
```
:::
Expand All @@ -898,7 +898,7 @@ dockerTools.pullImage {
imageDigest = "sha256:24a23053f29266fb2731ebea27f915bb0fb2ae1ea87d42d890fe4e44f2e27c5d";
finalImageName = "etcd";
finalImageTag = "v3.5.11";
sha256 = "Myw+85f2/EVRyMB3axECdmQ5eh9p1q77FWYKy8YpRWU=";
hash = "sha256-Myw+85f2/EVRyMB3axECdmQ5eh9p1q77FWYKy8YpRWU=";
}
```
:::
Expand All @@ -922,7 +922,7 @@ Writing manifest to image destination
{
imageName = "nixos/nix";
imageDigest = "sha256:498fa2d7f2b5cb3891a4edf20f3a8f8496e70865099ba72540494cd3e2942634";
sha256 = "1q6cf2pdrasa34zz0jw7pbs6lvv52rq2aibgxccbwcagwkg2qj1q";
hash = "sha256-OEgs3uRPMb4Y629FJXAWZW9q9LqHS/A/GUqr3K5wzOA=";
finalImageName = "nixos/nix";
finalImageTag = "latest";
}
Expand Down
60 changes: 29 additions & 31 deletions doc/languages-frameworks/python.section.md
Original file line number Diff line number Diff line change
Expand Up @@ -551,9 +551,9 @@ are used in [`buildPythonPackage`](#buildpythonpackage-function).

Several versions of the Python interpreter are available on Nix, as well as a
high amount of packages. The attribute `python3` refers to the default
interpreter, which is currently CPython 3.11. The attribute `python` refers to
interpreter, which is currently CPython 3.12. The attribute `python` refers to
CPython 2.7 for backwards-compatibility. It is also possible to refer to
specific versions, e.g. `python311` refers to CPython 3.11, and `pypy` refers to
specific versions, e.g. `python312` refers to CPython 3.12, and `pypy` refers to
the default PyPy interpreter.

Python is used a lot, and in different ways. This affects also how it is
Expand All @@ -569,10 +569,10 @@ however, are in separate sets, with one set per interpreter version.
The interpreters have several common attributes. One of these attributes is
`pkgs`, which is a package set of Python libraries for this specific
interpreter. E.g., the `toolz` package corresponding to the default interpreter
is `python3.pkgs.toolz`, and the CPython 3.11 version is `python311.pkgs.toolz`.
is `python3.pkgs.toolz`, and the CPython 3.12 version is `python312.pkgs.toolz`.
The main package set contains aliases to these package sets, e.g.
`pythonPackages` refers to `python.pkgs` and `python311Packages` to
`python311.pkgs`.
`pythonPackages` refers to `python.pkgs` and `python312Packages` to
`python312.pkgs`.

#### Installing Python and packages {#installing-python-and-packages}

Expand All @@ -597,7 +597,7 @@ with [`python.buildEnv`](#python.buildenv-function) or [`python.withPackages`](#
executables are wrapped to be able to find each other and all of the modules.

In the following examples we will start by creating a simple, ad-hoc environment
with a nix-shell that has `numpy` and `toolz` in Python 3.11; then we will create
with a nix-shell that has `numpy` and `toolz` in Python 3.12; then we will create
a re-usable environment in a single-file Python script; then we will create a
full Python environment for development with this same environment.

Expand All @@ -613,18 +613,18 @@ temporary shell session with a Python and a *precise* list of packages (plus
their runtime dependencies), with no other Python packages in the Python
interpreter's scope.

To create a Python 3.11 session with `numpy` and `toolz` available, run:
To create a Python 3.12 session with `numpy` and `toolz` available, run:

```sh
$ nix-shell -p 'python311.withPackages(ps: with ps; [ numpy toolz ])'
$ nix-shell -p 'python312.withPackages(ps: with ps; [ numpy toolz ])'
```

By default `nix-shell` will start a `bash` session with this interpreter in our
`PATH`, so if we then run:

```Python console
[nix-shell:~/src/nixpkgs]$ python3
Python 3.11.3 (main, Apr 4 2023, 22:36:41) [GCC 12.2.0] on linux
Python 3.12.4 (main, Jun 6 2024, 18:26:44) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy; import toolz
```
Expand All @@ -644,12 +644,8 @@ will still get 1 wrapped Python interpreter. We can start the interpreter
directly like so:

```sh
$ nix-shell -p "python311.withPackages (ps: with ps; [ numpy toolz requests ])" --run python3
this derivation will be built:
/nix/store/r19yf5qgfiakqlhkgjahbg3zg79549n4-python3-3.11.2-env.drv
building '/nix/store/r19yf5qgfiakqlhkgjahbg3zg79549n4-python3-3.11.2-env.drv'...
created 273 symlinks in user environment
Python 3.11.2 (main, Feb 7 2023, 13:52:42) [GCC 12.2.0] on linux
$ nix-shell -p "python312.withPackages (ps: with ps; [ numpy toolz requests ])" --run python3
Python 3.12.4 (main, Jun 6 2024, 18:26:44) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>>
Expand Down Expand Up @@ -689,7 +685,7 @@ Executing this script requires a `python3` that has `numpy`. Using what we learn
in the previous section, we could startup a shell and just run it like so:

```ShellSession
$ nix-shell -p 'python311.withPackages (ps: with ps; [ numpy ])' --run 'python3 foo.py'
$ nix-shell -p 'python312.withPackages (ps: with ps; [ numpy ])' --run 'python3 foo.py'
The dot product of [1 2] and [3 4] is: 11
```

Expand Down Expand Up @@ -752,12 +748,12 @@ create a single script with Python dependencies, but in the course of normal
development we're usually working in an entire package repository.

As explained [in the `nix-shell` section](https://nixos.org/manual/nix/stable/command-ref/nix-shell) of the Nix manual, `nix-shell` can also load an expression from a `.nix` file.
Say we want to have Python 3.11, `numpy` and `toolz`, like before,
Say we want to have Python 3.12, `numpy` and `toolz`, like before,
in an environment. We can add a `shell.nix` file describing our dependencies:

```nix
with import <nixpkgs> {};
(python311.withPackages (ps: with ps; [
(python312.withPackages (ps: with ps; [
numpy
toolz
])).env
Expand All @@ -774,7 +770,7 @@ What's happening here?
imports the `<nixpkgs>` function, `{}` calls it and the `with` statement
brings all attributes of `nixpkgs` in the local scope. These attributes form
the main package set.
2. Then we create a Python 3.11 environment with the [`withPackages`](#python.withpackages-function) function, as before.
2. Then we create a Python 3.12 environment with the [`withPackages`](#python.withpackages-function) function, as before.
3. The [`withPackages`](#python.withpackages-function) function expects us to provide a function as an argument
that takes the set of all Python packages and returns a list of packages to
include in the environment. Here, we select the packages `numpy` and `toolz`
Expand All @@ -785,7 +781,7 @@ To combine this with `mkShell` you can:
```nix
with import <nixpkgs> {};
let
pythonEnv = python311.withPackages (ps: [
pythonEnv = python312.withPackages (ps: [
ps.numpy
ps.toolz
]);
Expand Down Expand Up @@ -939,8 +935,8 @@ information. The output of the function is a derivation.

An expression for `toolz` can be found in the Nixpkgs repository. As explained
in the introduction of this Python section, a derivation of `toolz` is available
for each interpreter version, e.g. `python311.pkgs.toolz` refers to the `toolz`
derivation corresponding to the CPython 3.11 interpreter.
for each interpreter version, e.g. `python312.pkgs.toolz` refers to the `toolz`
derivation corresponding to the CPython 3.12 interpreter.

The above example works when you're directly working on
`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though,
Expand All @@ -953,7 +949,7 @@ and adds it along with a `numpy` package to a Python environment.
with import <nixpkgs> {};
( let
my_toolz = python311.pkgs.buildPythonPackage rec {
my_toolz = python312.pkgs.buildPythonPackage rec {
pname = "toolz";
version = "0.10.0";
pyproject = true;
Expand All @@ -964,7 +960,7 @@ with import <nixpkgs> {};
};
build-system = [
python311.pkgs.setuptools
python312.pkgs.setuptools
];
# has no tests
Expand All @@ -977,15 +973,15 @@ with import <nixpkgs> {};
};
};
in python311.withPackages (ps: with ps; [
in python312.withPackages (ps: with ps; [
numpy
my_toolz
])
).env
```

Executing `nix-shell` will result in an environment in which you can use
Python 3.11 and the `toolz` package. As you can see we had to explicitly mention
Python 3.12 and the `toolz` package. As you can see we had to explicitly mention
for which Python version we want to build a package.

So, what did we do here? Well, we took the Nix expression that we used earlier
Expand Down Expand Up @@ -1991,7 +1987,7 @@ has security implications and is relevant for those using Python in a

When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will
have timestamp 1. The [`buildPythonPackage`](#buildpythonpackage-function) function sets `DETERMINISTIC_BUILD=1`
and [PYTHONHASHSEED=0](https://docs.python.org/3.11/using/cmdline.html#envvar-PYTHONHASHSEED).
and [PYTHONHASHSEED=0](https://docs.python.org/3.12/using/cmdline.html#envvar-PYTHONHASHSEED).
Both are also exported in `nix-shell`.

### How to provide automatic tests to Python packages? {#automatic-tests}
Expand Down Expand Up @@ -2062,10 +2058,12 @@ The following rules are desired to be respected:
* `meta.platforms` takes the default value in many cases.
It does not need to be set explicitly unless the package requires a specific platform.
* The file is formatted with `nixfmt-rfc-style`.
* Commit names of Python libraries should reflect that they are Python
libraries, so write for example `python311Packages.numpy: 1.11 -> 1.12`.
It is highly recommended to specify the current default version to enable
automatic build by ofborg.
* Commit names of Python libraries must reflect that they are Python
libraries (e.g. `python312Packages.numpy: 1.11 -> 1.12` rather than `numpy: 1.11 -> 1.12`).
* The current default version of python should be included
in commit messages to enable automatic builds by ofborg.
For example `python312Packages.numpy: 1.11 -> 1.12` should be used rather
than `python3Packages.numpy: 1.11 -> 1.12`.
Note that `pythonPackages` is an alias for `python27Packages`.
* Attribute names in `python-packages.nix` as well as `pname`s should match the
library's name on PyPI, but be normalized according to [PEP
Expand Down
19 changes: 8 additions & 11 deletions lib/options.nix
Original file line number Diff line number Diff line change
Expand Up @@ -420,20 +420,17 @@ rec {
Placeholders will not be quoted as they are not actual values:
(showOption ["foo" "*" "bar"]) == "foo.*.bar"
(showOption ["foo" "<name>" "bar"]) == "foo.<name>.bar"
(showOption ["foo" "<myPlaceholder>" "bar"]) == "foo.<myPlaceholder>.bar"
*/
showOption = parts: let
# If the part is a named placeholder of the form "<...>" don't escape it.
# It may cause misleading escaping if somebody uses literally "<...>" in their option names.
# This is the trade-off to allow for placeholders in option names.
isNamedPlaceholder = builtins.match "\<(.*)\>";
escapeOptionPart = part:
let
# We assume that these are "special values" and not real configuration data.
# If it is real configuration data, it is rendered incorrectly.
specialIdentifiers = [
"<name>" # attrsOf (submodule {})
"*" # listOf (submodule {})
"<function body>" # functionTo
];
in if builtins.elem part specialIdentifiers
then part
else lib.strings.escapeNixIdentifier part;
if part == "*" || isNamedPlaceholder part != null
then part
else lib.strings.escapeNixIdentifier part;
in (concatStringsSep ".") (map escapeOptionPart parts);
showFiles = files: concatStringsSep " and " (map (f: "`${f}'") files);

Expand Down
38 changes: 38 additions & 0 deletions lib/tests/misc.nix
Original file line number Diff line number Diff line change
Expand Up @@ -1877,6 +1877,44 @@ runTests {
expected = [ [ "_module" "args" ] [ "foo" ] [ "foo" "<name>" "bar" ] [ "foo" "bar" ] ];
};

testAttrsWithName = {
expr = let
eval = evalModules {
modules = [
{
options = {
foo = lib.mkOption {
type = lib.types.attrsWith {
placeholder = "MyCustomPlaceholder";
elemType = lib.types.submodule {
options.bar = lib.mkOption {
type = lib.types.int;
default = 42;
};
};
};
};
};
}
];
};
opt = eval.options.foo;
in
(opt.type.getSubOptions opt.loc).bar.loc;
expected = [
"foo"
"<MyCustomPlaceholder>"
"bar"
];
};

testShowOptionWithPlaceholder = {
# <name>, *, should not be escaped. It is used as a placeholder by convention.
# Other symbols should be escaped. `{}`
expr = lib.showOption ["<name>" "<myName>" "*" "{foo}"];
expected = "<name>.<myName>.*.\"{foo}\"";
};

testCartesianProductOfEmptySet = {
expr = cartesianProduct {};
expected = [ {} ];
Expand Down
7 changes: 7 additions & 0 deletions lib/tests/modules.sh
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,9 @@ checkConfigOutput '^420$' config.bare-submodule.deep ./declare-bare-submodule.ni
checkConfigOutput '^2$' config.bare-submodule.deep ./declare-bare-submodule.nix ./declare-bare-submodule-deep-option.nix ./define-shorthandOnlyDefinesConfig-true.nix
checkConfigError 'The option .bare-submodule.deep. in .*/declare-bare-submodule-deep-option.nix. is already declared in .*/declare-bare-submodule-deep-option-duplicate.nix' config.bare-submodule.deep ./declare-bare-submodule.nix ./declare-bare-submodule-deep-option.nix ./declare-bare-submodule-deep-option-duplicate.nix

# Check that strMatching can be merged
checkConfigOutput '^"strMatching.*"$' options.sm.type.name ./strMatching-merge.nix

# Check integer types.
# unsigned
checkConfigOutput '^42$' config.value ./declare-int-unsigned-value.nix ./define-value-int-positive.nix
Expand Down Expand Up @@ -391,6 +394,10 @@ checkConfigError 'The option `mergedLazyNonLazy'\'' in `.*'\'' is already declar
checkConfigOutput '^11$' config.lazyResult ./lazy-attrsWith.nix
checkConfigError 'infinite recursion encountered' config.nonLazyResult ./lazy-attrsWith.nix

# AttrsWith placeholder tests
checkConfigOutput '^"mergedName.<id>.nested"$' config.result ./name-merge-attrsWith-1.nix
checkConfigError 'The option .mergedName. in .*\.nix. is already declared in .*\.nix' config.mergedName ./name-merge-attrsWith-2.nix

# Even with multiple assignments, a type error should be thrown if any of them aren't valid
checkConfigError 'A definition for option .* is not of type .*' \
config.value ./declare-int-unsigned-value.nix ./define-value-list.nix ./define-value-int-positive.nix
Expand Down
51 changes: 51 additions & 0 deletions lib/tests/modules/name-merge-attrsWith-1.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
{ lib, ... }:
let
inherit (lib) types mkOption;
in
{
imports = [
# Module A
(
{ ... }:
{
options.mergedName = mkOption {
default = { };
type = types.attrsWith {
placeholder = "id"; # <- This is beeing tested
elemType = types.submodule {
options.nested = mkOption {
type = types.int;
default = 1;
};
};
};
};
}
)
# Module B
(
{ ... }:
{
# defines the default placeholder "name"
# type merging should resolve to "id"
options.mergedName = mkOption {
type = types.attrsOf (types.submodule { });
};
}
)

# Output
(
{
options,
...
}:
{
options.result = mkOption {
default = lib.concatStringsSep "." (options.mergedName.type.getSubOptions options.mergedName.loc)
.nested.loc;
};
}
)
];
}
38 changes: 38 additions & 0 deletions lib/tests/modules/name-merge-attrsWith-2.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{ lib, ... }:
let
inherit (lib) types mkOption;
in
{
imports = [
# Module A
(
{ ... }:
{
options.mergedName = mkOption {
default = { };
type = types.attrsWith {
placeholder = "id"; # <- this is beeing tested
elemType = types.submodule {
options.nested = mkOption {
type = types.int;
default = 1;
};
};
};
};
}
)
# Module B
(
{ ... }:
{
options.mergedName = mkOption {
type = types.attrsWith {
placeholder = "other"; # <- define placeholder = "other" (conflict)
elemType = types.submodule { };
};
};
}
)
];
}
Loading

0 comments on commit a020629

Please sign in to comment.