-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
A new YAML based format for "conda-build" files #54
Changes from 1 commit
6242c1a
34917a8
7a22c07
a4f15ae
628bffb
918ae80
878053c
85a5a3f
c49d58b
59c4d0a
2c8ad62
cd196f4
b7d677b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
- Loading branch information
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,166 @@ | ||||||
# A new recipe format – part 1 | ||||||
|
||||||
<table> | ||||||
<tr><td> Title </td><td> A new recipe format </td> | ||||||
<tr><td> Status </td><td> Proposed</td></tr> | ||||||
<tr><td> Author(s) </td><td> Wolf Vollprecht <wolf@prefix.dev></td></tr> | ||||||
<tr><td> Created </td><td> May 23, 2023</td></tr> | ||||||
<tr><td> Updated </td><td> May 23, 2023</td></tr> | ||||||
<tr><td> Discussion </td><td> </td></tr> | ||||||
<tr><td> Implementation </td><td>https://github.com/prefix-dev/rattler-build</td></tr> | ||||||
</table> | ||||||
|
||||||
## Abstract | ||||||
|
||||||
We propose a new recipe format that is heavily inspired by conda-build. The main change is a pure YAML format without arbitrary Jinja or comments with semantic meaning. | ||||||
|
||||||
## Motivation | ||||||
|
||||||
The conda-build format has grown over the years to become quite complex. Unfortunately it has never been formally "specified" and it has grown some features over time that make it hard to parse as straightforward YAML. | ||||||
|
||||||
The CEP attempts to introduce a subset of the conda build format that allows for fast parsing and building of recipes. | ||||||
|
||||||
### History | ||||||
|
||||||
A discussion was started on what a new recipe spec could or should look like. The fragments of this discussion can be found here: https://github.com/mamba-org/conda-specs/blob/master/proposed_specs/recipe.md | ||||||
The reason for a new spec are: | ||||||
|
||||||
- Make it easier to parse ("pure yaml"). conda-build uses a mix of comments and jinja to achieve a great deal of flexibility, but it's hard to parse the recipe with a computer | ||||||
- iron out some inconsistencies around multiple outputs (build vs. build/script and more) | ||||||
- remove any need for recursive parsing & solving | ||||||
wolfv marked this conversation as resolved.
Show resolved
Hide resolved
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The recursive parsing and solving was mainly to support things like inheriting specs, like run_constrained, from explicitly specified dependencies. Is there any kind of explicit spec for whether/how packages add/impose dependencies on sections outside of the one where they are specified? What about the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In boa (and rattler-build) we have "deferred" evaluation for the pins, so that still works. We just require that the name of the pin is available, in order to build the proper topologically sorted build-graph :) A dependency list in Indeed, There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sometimes the package name it isn't known. For example cross-compilation usage or compilers themselves There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @jakirkham in these cases you can infer those variables from the build-time config though (or variant config) so no recursion needed :) |
||||||
|
||||||
## Major differences with conda-build | ||||||
|
||||||
- no full Jinja2 support: no conditional or `{% set ...` support, only string interpolation. Variables can be set in the toplevel "context" which is valid YAML | ||||||
- Jinja string interpolation needs to be quoted at the beginning of a string, e.g. `- "{{ version }}"` in order for it to be valid YAML | ||||||
- Selectors use a YAML dictionary style (vs. comments in conda-build). E.g. `- sel(osx): somepkg` instead of `- somepkg # [osx]` | ||||||
|
||||||
## Selectors | ||||||
|
||||||
Selectors in the new spec take the following format: | ||||||
|
||||||
`sel(unix): selected_value` | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How about a simplified variant: There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not to bikeshed, but what about something like |
||||||
|
||||||
This is a valid YAML dictionary. Selector contents are simple boolean expressions and follow Python syntax. The following selectors are all valid: | ||||||
|
||||||
``` | ||||||
win and arm64 | ||||||
(osx or linux) and aarch64 | ||||||
something == "test" | ||||||
``` | ||||||
|
||||||
### The cmp function for variant selection | ||||||
|
||||||
Furthermore, we have a special "cmp" function that can be used to run a check against a selected variant version. The `cmp` function looks like the following: | ||||||
|
||||||
``` | ||||||
cmp(python, "3.6") | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I wonder if we have to use
Suggested change
Where it would then look like:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The problem is that operator overloading might be harder to implement vs. the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Or maybe if we know that the left hand side is a VariantSpecifier we could overload the comparison with strings ... but yeah, would need to figure out if that's easy to implement with the Jinja expression system(s). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Personally I also find the You could also do something like There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. +1 on There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. One caveat with
Not to bikeshed, but maybe the critiques in the comments above (and the caveat around
Same syntax, but maybe this already implies the conditional case evaluating to |
||||||
cmp(python, ">=3.6") | ||||||
cmp(python, ">=3.8,<3.10") | ||||||
etc | ||||||
``` | ||||||
|
||||||
This can be used in a selector like so: | ||||||
|
||||||
``` | ||||||
requirements: | ||||||
build: | ||||||
- sel(cmp(python, ">=3.6,<3.10")): dataclasses | ||||||
``` | ||||||
|
||||||
This functionality generalizes and replaces the previous special variables such as `py2k`, `py3k`, `py36`, `py37`, and works just as well for NumPy, Ruby, R, or any other variant that might be of interest in the future. | ||||||
|
||||||
### Preprocessing selectors | ||||||
|
||||||
You can add selectors to any item, and the selector is evaluated in | ||||||
a preprocessing stage. If a selector evaluates to `true`, the item is | ||||||
flattened into the parent element. If a selector evaluates to `false`, | ||||||
the item is removed. | ||||||
|
||||||
```yaml | ||||||
source: | ||||||
- sel(not win): | ||||||
url: http://path/to/unix/source | ||||||
- sel(win): | ||||||
url: http://path/to/windows/source | ||||||
``` | ||||||
|
||||||
Because the selector is a valid Jinja expression, complicated logic | ||||||
is possible: | ||||||
|
||||||
```yaml | ||||||
source: | ||||||
- sel(win): | ||||||
url: http://path/to/windows/source | ||||||
- sel(unix and cmp(python, "2")): | ||||||
url: http://path/to/python2/unix/source | ||||||
- sel(unix and cmp(python, "3")): | ||||||
url: http://path/to/python3/unix/source | ||||||
``` | ||||||
|
||||||
Lists are automatically "merged" upwards, so it is possible to group multiple items under a single selector: | ||||||
|
||||||
```yaml | ||||||
test: | ||||||
commands: | ||||||
- sel(unix): | ||||||
- test -d ${PREFIX}/include/xtensor | ||||||
- test -f ${PREFIX}/include/xtensor/xarray.hpp | ||||||
- test -f ${PREFIX}/lib/cmake/xtensor/xtensorConfig.cmake | ||||||
- test -f ${PREFIX}/lib/cmake/xtensor/xtensorConfigVersion.cmake | ||||||
- sel(win): | ||||||
- if not exist %LIBRARY_PREFIX%\include\xtensor\xarray.hpp (exit 1) | ||||||
- if not exist %LIBRARY_PREFIX%\lib\cmake\xtensor\xtensorConfig.cmake (exit 1) | ||||||
- if not exist %LIBRARY_PREFIX%\lib\cmake\xtensor\xtensorConfigVersion.cmake (exit 1) | ||||||
|
||||||
# On unix this is rendered to: | ||||||
test: | ||||||
commands: | ||||||
- test -d ${PREFIX}/include/xtensor | ||||||
- test -f ${PREFIX}/include/xtensor/xarray.hpp | ||||||
- test -f ${PREFIX}/lib/cmake/xtensor/xtensorConfig.cmake | ||||||
- test -f ${PREFIX}/lib/cmake/xtensor/xtensorConfigVersion.cmake | ||||||
``` | ||||||
|
||||||
## Templating with Jinja | ||||||
|
||||||
The spec supports simple Jinja templating in the `recipe.yaml` file. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It would be worth elaborating what "simple" means in this context, what wouldn't be covered by this? Would this also work with MiniJinja for example? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Simple means:
context:
version: "0.1.2.POST1234"
lower_version: "{{ version | lower }}" # evaluates to 0.1.2.post1234
This works fantastically with MiniJinja (which is what we use in There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is there a way to preserve the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Honestly since we are doing a bit of a redesign here, think it is worth asking whether we still want Jinja to play a role here. In other words, if we were not already using Jinja today, how would we have solved these use cases? How do others approach YAML templating? Could we think of pure YAML solutions? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree, what does jinja accomplish that we can't do with the other new features? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Well other new features would also have to be implemented and I am not sure we would necessary end up with something simpler. But I am not sure how to proceed here - we could either go down that road right now or build something that is more reasonable for people to port existing recipes to and then iterate in a next step on the recipe format (when we have the necessary infrastructure nicely implemented in rattler-build). |
||||||
|
||||||
You can set up Jinja variables in the context YAML section: | ||||||
|
||||||
```yaml | ||||||
context: | ||||||
name: "test" | ||||||
version: "5.1.2" | ||||||
major_version: "{{ version.split('.')[0] }}" | ||||||
``` | ||||||
|
||||||
Later in your `recipe.yaml` you can use these values in string interpolation | ||||||
with Jinja. For example: | ||||||
|
||||||
```yaml | ||||||
source: | ||||||
url: https://github.com/mamba-org/{{ name }}/v{{ version }}.tar.gz | ||||||
``` | ||||||
|
||||||
Jinja has built-in support for some common string manipulations. | ||||||
|
||||||
In the new spec, complex Jinja is completely disallowed as we try to produce YAML that is valid at all times. | ||||||
So you should not use any `{% if ... %}` or similar Jinja constructs that produce invalid yaml. | ||||||
Furthermore, quotes need to be applied when starting a value with double-curly brackets like so: | ||||||
|
||||||
```yaml | ||||||
package: | ||||||
name: {{ name }} # WRONG: invalid yaml | ||||||
name: "{{ name }}" # correct | ||||||
``` | ||||||
|
||||||
Some Jinja functions remain valid, but this is out of the scope of this spec. However, as an example, | ||||||
the `compiler` Jinja function will still be available, with the main difference being the quoting of the | ||||||
brackets. | ||||||
|
||||||
```yaml | ||||||
requirements: | ||||||
build: | ||||||
- "{{ compiler('cxx') }}" | ||||||
``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think one really important motivating factor that isn't mentioned in the motivation section is increasing the ease by which we can parse and modify these files (making them more machine readable friendly). I know that there is currently a desire to be able to do this, but the old format holds us back. It's very hard to parse and multiple attempts have already been made to try to do this if I'm not mistaken.
I know it's mentioned below, but it might be worth emphasizing here briefly too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1000