No way to pass --experimental to rattler-build

[h-vetinari]

I'm trying to convert zlib to rattler-build (tentatively towards emscripten support) broadly following this. However, it turns out that some of the needed functionality is currently only available in rattler-build with an --experimental flag.

Smithy currently fails to rerender with the following

Error:   × Parsing: experimental only: `cache`.
[...]

  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\configure_feedstock.py", line 1081, in _render_ci_provider
    metas = rattler_render(
            ^^^^^^^^^^^^^^^
  File "E:\miniforge\envs\builder\Lib\site-packages\rattler_build_conda_compat\render.py", line 280, in render
    metadata_tuples = render_recipe(
                      ^^^^^^^^^^^^^^
  [...]
  File "E:\miniforge\envs\builder\Lib\subprocess.py", line 571, in run
subprocess.CalledProcessError: Command '['rattler-build', 'build', '--render-only', '--recipe', 'E:\\conda-forge\\zlib-feedstock\\recipe', '--target-platform', 'linux-64', '--build-platform', 'linux-64', '-m', 'C:\\Users\\[xxx]\\AppData\\Local\\Temp\\tmpvu8htd_q']' returned non-zero exit status 1.

AFAIU the main fix will need to happen in https://github.com/prefix-dev/rattler-build-conda-compat (which needs to define the API how to pass that on), and smithy will need to figure out how to allow that to be configured and pass it on here:

conda-smithy/conda_smithy/configure_feedstock.py

Lines 1081 to 1088 in f7eb69b

	metas = rattler_render(
	os.path.join(forge_dir, forge_config["recipe_dir"]),
	platform=platform,
	arch=arch,
	ignore_system_variants=True,
	variants=migrated_combined_variant_spec,
	channel_urls=channel_sources,
	)

Of course, one could say we don't want to support experimental features, but with stuff as fundamental as the global build cache, we don't have a choice, and I think it would be worthwhile to support as opt-in also going forward.

However we configure it, it needs to be injected both for rerendering and for building, which I'm currently doing manually as follows:

diff --git a/.scripts/build_steps.sh b/.scripts/build_steps.sh
index 511960d..94e14fb 100755
--- a/.scripts/build_steps.sh
+++ b/.scripts/build_steps.sh
@@ -52,6 +52,8 @@ if [[ "${HOST_PLATFORM}" != "${BUILD_PLATFORM}" ]] && [[ "${HOST_PLATFORM}" != l
     EXTRA_CB_OPTIONS="${EXTRA_CB_OPTIONS:-} --no-test"
 fi

+# build portion of https://github.com/conda-forge/conda-smithy/issues/2057
+EXTRA_CB_OPTIONS="${EXTRA_CB_OPTIONS:-} --experimental"

 ( endgroup "Configuring conda" ) 2> /dev/null

diff --git a/.scripts/run_osx_build.sh b/.scripts/run_osx_build.sh
index aa5746d..dd05290 100755
--- a/.scripts/run_osx_build.sh
+++ b/.scripts/run_osx_build.sh
@@ -70,6 +70,8 @@ else
         EXTRA_CB_OPTIONS="${EXTRA_CB_OPTIONS:-} --no-test"
     fi

+    # build portion of https://github.com/conda-forge/conda-smithy/issues/2057
+    EXTRA_CB_OPTIONS="${EXTRA_CB_OPTIONS:-} --experimental"

     rattler-build build --recipe ./recipe \
         -m ./.ci_support/${CONFIG}.yaml \
diff --git a/.scripts/run_win_build.bat b/.scripts/run_win_build.bat
index 51d0d75..69f4f28 100755
--- a/.scripts/run_win_build.bat
+++ b/.scripts/run_win_build.bat
@@ -51,6 +51,9 @@ if NOT [%flow_run_id%] == [] (
         set "EXTRA_CB_OPTIONS=%EXTRA_CB_OPTIONS% --extra-meta flow_run_id=%flow_run_id% --extra-meta remote_url=%remote_url% --extra-meta sha=%sha%"
 )

+:: build portion of https://github.com/conda-forge/conda-smithy/issues/2057
+set "EXTRA_CB_OPTIONS=%EXTRA_CB_OPTIONS% --experimental"
+
 call :end_group

 :: Build the recipe

[bollwyvl]

Ran into this. How bad would a conda_build_tool: rattler-build-experimental be?

[hmaarrfk]

i feel like we shouldn't be integrating "experimental" stuff too much.

It seems the better ask would be to the rattler devs to ask them to make this feature less experimental

[wolfv]

yes, or help with the CEP 🙏 we need to write and get it voted on.conda/ceps#102