kubeflow-pipelines-visualization-server/2.4.0 package update

[octo-sts]

[octo-sts]

⚠️ EXPERIMENTAL

Please use 👍 or 👎 on this comment to indicate if you agree or disagree with the recommendation.

To provide more detailed feedback please comment on the recommendation prefixed with /ai-verify:

e.g. /ai-verify partially helpful but I also added bash to the build environment

Gen AI suggestions to solve the build error:

Based on the error output, I'll analyze and provide specific recommendations:

• Detected Error:

patching file backend/src/apiserver/visualization/requirements.in
Hunk #1 FAILED at 5.
[...]
12 out of 15 hunks FAILED -- saving rejects to file backend/src/apiserver/visualization/requirements.txt.rej

• Error Category: Build Configuration/Patch Failure

• Failure Point: Patch application step in the pipeline

• Root Cause Analysis:
The patch file 0001-Bump-dependencies.patch is failing to apply cleanly to the requirements files, likely because:

1. The patch was created against a different version of the source
2. The upstream requirements files have changed significantly since the patch was created
3. Line numbers and context in the patch file don't match the current source

• Suggested Fix:

1. Generate a new patch file based on the current 2.4.0 tag:

cd /tmp
git clone https://github.com/kubeflow/pipelines
cd pipelines
git checkout 2.4.0
# Make your dependency updates
git diff > 0001-Bump-dependencies.patch

2. Or modify the pipeline to directly update requirements without patches:

  - runs: |
      cd ${{targets.destdir}}/usr/share/app/
      # Direct dependency updates
      sed -i 's/^tensorflow==.*$/tensorflow>=2.13.0,<3.0.0/' requirements.txt
      sed -i 's/^numpy==.*$/numpy>=1.24.0/' requirements.txt
      # Add other direct replacements as needed

• Explanation:
The patch failure indicates version drift between when the patch was created and the current source. Direct modifications are more maintainable than patches when dealing with frequently changing dependency files.

• Additional Notes:

• Consider using requirements.txt generation from requirements.in with pip-compile
• Version constraints should align with Python 3.10 compatibility
• Security updates should be prioritized for dependencies

• References:

• Kubeflow Pipelines Release 2.4.0
• Wolfi OS Package Guidelines
• Melange Build System Documentation

[debasishbsws]

Hm, for some reason, recreation of these patches is not working. When the patches don’t apply cleanly, We updated the requirements.in file and attempted to run the following command to generate the updated requirements.txt file:

pip-compile --output-file=requirements.txt --resolver=backtracking requirements.in

However, it seems like pip-compile is now failing.

/work/backend/src/apiserver/visualization # pip-compile --output-file=requirements.txt --resolver=backtracking requirements.in
    error: subprocess-exited-with-error
    
    × Preparing metadata (pyproject.toml) did not run successfully.
    │ exit code: 1
    ╰─> [64 lines of output]
        <string>:17: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
        Partial import of sklearn during the build process.
        <string>:116: DeprecationWarning:
        
          `numpy.distutils` is deprecated since NumPy 1.23.0, as a result
          of the deprecation of `distutils` itself. It will be removed for
          Python >= 3.12. For older Python versions it will remain present.
          It is recommended to use `setuptools < 60.0` for those Python versions.
          For more details, see:
            https://numpy.org/devdocs/reference/distutils_status_migration.html
        
        
        INFO: C compiler: aarch64-unknown-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC
        
        INFO: compile options: '-c'
        INFO: aarch64-unknown-linux-gnu-gcc: test_program.c
        
        
        [Errno 2] No such file or directory: 'aarch64-unknown-linux-gnu-gcc'

I am encountering this error without making any changes to the requirements.in file, It is same as upstream.

See full ERROR:

/work/backend/src/apiserver/visualization # pip-compile --output-file=requirements.txt --resolver=backtracking requirements.in
    error: subprocess-exited-with-error
    
    × Preparing metadata (pyproject.toml) did not run successfully.
    │ exit code: 1
    ╰─> [64 lines of output]
        <string>:17: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
        Partial import of sklearn during the build process.
        <string>:116: DeprecationWarning:
        
          `numpy.distutils` is deprecated since NumPy 1.23.0, as a result
          of the deprecation of `distutils` itself. It will be removed for
          Python >= 3.12. For older Python versions it will remain present.
          It is recommended to use `setuptools < 60.0` for those Python versions.
          For more details, see:
            https://numpy.org/devdocs/reference/distutils_status_migration.html
        
        
        INFO: C compiler: aarch64-unknown-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC
        
        INFO: compile options: '-c'
        INFO: aarch64-unknown-linux-gnu-gcc: test_program.c
        
        
        [Errno 2] No such file or directory: 'aarch64-unknown-linux-gnu-gcc'
        
        
        Traceback (most recent call last):
          File "/usr/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
            main()
          File "/usr/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
            json_out['return_val'] = hook(**hook_input['kwargs'])
          File "/usr/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
            return hook(metadata_directory, config_settings)
          File "/tmp/pip-build-env-sylm65ig/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 377, in prepare_metadata_for_build_wheel
            self.run_setup()
          File "/tmp/pip-build-env-sylm65ig/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 522, in run_setup
            super().run_setup(setup_script=setup_script)
          File "/tmp/pip-build-env-sylm65ig/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in run_setup
            exec(code, locals())
          File "<string>", line 301, in <module>
          File "<string>", line 297, in setup_package
          File "/tmp/pip-build-env-sylm65ig/overlay/lib/python3.10/site-packages/numpy/distutils/core.py", line 135, in setup
            config = configuration()
          File "<string>", line 188, in configuration
          File "/tmp/pip-build-env-sylm65ig/overlay/lib/python3.10/site-packages/numpy/distutils/misc_util.py", line 1041, in add_subpackage
            config_list = self.get_subpackage(subpackage_name, subpackage_path,
          File "/tmp/pip-build-env-sylm65ig/overlay/lib/python3.10/site-packages/numpy/distutils/misc_util.py", line 1007, in get_subpackage
            config = self._get_configuration_from_setup_py(
          File "/tmp/pip-build-env-sylm65ig/overlay/lib/python3.10/site-packages/numpy/distutils/misc_util.py", line 949, in _get_configuration_from_setup_py
            config = setup_module.configuration(*args)
          File "/tmp/pip-resolve-ftya5g8k/scikit-learn_cec17a5f7e4146d6b8ca1ef8e2925688/sklearn/setup.py", line 83, in configuration
            cythonize_extensions(top_path, config)
          File "/tmp/pip-resolve-ftya5g8k/scikit-learn_cec17a5f7e4146d6b8ca1ef8e2925688/sklearn/_build_utils/__init__.py", line 45, in cythonize_extensions
            basic_check_build()
          File "/tmp/pip-resolve-ftya5g8k/scikit-learn_cec17a5f7e4146d6b8ca1ef8e2925688/sklearn/_build_utils/pre_build_helpers.py", line 106, in basic_check_build
            compile_test_program(code)
          File "/tmp/pip-resolve-ftya5g8k/scikit-learn_cec17a5f7e4146d6b8ca1ef8e2925688/sklearn/_build_utils/pre_build_helpers.py", line 66, in compile_test_program
            ccompiler.compile(['test_program.c'], output_dir='objects',
          File "/tmp/pip-build-env-sylm65ig/overlay/lib/python3.10/site-packages/numpy/distutils/ccompiler.py", line 91, in <lambda>
            m = lambda self, *args, **kw: func(self, *args, **kw)
          File "/tmp/pip-build-env-sylm65ig/overlay/lib/python3.10/site-packages/numpy/distutils/ccompiler.py", line 369, in CCompiler_compile
            single_compile(o)
          File "/tmp/pip-build-env-sylm65ig/overlay/lib/python3.10/site-packages/numpy/distutils/ccompiler.py", line 328, in single_compile
            self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
          File "/tmp/pip-build-env-sylm65ig/overlay/lib/python3.10/site-packages/numpy/distutils/ccompiler.py", line 91, in <lambda>
            m = lambda self, *args, **kw: func(self, *args, **kw)
          File "/tmp/pip-build-env-sylm65ig/overlay/lib/python3.10/site-packages/numpy/distutils/unixccompiler.py", line 58, in UnixCCompiler__compile
            raise CompileError(msg) from None
        distutils.errors.CompileError: Command "aarch64-unknown-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -c test_program.c -o objects/test_program.o" failed with exit status 127
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/pip/_internal/operations/build/metadata.py", line 35, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(metadata_dir)
  File "/usr/lib/python3.10/site-packages/pip/_internal/utils/misc.py", line 722, in prepare_metadata_for_build_wheel
    return super().prepare_metadata_for_build_wheel(
  File "/usr/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 186, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
  File "/usr/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 311, in _call_hook
    self._subprocess_runner(
  File "/usr/lib/python3.10/site-packages/pip/_internal/utils/subprocess.py", line 237, in runner
    call_subprocess(
  File "/usr/lib/python3.10/site-packages/pip/_internal/utils/subprocess.py", line 209, in call_subprocess
    raise error
pip._internal.exceptions.InstallationSubprocessError: Preparing metadata (pyproject.toml) exited with 1

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/piptools/scripts/compile.py", line 470, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/usr/lib/python3.10/site-packages/piptools/resolver.py", line 604, in resolve
    is_resolved = self._do_resolve(
  File "/usr/lib/python3.10/site-packages/piptools/resolver.py", line 636, in _do_resolve
    resolver.resolve(
  File "/usr/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
  File "/usr/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/usr/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/usr/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
  File "/usr/lib/python3.10/site-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
    return bool(self._sequence)
  File "/usr/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 174, in __bool__
    return any(self)
  File "/usr/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 162, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "/usr/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 53, in _iter_built
    candidate = func()
  File "/usr/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 187, in _make_candidate_from_link
    base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
  File "/usr/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 233, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/usr/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 304, in __init__
    super().__init__(
  File "/usr/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 159, in __init__
    self.dist = self._prepare()
  File "/usr/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 236, in _prepare
    dist = self._prepare_distribution()
  File "/usr/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 315, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "/usr/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 527, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/usr/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 642, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
  File "/usr/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 72, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "/usr/lib/python3.10/site-packages/pip/_internal/distributions/sdist.py", line 69, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/usr/lib/python3.10/site-packages/pip/_internal/req/req_install.py", line 575, in prepare_metadata
    self.metadata_directory = generate_metadata(
  File "/usr/lib/python3.10/site-packages/pip/_internal/operations/build/metadata.py", line 37, in generate_metadata
    raise MetadataGenerationFailed(package_details=details) from error
pip._internal.exceptions.MetadataGenerationFailed: metadata generation failed

[OddBloke]

I've got a working 2.4.0 build in OddBloke:oddbloke/kubeflow-pipelines-visualization-server-2.4.0, but I based that on #39639 so we're blocked until that lands.