Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: pip release 23.2 breaks ucc-gen #808

Closed
guilhemmarchand opened this issue Jul 15, 2023 · 4 comments
Closed

BUG: pip release 23.2 breaks ucc-gen #808

guilhemmarchand opened this issue Jul 15, 2023 · 4 comments
Labels
bug Something isn't working

Comments

@guilhemmarchand
Copy link

Description

On Saturday 15th of July 2023 was released pip version 23.2 on pypi and it became the default.

When running ucc-gen, the new release of pip is automatically used and ucc-gen systematically fails to generate any package with:

Building wheels for collected packages: splunk-sdk
  Building wheel for splunk-sdk (setup.py) ... done
  Created wheel for splunk-sdk: filename=splunk_sdk-1.7.3-py3-none-any.whl size=125267 sha256=1968a464bc5315ec9a66e5a2aa84305d81207325a6e47c9f981623d23044098c
  Stored in directory: /home/gmarchand/.cache/pip/wheels/85/7d/26/d4c4d6e96b70a9809ac667d226b9d2fe4cce761a9c435e03e3
Successfully built splunk-sdk
ERROR: Error while checking for conflicts. Please file an issue on pip's issue tracker: https://github.com/pypa/pip/issues/new
Traceback (most recent call last):
  File "/home/gmarchand/.local/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 575, in _determine_conflicts
    return check_install_conflicts(to_install)
  File "/home/gmarchand/.local/lib/python3.10/site-packages/pip/_internal/operations/check.py", line 108, in check_install_conflicts
    would_be_installed = _simulate_installation_of(to_install, package_set)
  File "/home/gmarchand/.local/lib/python3.10/site-packages/pip/_internal/operations/check.py", line 131, in _simulate_installation_of
    dist = abstract_dist.get_metadata_distribution()
  File "/home/gmarchand/.local/lib/python3.10/site-packages/pip/_internal/distributions/wheel.py", line 23, in get_metadata_distribution
    assert self.req.local_file_path, "Set as part of preparation during download"
AssertionError: Set as part of preparation during download
Installing collected packages: PySocks, charset-normalizer, idna, urllib3, certifi, requests, defusedxml, sortedcontainers, splunk-sdk, solnlib, splunktalib, splunktaucclib, pip
ERROR: Exception:
Traceback (most recent call last):
  File "/home/gmarchand/.local/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
  File "/home/gmarchand/.local/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 248, in wrapper
    return func(self, options, args)
  File "/home/gmarchand/.local/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 452, in run
    installed = install_given_reqs(
  File "/home/gmarchand/.local/lib/python3.10/site-packages/pip/_internal/req/__init__.py", line 72, in install_given_reqs
    requirement.install(
  File "/home/gmarchand/.local/lib/python3.10/site-packages/pip/_internal/req/req_install.py", line 805, in install
    assert self.local_file_path
AssertionError

I have tried many various things, all unsuccessful for now, to prevent this pip version from being used by ucc-gen.
The same bevhiour can be observed on Mac & Linux, clearing pip cache, uninstall or other trick does help as ucc-gen systematically tries to acquire the latest version available on-line.

Super annoying and blocking issue, is there any way to force ucc-gen behaviour regarding the pip version it uses?(requirements.txt seems to be ignored in that case?)

What ucc-gen version are you using?

splunk-add-on-ucc-framework==5.28.2

Additional System Info

Tested on different Mac & Linux OS, Python 3.10.6
@guilhemmarchand guilhemmarchand added the bug Something isn't working label Jul 15, 2023
@guilhemmarchand
Copy link
Author

I could locate the following ucc-gen behaviour, hard coded:

https://github.com/splunk/addonfactory-ucc-generator/blob/main/splunk_add_on_ucc_framework/install_python_libraries.py

The issue happening here:

def install_libraries(
    requirements_file_path: str,
    installation_path: str,
    installer: str,
):
    """
    Upgrades `pip` version to the latest one and installs requirements to the
    specified path.
    """
    pip_update_command = f"{installer} -m pip install pip --upgrade"
    pip_install_command = (
        f"{installer} "
        f"-m pip "
        f"install "
        f'-r "{requirements_file_path}" '
        f"--no-compile "
        f"--prefer-binary "
        f"--ignore-installed "
        f"--use-deprecated=legacy-resolver "
        f'--target "{installation_path}"'
    )

By temporarily forcing the pip upgrade to be restricted to the previous release of pip, the build work again:

pip_update_command = f"{installer} -m pip install pip --upgrade pip==23.1.2"

Not a permanent workaround obviously, I do not know what makes the process to be failing systematically with latest release of pip.

Here are the traces of my successful build process bypassing the upgrade of pip to 23.2:

2023-07-15 20:02:04,466 - INFO - **** TrackMe package generation, version="2.0.45" ****
2023-07-15 20:02:04,466 - INFO - version_data="{'version': '2.0.45', 'appID': 'trackme'}"
2023-07-15 20:02:04,467 - INFO - **** TrackMe app generation, appID="trackme" ****
2023-07-15 20:02:04,470 - INFO - Call ucc-gen
INFO: ucc-gen version 5.28.2 is used
INFO: Python binary name to use: python3
INFO: Output folder is /home/gmarchand/git/trackme-app/output
INFO: Add-on will be built with version '2.0.45'
INFO: Cleaned out directory /home/gmarchand/git/trackme-app/output
INFO: Using globalConfig file located @ /home/gmarchand/git/trackme-app/globalConfig.json
INFO: globalConfig file is valid
INFO: Updated and saved add-on version in the globalConfig file to 2.0.45
INFO: Current globalConfig schema version is 0.0.3
INFO: Copied UCC template directory
INFO: Copied globalConfig to output
INFO: Installing requirements from package/lib/requirements.txt
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: pip in /home/gmarchand/.local/lib/python3.10/site-packages (23.1.2)
Collecting pip
  Downloading pip-23.2-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 22.6 MB/s eta 0:00:00

[notice] A new release of pip is available: 23.1.2 -> 23.2
[notice] To update, run: python3 -m pip install --upgrade pip
Collecting splunktaucclib>=5.0.4 (from -r package/lib/requirements.txt (line 1))
  Using cached splunktaucclib-6.0.9-py3-none-any.whl (104 kB)
Collecting pip==23.1.2 (from -r package/lib/requirements.txt (line 2))
  Using cached pip-23.1.2-py3-none-any.whl (2.1 MB)
Collecting PySocks<2.0.0,>=1.7.1 (from splunktaucclib>=5.0.4->-r package/lib/requirements.txt (line 1))
  Using cached PySocks-1.7.1-py3-none-any.whl (16 kB)
Collecting requests<3.0.0,>=2.31.0 (from splunktaucclib>=5.0.4->-r package/lib/requirements.txt (line 1))
  Using cached requests-2.31.0-py3-none-any.whl (62 kB)
Collecting solnlib<5.0.0,>=4.11.2 (from splunktaucclib>=5.0.4->-r package/lib/requirements.txt (line 1))
  Using cached solnlib-4.11.2-py3-none-any.whl (59 kB)
Processing /home/gmarchand/.cache/pip/wheels/85/7d/26/d4c4d6e96b70a9809ac667d226b9d2fe4cce761a9c435e03e3/splunk_sdk-1.7.3-py3-none-any.whl (from splunktaucclib>=5.0.4->-r package/lib/requirements.txt (line 1))
Collecting splunktalib<4.0.0,>=3.0.4 (from splunktaucclib>=5.0.4->-r package/lib/requirements.txt (line 1))
  Using cached splunktalib-3.0.4-py3-none-any.whl (52 kB)
Collecting urllib3<2 (from splunktaucclib>=5.0.4->-r package/lib/requirements.txt (line 1))
  Using cached urllib3-1.26.16-py2.py3-none-any.whl (143 kB)
Collecting charset-normalizer<4,>=2 (from requests<3.0.0,>=2.31.0->splunktaucclib>=5.0.4->-r package/lib/requirements.txt (line 1))
  Using cached charset_normalizer-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (201 kB)
Collecting idna<4,>=2.5 (from requests<3.0.0,>=2.31.0->splunktaucclib>=5.0.4->-r package/lib/requirements.txt (line 1))
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17 (from requests<3.0.0,>=2.31.0->splunktaucclib>=5.0.4->-r package/lib/requirements.txt (line 1))
  Using cached certifi-2023.5.7-py3-none-any.whl (156 kB)
Collecting defusedxml<0.8.0,>=0.7.1 (from solnlib<5.0.0,>=4.11.2->splunktaucclib>=5.0.4->-r package/lib/requirements.txt (line 1))
  Using cached defusedxml-0.7.1-py2.py3-none-any.whl (25 kB)
Collecting sortedcontainers<3.0,>=2.2 (from solnlib<5.0.0,>=4.11.2->splunktaucclib>=5.0.4->-r package/lib/requirements.txt (line 1))
  Using cached sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
Installing collected packages: PySocks, charset-normalizer, idna, urllib3, certifi, requests, defusedxml, sortedcontainers, splunk-sdk, solnlib, splunktalib, splunktaucclib, pip
Successfully installed PySocks-1.7.1 certifi-2023.5.7 charset-normalizer-3.2.0 defusedxml-0.7.1 idna-3.4 pip-23.1.2 requests-2.31.0 solnlib-4.11.2 sortedcontainers-2.4.0 splunk-sdk-1.7.3 splunktalib-3.0.4 splunktaucclib-6.0.9 urllib3-1.26.16

[notice] A new release of pip is available: 23.1.2 -> 23.2
[notice] To update, run: python3 -m pip install --upgrade pip
INFO:   removing directory /home/gmarchand/git/trackme-app/output/trackme/lib/bin from /home/gmarchand/git/trackme-app/output/trackme/lib
INFO:   removing directory /home/gmarchand/git/trackme-app/output/trackme/lib/pip-23.1.2.dist-info from /home/gmarchand/git/trackme-app/output/trackme/lib
INFO:   removing directory /home/gmarchand/git/trackme-app/output/trackme/lib/pip from /home/gmarchand/git/trackme-app/output/trackme/lib
INFO:   fixing /home/gmarchand/git/trackme-app/output/trackme/lib/charset_normalizer/md.cpython-310-x86_64-linux-gnu.so execute bit
INFO:   fixing /home/gmarchand/git/trackme-app/output/trackme/lib/charset_normalizer/md__mypyc.cpython-310-x86_64-linux-gnu.so execute bit
INFO: Installed add-on requirements into /home/gmarchand/git/trackme-app/output/trackme/lib from package
INFO: Generating alerts code
INFO: status="success", operation="Add default setting", alert_name="trackme_smart_status", "python.version"="python3"
INFO: status="success", operation="Add default setting", alert_name="trackme_smart_status", "is_custom"="1"
INFO: status="success", operation="Add default setting", alert_name="trackme_smart_status", "payload_format"="json"
INFO: status="success", operation="Add default setting", alert_name="trackme_smart_status", "icon_path"="alerticon.png"
INFO: status="success", operation="Add default setting", alert_name="trackme_auto_ack", "python.version"="python3"
INFO: status="success", operation="Add default setting", alert_name="trackme_auto_ack", "is_custom"="1"
INFO: status="success", operation="Add default setting", alert_name="trackme_auto_ack", "payload_format"="json"
INFO: status="success", operation="Add default setting", alert_name="trackme_auto_ack", "icon_path"="alerticon.png"
INFO: status="success", operation="Add default setting", alert_name="trackme_free_style_rest_call", "python.version"="python3"
INFO: status="success", operation="Add default setting", alert_name="trackme_free_style_rest_call", "is_custom"="1"
INFO: status="success", operation="Add default setting", alert_name="trackme_free_style_rest_call", "payload_format"="json"
INFO: status="success", operation="Add default setting", alert_name="trackme_free_style_rest_call", "icon_path"="alerticon.png"
INFO: status="success", operation="Add default setting", alert_name="trackme_notable", "python.version"="python3"
INFO: status="success", operation="Add default setting", alert_name="trackme_notable", "is_custom"="1"
INFO: status="success", operation="Add default setting", alert_name="trackme_notable", "payload_format"="json"
INFO: status="success", operation="Add default setting", alert_name="trackme_notable", "icon_path"="alerticon.png"
INFO: status="starting", operation="generate", object="alert_actions.conf", object_type="file"
INFO: status="success", operation="generate", object="alert_actions.conf", object_type="file"
INFO: status="starting", operation="generate", object="alert_actions.conf.spec", object_type="file"
INFO: status="success", operation="generate", object="alert_actions.conf.spec", object_type="file"
INFO: status="starting", operation="generate", object="eventtypes.conf", object_type="file"
INFO: status="success", operation="generate", object="eventtypes.conf", object_type="file"
INFO: status="starting", operation="generate", object="tags.conf", object_type="file"
INFO: status="success", operation="generate", object="tags.conf", object_type="file"
INFO: Started generating alert actions HTML files
INFO: Generating HTML file for 'trackme_smart_status'
INFO: Creating file @ /home/gmarchand/git/trackme-app/output/trackme/default/data/ui/alerts/trackme_smart_status.html
INFO: Generating HTML file for 'trackme_auto_ack'
INFO: Creating file @ /home/gmarchand/git/trackme-app/output/trackme/default/data/ui/alerts/trackme_auto_ack.html
INFO: Generating HTML file for 'trackme_free_style_rest_call'
INFO: Creating file @ /home/gmarchand/git/trackme-app/output/trackme/default/data/ui/alerts/trackme_free_style_rest_call.html
INFO: Generating HTML file for 'trackme_notable'
INFO: Creating file @ /home/gmarchand/git/trackme-app/output/trackme/default/data/ui/alerts/trackme_notable.html
INFO: Finished generating alert actions HTML files
INFO: operation="Generate py file", alert_action="trackme_smart_status"
INFO: operation="Generate py file", alert_action="trackme_auto_ack"
INFO: operation="Generate py file", alert_action="trackme_free_style_rest_call"
INFO: operation="Generate py file", alert_action="trackme_notable"
INFO: Copied package directory
INFO: Updated VERSION file
INFO: Updated app.manifest file in the output folder
INFO: Updated app.conf file in the output folder
INFO: Generating OpenAPI file
2023-07-15 20:02:10,918 - INFO - "appID="trackme", Achive tar file creation, archive_file="../output/trackme_v2045_1689447728.tgz"
2023-07-15 20:02:10,919 - INFO - Get and store the sha256 control sum
2023-07-15 20:02:10,937 - INFO - sha256 control sum="44d64c788324e53428a1d445f810d13e1dda306fd01257c28adb5e5ecc81cde3"
2023-07-15 20:02:10,937 - INFO - **** TrackMe app generation terminated, appID="trackme", build_number="1689447728", sha256="44d64c788324e53428a1d445f810d13e1dda306fd01257c28adb5e5ecc81cde3" ****
2023-07-15 20:02:10,970 - INFO - version_data="{'version': '2.0.45', 'appID': 'trackme'}"
2023-07-15 20:02:10,970 - INFO - **** TrackMe app generation, appID="trackme" ****

dfederschmidt added a commit to dfederschmidt/addonfactory-ucc-generator that referenced this issue Jul 16, 2023
@dfederschmidt
fix: resolve failing ucc-gen by pinning pip version
e31efa7 
With the latest pip version, the build started failing because the pip update command is not
deterministic. Since this potentially broke existing pipelines, even though ucc-gen was not  updated

splunk#808
@dfederschmidt dfederschmidt mentioned this issue Jul 16, 2023
Merged
@dfederschmidt
Copy link
Contributor

Added some more findings here:
#809 (comment)

artemrys added a commit that referenced this issue Jul 17, 2023
@dfederschmidt @artemrys
fix: resolve failing ucc-gen by pinning pip version (#809)
33676a9 
* fix: resolve failing ucc-gen by pinning pip version

With the latest pip version, the build started failing because the pip update command is not
deterministic. Since this potentially broke existing pipelines, even though ucc-gen was not  updated

#808

* fix: adjust the pip installation command

---------

Co-authored-by: Artem Rys <rysartem@gmail.com>
srv-rr-github-token added a commit that referenced this issue Jul 17, 2023
@srv-rr-github-token
chore(release): 5.28.3
9def316 
## [5.28.3](v5.28.2...v5.28.3) (2023-07-17)

### Bug Fixes

* resolve failing ucc-gen by pinning pip version ([#809](#809)) ([33676a9](33676a9)), closes [#808](#808)
@artemrys
Copy link
Member

Version 5.28.3 was released with pip pinned to 23.1.2 version which fixes the issue for now. I'll investigate the issue further later this week.

@artemrys
Copy link
Member

I'll close this one, as fix was released and pip was also released with a fix for the same issue.

We will later investigate the removal of the deprecate functionality we rely on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dfederschmidt @guilhemmarchand @artemrys