Pip causes problems when installing package(s) in docker image if kaniko cache is activated. #1158

christianbeland · 2020-03-25T11:05:59Z

Pip causes problems when installing package(s) in docker image if kaniko cache is activated.

Actual behavior
I want to build following image:

ARG docker_registry
FROM ${docker_registry}tensorflow/tensorflow:1.13.2-gpu-py3

USER root
RUN mkdir -p ~/.pip
ADD deployment/pip.conf ~/.pip/pip.conf

RUN pip3 install --upgrade pip

ADD requirements.txt requirements.txt
RUN echo "pip3 install requirements.txt"
RUN pip3 install -r requirements.txt

RUN mkdir /dist /app
ADD dist/${PACKAGE_NAME} /dist/
RUN echo "pip3 install /dist/*.whl"
RUN pip3 install /dist/*.whl --target /app

ENV PYTHONPATH "${PYTHONPATH}:/app"

Building using:

   /kaniko/executor --context $CI_PROJECT_DIR \
             --dockerfile $BASE_PATH/deployment/Dockerfile \
             --destination ${DOCKER_REGISTRY}/$FULL_IMAGE_NAME \
             --build-arg docker_registry="$DOCKER_REGISTRY_MIRROR/" \
             --build-arg http_proxy=$http_proxy \
             --build-arg https_proxy=$https_proxy \
             --build-arg no_proxy=$no_proxy \
             --insecure \
             --skip-tls-verify \
             --insecure-registry=${DOCKER_REGISTRY} \
             --cache=true \
             --cache-dir=$BASE_PATH/cache/

Causes:

INFO[0093] RUN echo "pip3 install requirements.txt"     
INFO[0093] cmd: /bin/bash                               
INFO[0093] args: [-c echo "pip3 install requirements.txt"] 
pip3 install requirements.txt
INFO[0093] Taking snapshot of full filesystem...        
INFO[0093] Resolving paths                              
INFO[0097] Pushing layer <repo>/<image_name>/cache:aa2b060b29af65575a6c64c9df27cd498e894526042ef0e700751a7df042ed29 to cache now 
INFO[0097] RUN pip3 install -r requirements.txt         
INFO[0097] cmd: /bin/bash                               
INFO[0097] args: [-c pip3 install -r requirements.txt]  
Collecting click
  Downloading click-7.1.1-py2.py3-none-any.whl (82 kB)
Collecting tokenization
  Downloading tokenization-1.0.7-py3-none-any.whl (10 kB)
Collecting cbor>=0.1.4
  Downloading cbor-1.0.0.tar.gz (20 kB)
Collecting typing
  Downloading typing-3.7.4.1-py3-none-any.whl (25 kB)
Collecting regex
  Downloading regex-2020.2.20-cp36-cp36m-manylinux2010_x86_64.whl (690 kB)
Building wheels for collected packages: cbor
  Building wheel for cbor (setup.py): started
  Building wheel for cbor (setup.py): finished with status 'done'
  Created wheel for cbor: filename=cbor-1.0.0-cp36-cp36m-linux_x86_64.whl size=51512 sha256=8d6ce77b82f6b6fc5d3e83f0eaa1b89f03357a0539430ebd033ee692b5624dd7
  Stored in directory: /root/.cache/pip/wheels/14/62/c6/8c93de62510c9c7a9c6d5b3a09fd74b4c3943ec0e32c431753
Successfully built cbor
Installing collected packages: click, regex, tokenization, cbor, typing
ERROR: Error checking for conflicts.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 3021, in _dep_map
    return self.__dep_map
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 3012, in _parsed_pkg_info
    return self._pkg_info
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
    raise AttributeError(attr)
AttributeError: _pkg_info

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/commands/install.py", line 517, in _warn_about_conflicts
    package_set, _dep_info = check_install_conflicts(to_install)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/check.py", line 114, in check_install_conflicts
    package_set, _ = create_package_set_from_installed()
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/check.py", line 53, in create_package_set_from_installed
    package_set[name] = PackageDetails(dist.version, dist.requires())
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 2736, in requires
    dm = self._dep_map
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 3023, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 3032, in _compute_dependencies
    for req in self._parsed_pkg_info.get_all('Requires-Dist') or []:
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 3014, in _parsed_pkg_info
    metadata = self.get_metadata(self.PKG_INFO)
  File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 1895, in get_metadata
    raise KeyError("No metadata except PKG-INFO is available")
KeyError: 'No metadata except PKG-INFO is available'
Successfully installed cbor-1.0.0 click-7.1.1 regex-2020.2.20 tokenization-1.0.7 typing-3.7.4.1
INFO[0102] Taking snapshot of full filesystem...        
INFO[0102] Resolving paths

The image gets pushed to the repository but I cannot pull it. I receive following error:

Failed to pull image "<repo>/<image_name>:0.1.dev65-g9165108": rpc error: code = Unknown desc = failed to register layer: Error processing tar file(exit status 1): failed to mknod("/usr/bin/pip3", S_IFCHR, 0): file exists

Expected behavior
If I comment out --cache=true and --cache-dir= from the kaniko build command, everything works fine. The build also works without problems using docker.

The text was updated successfully, but these errors were encountered:

swist · 2020-03-25T21:35:09Z

Hitting the same issue intermittently on most versions from 0.17.1 to 0.19.0

hochreitercorpuls · 2020-03-27T12:09:04Z

we are also experiencing this issue resulting in exactly the same behaviour in our pipelines.

Hitting the same issue intermittently on most versions from 0.17.1 to 0.19.0

are you saying that 0.16 is safe?

tejal29 · 2020-04-02T06:43:17Z

@christianbeland i am able to reproduce this error at my end. Looking into it now,.

tejal29 · 2020-04-02T17:58:08Z

@christianbeland , i tried your dockerfile on latest master.
Looks like this is fixed now.
Can you please try,

gcr.io/kaniko-project/executor:edge
gcr.io/kaniko-project/executor:debug-edge

The dockerfile i used is

FROM tensorflow/tensorflow:1.13.2-gpu-py3
  
USER root
RUN mkdir -p ~/.pip

RUN pip3 install --upgrade pip

ADD requirements.txt requirements.txt
RUN echo "pip3 install requirements.txt"
RUN pip3 install -r requirements.txt

RUN mkdir /dist /app
ADD dist/*.whl /dist/
RUN echo "pip3 install /dist/*.whl"
RUN pip3 install /dist/*.whl --target /app

ENV PYTHONPATH "${PYTHONPATH}:/app"

christianbeland · 2020-04-02T19:05:43Z

@tejal29 I just tried and still have the issue.

You may have to try the build twice. I noticed the first build often work.

tejal29 · 2020-04-02T19:12:42Z

okay let me do that. Thanks for confirming so fast

christianbeland · 2020-04-09T11:42:51Z

You should probably remove the 'fixed-needs-verfication' tag.

astleychen · 2020-04-21T00:45:20Z

I ran into same issue here and confirmed it's fixed on gcr.io/kaniko-project/executor:debug-edge.

christianbeland · 2020-04-23T13:10:20Z

I tested gcr.io/kaniko-project/executor:debug-edge again this morning and still does not work.

I get:

in _compute_dependencies for req in self._parsed_pkg_info.get_all('Requires-Dist') or []: File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3014, in _parsed_pkg_info metadata = self.get_metadata(self.PKG_INFO) File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1895, in get_metadata raise KeyError("No metadata except PKG-INFO is available") KeyError: 'No metadata except PKG-INFO is available'

figroc · 2021-07-12T08:07:12Z

Has this been fixed or not? We currently use 1.6.0 but encounter the same issue.

TheMatrix97 · 2024-03-20T00:24:32Z

Facing a similar issue here. In my case if cache=true it throws the following error when I try to run python -m pip install kubernetes==11.0.0 with python3.11

Processing triggers for libc-bin (2.36-9+deb12u4) ...
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.11/README.venv for more information.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 1

The docker image is based in ubuntu:22.04

If I set the attribute cache=false the image is built without any issue...
Just as a note, I'm using executor v1.14.0-debug if I try to use the latest one v1.21.1-debug it fails for both cache = false and true....

Any idea? Thanks!

TheMatrix97 · 2024-03-20T07:02:31Z

Facing a similar issue here. In my case if cache=true it throws the following error when I try to run python -m pip install kubernetes==11.0.0 with python3.11

Processing triggers for libc-bin (2.36-9+deb12u4) ...
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.11/README.venv for more information.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 1

The docker image is based in ubuntu:22.04

If I set the attribute cache=false the image is built without any issue... Just as a note, I'm using executor v1.14.0-debug if I try to use the latest one v1.21.1-debug it fails for both cache = false and true....

Any idea? Thanks!

Well... I just solved it, I'm running different builds on the same container for different Dockerfile, and I was missing the --cleanup flag, adding this fixes the issue

ch9hn · 2024-05-22T09:05:19Z

Hello,
we are facing exact the same issue - we can't install prebuild Python Wheels with Kaniko.
With Docker (BuildKit) its working fine.
We also try to disable the cache and run it also with --cleanup, but had no luck.

Our Kaniko Version is 1.21.0-debug, execution via Gitlab CI/CD.

INFO[0579] Taking snapshot of full filesystem...        
INFO[0735] RUN --mount=type=bind,from=flash-attn-builder,src=/usr/src/flash-attention-v2,target=/usr/src/flash-attention-v2     --mount=type=cache,target=/root/.cache/pip     pip install /usr/src/flash-attention-v2/*.whl --no-cache-dir     && python3 -m pip cache purge      && rm -rf /tmp/* RUN ldconfig /usr/local/cuda-11.8/compat/ 
INFO[0735] Cmd: /bin/sh                                 
INFO[0735] Args: [-c pip install /usr/src/flash-attention-v2/*.whl --no-cache-dir     && python3 -m pip cache purge      && rm -rf /tmp/* RUN ldconfig /usr/local/cuda-11.8/compat/] 
INFO[0735] Running: [/bin/sh -c pip install /usr/src/flash-attention-v2/*.whl --no-cache-dir     && python3 -m pip cache purge      && rm -rf /tmp/* RUN ldconfig /usr/local/cuda-11.8/compat/] 
WARNING: Requirement '/usr/src/flash-attention-v2/*.whl' looks like a filename, but the file does not exist
ERROR: *.whl is not a valid wheel filename.
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 1

ch9hn · 2024-05-22T12:31:54Z

Found the source of that issue:
#1568

The unsupported options are not failing the build, the files are not mounted and therefore Python can't install the wheel.

daverin · 2024-08-19T19:13:21Z

With cache enabled I'm getting the following error thrown from poetry:

AttributeError
557 |  
558 | module 'virtualenv.create.via_global_ref.builtin.cpython.mac_os' has no attribute 'CPython3macOsBrew'
559 |  
560 | at /opt/bitnami/python/lib/python3.11/site-packages/importlib_metadata/__init__.py:186 in load
561 | 182│         """
562 | 183│         match = cast(Match, self.pattern.match(self.value))
563 | 184│         module = import_module(match.group('module'))
564 | 185│         attrs = filter(None, (match.group('attr') or '').split('.'))
565 | →  186│         return functools.reduce(getattr, attrs, module)
566 | 187│
567 | 188│     @property
568 | 189│     def module(self) -> str:
569 | 190│         match = self.pattern.match(self.value)
570

According to stack overflow this issue is resolvable by updating virtualenv package. Any idea whats going on here ?

christianbeland changed the title ~~Pip cannot install package in docker image when cache is activated.~~ Pip causes problems when installing package(s) in docker image if cache is activated. Mar 25, 2020

christianbeland changed the title ~~Pip causes problems when installing package(s) in docker image if cache is activated.~~ Pip causes problems when installing package(s) in docker image if kaniko cache is activated. Mar 25, 2020

tejal29 self-assigned this Mar 27, 2020

hochreitercorpuls mentioned this issue Mar 28, 2020

Repeated builds using cache produce broken images #1162

Closed

tejal29 added the fixed-needs-verfication label Apr 2, 2020

tejal29 removed the fixed-needs-verfication label Apr 10, 2020

tejal29 added needs-reproduction area/caching For all bugs related to cache issues labels May 7, 2020

tejal29 added this to the Release v1.1.0 milestone May 7, 2020

aaron-prindle unassigned tejal29 Jun 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pip causes problems when installing package(s) in docker image if kaniko cache is activated. #1158

Pip causes problems when installing package(s) in docker image if kaniko cache is activated. #1158

christianbeland commented Mar 25, 2020 •

edited

Loading

swist commented Mar 25, 2020

hochreitercorpuls commented Mar 27, 2020 •

edited

Loading

tejal29 commented Apr 2, 2020

tejal29 commented Apr 2, 2020 •

edited

Loading

christianbeland commented Apr 2, 2020

tejal29 commented Apr 2, 2020

christianbeland commented Apr 9, 2020

astleychen commented Apr 21, 2020

christianbeland commented Apr 23, 2020

figroc commented Jul 12, 2021

TheMatrix97 commented Mar 20, 2024 •

edited

Loading

TheMatrix97 commented Mar 20, 2024 •

edited

Loading

ch9hn commented May 22, 2024

ch9hn commented May 22, 2024 •

edited

Loading

daverin commented Aug 19, 2024

Pip causes problems when installing package(s) in docker image if kaniko cache is activated. #1158

Pip causes problems when installing package(s) in docker image if kaniko cache is activated. #1158

Comments

christianbeland commented Mar 25, 2020 • edited Loading

swist commented Mar 25, 2020

hochreitercorpuls commented Mar 27, 2020 • edited Loading

tejal29 commented Apr 2, 2020

tejal29 commented Apr 2, 2020 • edited Loading

christianbeland commented Apr 2, 2020

tejal29 commented Apr 2, 2020

christianbeland commented Apr 9, 2020

astleychen commented Apr 21, 2020

christianbeland commented Apr 23, 2020

figroc commented Jul 12, 2021

TheMatrix97 commented Mar 20, 2024 • edited Loading

TheMatrix97 commented Mar 20, 2024 • edited Loading

ch9hn commented May 22, 2024

ch9hn commented May 22, 2024 • edited Loading

daverin commented Aug 19, 2024

christianbeland commented Mar 25, 2020 •

edited

Loading

hochreitercorpuls commented Mar 27, 2020 •

edited

Loading

tejal29 commented Apr 2, 2020 •

edited

Loading

TheMatrix97 commented Mar 20, 2024 •

edited

Loading

TheMatrix97 commented Mar 20, 2024 •

edited

Loading

ch9hn commented May 22, 2024 •

edited

Loading