Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] docker release build failing in branch-22.06 #223

Closed
pdmack opened this issue Jun 27, 2022 · 1 comment
Closed

[BUG] docker release build failing in branch-22.06 #223

pdmack opened this issue Jun 27, 2022 · 1 comment
Assignees
@mdemoret-nv
Labels
bug Something isn't working

Comments

@pdmack
Copy link
Contributor

pdmack commented Jun 27, 2022

Describe the bug
./docker/build_container_release.sh fails to build from branch-22.06 at d35593c

#19 [conda_bld_morpheus 2/2] RUN --mount=type=cache,id=workspace_cache,target=/workspace/.cache,sharing=locked     --mount=type=cache,id=conda_pkgs,target=/opt/conda/pkgs,sharing=locked     source activate base &&    MORPHEUS_ROOT=/workspace MORPHEUS_BUILD_PYTHON_STUBS=OFF CONDA_BLD_PATH=/opt/conda/conda-bld CONDA_ARGS="--no-test" ./ci/conda/recipes/run_conda_build.sh morpheus
#19 19.89 /opt/conda/lib/python3.8/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.9) or chardet (5.0.0)/charset_normalizer (2.0.12) doesn't match a supported version!
#19 19.89   warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
#19 19.89 No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.16
#19 19.89 WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.16
#19 19.94 Cloning into bare repository '/opt/conda/conda-bld/git_cache/workspace'...
#19 20.06 done.
#19 20.08 Cloning into '/opt/conda/conda-bld/morpheus-split-22.04.00/work'...
#19 20.11 done.
#19 20.28 Downloading morpheus/data/bert-base-cased-hash.txt (756 KB)
#19 20.30 Error downloading object: morpheus/data/bert-base-cased-hash.txt (aa22859): Smudge error: Error downloading morpheus/data/bert-base-cased-hash.txt (aa228593db3f947d342f85e234ff939b5c141b3c4a4956359697a14b40cba081): EOF
#19 20.31
#19 20.31 Errors logged to /opt/conda/conda-bld/morpheus-split-22.04.00/work/.git/lfs/logs/20220627T211902.745753875.log
#19 20.31 Use `git lfs logs last` to view the log.
#19 20.31 error: external filter 'git-lfs filter-process' failed
#19 20.31 fatal: morpheus/data/bert-base-cased-hash.txt: smudge filter lfs failed
#19 20.31 warning: Clone succeeded, but checkout failed.
#19 20.31 You can inspect what was checked out with 'git status'
#19 20.31 and retry with 'git restore --source=HEAD :/'
#19 20.31
#19 20.32 Updating build index: /opt/conda/conda-bld
#19 20.32
#19 20.32 Warning: failed to download source.  If building, will try again after downloading recipe dependencies.
#19 20.32 Error was:
#19 20.32 Command '['/usr/bin/git', 'clone', '/opt/conda/conda-bld/git_cache/workspace', '/opt/conda/conda-bld/morpheus-split-22.04.00/work']' returned non-zero exit status 128.
#19 20.32 Adding in variants from internal_defaults
#19 20.32 INFO:conda_build.variants:Adding in variants from internal_defaults
#19 20.32 Adding in variants from /workspace/ci/conda/recipes/morpheus/conda_build_config.yaml
#19 20.32 INFO:conda_build.variants:Adding in variants from /workspace/ci/conda/recipes/morpheus/conda_build_config.yaml
#19 20.32 Adding in variants from argument_variants
#19 20.32 INFO:conda_build.variants:Adding in variants from argument_variants
#19 20.43 From /workspace
#19 20.43  * [new ref]                    -> _conda_cache_origin_head
#19 20.45 Cloning into '/opt/conda/conda-bld/morpheus-split-22.04.00/work'...
#19 20.48 done.
#19 20.66 Downloading morpheus/data/bert-base-cased-hash.txt (756 KB)
#19 20.69 Error downloading object: morpheus/data/bert-base-cased-hash.txt (aa22859): Smudge error: Error downloading morpheus/data/bert-base-cased-hash.txt (aa228593db3f947d342f85e234ff939b5c141b3c4a4956359697a14b40cba081): EOF
#19 20.69
#19 20.69 Errors logged to /opt/conda/conda-bld/morpheus-split-22.04.00/work/.git/lfs/logs/20220627T211903.128637099.log
#19 20.69 Use `git lfs logs last` to view the log.
#19 20.69 error: external filter 'git-lfs filter-process' failed
#19 20.69 fatal: morpheus/data/bert-base-cased-hash.txt: smudge filter lfs failed
#19 20.69 warning: Clone succeeded, but checkout failed.
#19 20.69 You can inspect what was checked out with 'git status'
#19 20.69 and retry with 'git restore --source=HEAD :/'
#19 20.69
#19 20.80 Warning: failed to download source.  If building, will try again after downloading recipe dependencies.
#19 20.80 Error was:
#19 20.80 Command '['/usr/bin/git', 'clone', '/opt/conda/conda-bld/git_cache/workspace', '/opt/conda/conda-bld/morpheus-split-22.04.00/work']' returned non-zero exit status 128.
#19 20.80 Attempting to finalize metadata for morpheus
#19 20.80 /opt/conda/lib/python3.8/site-packages/conda_build/environ.py:444: UserWarning: The environment variable 'CMAKE_CUDA_ARCHITECTURES' is being passed through with value 'ALL'.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
#19 20.80   warnings.warn(
#19 20.80 /opt/conda/lib/python3.8/site-packages/conda_build/environ.py:444: UserWarning: The environment variable 'MORPHEUS_BUILD_PYTHON_STUBS' is being passed through with value 'OFF'.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
#19 20.80   warnings.warn(
#19 20.80 /opt/conda/lib/python3.8/site-packages/conda_build/environ.py:444: UserWarning: The environment variable 'MORPHEUS_CACHE_DIR' is being passed through with value '/workspace/.cache'.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
#19 20.80   warnings.warn(
#19 20.80 /opt/conda/lib/python3.8/site-packages/conda_build/environ.py:444: UserWarning: The environment variable 'PARALLEL_LEVEL' is being passed through with value '80'.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
#19 20.80   warnings.warn(
#19 20.80 INFO:conda_build.metadata:Attempting to finalize metadata for morpheus
#19 20.81 Error: Failed to render jinja template in /workspace/ci/conda/recipes/morpheus/meta.yaml:
#19 20.81 'GIT_DESCRIBE_NUMBER' is undefined
#19 ERROR: executor failed running [/bin/bash -c source activate base &&    MORPHEUS_ROOT=/workspace MORPHEUS_BUILD_PYTHON_STUBS=OFF CONDA_BLD_PATH=/opt/conda/conda-bld CONDA_ARGS="--no-test" ./ci/conda/recipes/run_conda_build.sh morpheus]: exit code: 1
------
 > [conda_bld_morpheus 2/2] RUN --mount=type=cache,id=workspace_cache,target=/workspace/.cache,sharing=locked     --mount=type=cache,id=conda_pkgs,target=/opt/conda/pkgs,sharing=locked     source activate base &&    MORPHEUS_ROOT=/workspace MORPHEUS_BUILD_PYTHON_STUBS=OFF CONDA_BLD_PATH=/opt/conda/conda-bld CONDA_ARGS="--no-test" ./ci/conda/recipes/run_conda_build.sh morpheus:
#19 20.80   warnings.warn(
#19 20.80 /opt/conda/lib/python3.8/site-packages/conda_build/environ.py:444: UserWarning: The environment variable 'MORPHEUS_BUILD_PYTHON_STUBS' is being passed through with value 'OFF'.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
#19 20.80   warnings.warn(
#19 20.80 /opt/conda/lib/python3.8/site-packages/conda_build/environ.py:444: UserWarning: The environment variable 'MORPHEUS_CACHE_DIR' is being passed through with value '/workspace/.cache'.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
#19 20.80   warnings.warn(
#19 20.80 /opt/conda/lib/python3.8/site-packages/conda_build/environ.py:444: UserWarning: The environment variable 'PARALLEL_LEVEL' is being passed through with value '80'.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
#19 20.80   warnings.warn(
#19 20.80 INFO:conda_build.metadata:Attempting to finalize metadata for morpheus
#19 20.81 Error: Failed to render jinja template in /workspace/ci/conda/recipes/morpheus/meta.yaml:
#19 20.81 'GIT_DESCRIBE_NUMBER' is undefined
------
error: failed to solve: executor failed running [/bin/bash -c source activate base &&    MORPHEUS_ROOT=/workspace MORPHEUS_BUILD_PYTHON_STUBS=OFF CONDA_BLD_PATH=/opt/conda/conda-bld CONDA_ARGS="--no-test" ./ci/conda/recipes/run_conda_build.sh morpheus]: exit code: 1

Steps/Code to reproduce bug

docker buildx build --no-cache --progress plain -t nvcr.io/nvidia/morpheus/morpheus:v22.04.00-runtime --target runtime --build-arg FROM_IMAGE=gpuci/miniforge-cuda --build-arg CUDA_VER=11.5 --build-arg LINUX_DISTRO=ubuntu --build-arg LINUX_VER=20.04 --build-arg RAPIDS_VER=22.06 --build-arg PYTHON_VER=3.8 --build-arg TENSORRT_VERSION=8.2.1.3 --network=host  -f docker/Dockerfile .

Expected behavior
Successful build.

Environment overview (please complete the following information)

  • Environment location: [Bare-metal, Docker]
  • Method of Morpheus build: [Docker]

Environment details
Please run and paste the output of the morpheus/scripts/print_env.sh script here, to gather any other relevant environment details

Additional context
Add any other context about the problem here.
@pdmack pdmack added bug Something isn't working Needs Triage Need team to review and classify labels Jun 27, 2022
@mdemoret-nv mdemoret-nv mentioned this issue Jun 27, 2022
Merged
@mdemoret-nv mdemoret-nv removed the Needs Triage Need team to review and classify label Jun 27, 2022
ghost pushed a commit that referenced this issue Jun 28, 2022
@mdemoret-nv
Use git version 2.35.3 in release build (#224)
8a40730 
Forces the docker build to use git version 2.35.3. Otherwise conda build breaks during release container builds.

Fixes #223

Authors:
  - Michael Demoret (https://github.com/mdemoret-nv)

Approvers:
  - https://github.com/bsuryadevara

URL: #224
@mdemoret-nv
Copy link
Contributor

mdemoret-nv commented Jun 28, 2022 •
edited
Loading

Not sure why this was not closed with PR #224

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mdemoret-nv mdemoret-nv

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pdmack @mdemoret-nv