Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Setup CI to use new Jenkins Build #102

Closed
mdemoret-nv opened this issue May 13, 2022 · 0 comments · Fixed by #80
Closed

[FEA] Setup CI to use new Jenkins Build #102

mdemoret-nv opened this issue May 13, 2022 · 0 comments · Fixed by #80
Assignees
@dagardner-nv
Labels
feature request New feature or request

Comments

@mdemoret-nv
Copy link
Contributor

Is your feature request related to a problem? Please describe.
In transitioning to Github, much of our CI system was lost. The style checks, build and tests need to be re-enabled using the new Jenkins build from RAPIDS.

Describe the solution you'd like
Convert the previous CI system to use the new Jenkins build from RAPIDS.
@mdemoret-nv mdemoret-nv mentioned this issue May 13, 2022
Merged
9 tasks
@ghost ghost closed this as completed in #80 May 23, 2022
ghost pushed a commit that referenced this issue May 23, 2022
@dagardner-nv
Jenkins integration (#80)
c52cb24 
Updates conda scripts to uses sccache (ccache work-alike with an S3 backend)
Adds 4 Jenkins stages:
1. check
1. build
1. test
1. docs build

Results are publised into S3 with a 30 day retention police and are available via:
`https://downloads.rapids.ai/ci/morpheus/pull-request/<pull-req id>/<git commit>/<arch>/`
ex:
https://downloads.rapids.ai/ci/morpheus/pull-request/80/5161d30673f4d5fb34452df5a8ac6f68b8462685/x86_64/

Intermediate Artifacts pushed to S3:
* conda_env.tar.gz - Morpheus conda environment used by test & documentation stages
* workspace.tar.bz - Morpheus workspace used by test & documentation stages

Results pushed to S3:
* docs.tar.bz - Sphinx HTML documentation
* report_pytest.xml - Pytest junit report
* report_pytest_coverage.xml - Pytest coverage report
* conda_build.tar.bz - Morpheus conda build (not currently working)

Conda builds for cudf are cached seperately (even with sscache this step was still took 17 minutes).
S3 cache location at:
`https://downloads.rapids.ai/ci/morpheus/cudf/<cuda ver>/<python ver>/<rapids ver>/<latest git commit to ci/conda>/<arch>cudf_conda.tar.bz`
ex:
https://downloads.rapids.ai/ci/morpheus/cudf/11.4/3.8/21.10/b7f728e6db4fc6124970935f461a6791818bfb1d/x86_64/cudf_conda.tar.bz

TODO List:

- [x] Need a test stage
- [x] Need to run style checks
- [x] Need to add sccache
- [x] Need to cache cudf conda packages
- [x] Need to be able to build without using a gpu
- [x] Output junit style xml
- [x] Output docs build
- [ ] Add conda build step
- [ ] Need to figure out a way to cache lfs

Closes #102
Closes #104

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #80
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dagardner-nv dagardner-nv

Labels
feature request New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Jenkins integration dagardner-nv/Morpheus
2 participants
@mdemoret-nv @dagardner-nv