From a3a1bf39da7b5a4f0ca6d8db40097ec88f7f815c Mon Sep 17 00:00:00 2001 From: lucalavezzo Date: Tue, 30 Jan 2024 15:39:38 -0500 Subject: [PATCH 1/3] added PnR projects --- project_metadata.yml | 1 + projects/pnr-cicd-automation.yml | 38 ++++++++++++++++++++++++++++++ projects/pnr-smart-job-retries.yml | 37 +++++++++++++++++++++++++++++ 3 files changed, 76 insertions(+) create mode 100644 projects/pnr-cicd-automation.yml create mode 100644 projects/pnr-smart-job-retries.yml diff --git a/project_metadata.yml b/project_metadata.yml index 432d2e13..f543a0b 100644 --- a/project_metadata.yml +++ b/project_metadata.yml @@ -6,6 +6,7 @@ categories: - ML/AI - Analytics - Networking + - Computing durations: - 3 months diff --git a/projects/pnr-cicd-automation.yml b/projects/pnr-cicd-automation.yml new file mode 100644 index 0000000..2d4e29f --- /dev/null +++ b/projects/pnr-cicd-automation.yml @@ -0,0 +1,38 @@ +--- +name: CI/CD and Automation of Manual Operations +postdate: 2024-01-30 +categories: + - Computing +durations: + - 3 months +experiments: + - CMS +skillset: + - Python + - CI/CD +status: + - In progress +project: + - IRIS-HEP +location: + - Any +commitment: + - Full time +program: + - IRIS-HEP fellow +shortdescription: Automate manual operations and implement CI/CD for CMS Production & Reprocessing group. +description: > + The Production & Reprocessing (P&R) group is responsible for maintaining and operating the CMS central workload management system, which processes hundreds of physics workflows daily which produce the data that physicists use in their analyses. The requests which have a similar physics goal are grouped as ‘campaigns’. P&R manages the lifecycle of hundreds of campaigns, each with its unique needs. The objective of this project is to automate the checks that are performed manually on these campaigns. This involves creating a system to automatically set up, verify, and activate new campaigns, along with managing data storage and allocation. The second part of the project focuses on implementing a Continuous Integration and Continuous Deployment (CI/CD) pipeline for efficiently deploying and maintaining software services. This will include converting manual testing procedures into automated ones, improving overall efficiency and reducing errors. Tools such as Gitlab Pipelines for CI/CD, Python for scripting, and various automated testing frameworks will be employed. This initiative is designed to streamline operations, making them more efficient and effective. +contacts: + - name: Hassan Ahmed + email: m.hassan@cern.ch + - name: Hasan Ozturk + email: h.ozturk@cern.ch + - name: Luca Lavezzo + email: luca.marco.lavezzo@cern.ch + - name: Jennifer Adelman McCarthy + email: jennifer.kathryn.adelman-mc.carthy@cern.ch + - name: Zhangqier Wang + email: wangzqe@mit.edu + - name: Dmytro Kovalskyi + email: kdv@mit.edu \ No newline at end of file diff --git a/projects/pnr-smart-job-retries.yml b/projects/pnr-smart-job-retries.yml new file mode 100644 index 0000000..1356c7b --- /dev/null +++ b/projects/pnr-smart-job-retries.yml @@ -0,0 +1,37 @@ +--- +name: Smart job retries for CMS workload management system +postdate: 2024-01-30 +categories: + - Computing +durations: + - 3 months +experiments: + - CMS +skillset: + - Python +status: + - In progress +project: + - IRIS-HEP +location: + - Any +commitment: + - Full time +program: + - IRIS-HEP fellow +shortdescription: Develop a tool to monitor and make smart decisions on how to retry CMS grid jobs. +description: > + The CMS experiment runs its data processing and simulation jobs on the Worldwide LHC Computing Grid in the scale of ~100k jobs in parallel. It’s inevitable to avoid job failures on this scale, and thus it’s crucial to have an effective failure recovery system. The existing algorithm is agnostic to the information of other jobs which run at the same site or belong to the same physics class. The objective of this project is to develop a tool which will monitor all the CMS grid jobs and make smart decisions on how to retry them by aggregating the data coming from different jobs across the globe. Such decisions can potentially be: reducing the job submission to computing sites experiencing particular failures, changing the job configuration in case of inaccurate configurations, and not retrying potentially ill-configured jobs. This project has the potential to significantly improve efficiency of the whole CMS computing grid, reducing the wasted cpu cycles and increasing the overall throughput. +contacts: + - name: Hassan Ahmed + email: m.hassan@cern.ch + - name: Hasan Ozturk + email: h.ozturk@cern.ch + - name: Luca Lavezzo + email: luca.marco.lavezzo@cern.ch + - name: Jennifer Adelman McCarthy + email: jennifer.kathryn.adelman-mc.carthy@cern.ch + - name: Zhangqier Wang + email: wangzqe@mit.edu + - name: Dmytro Kovalskyi + email: kdv@mit.edu \ No newline at end of file From 2471778fd4055a67dabeb0cead5fdbb2c6c1bd4a Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Tue, 30 Jan 2024 20:42:22 +0000 Subject: [PATCH 2/3] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- projects/pnr-cicd-automation.yml | 4 ++-- projects/pnr-smart-job-retries.yml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/projects/pnr-cicd-automation.yml b/projects/pnr-cicd-automation.yml index 2d4e29f..92916e2 100644 --- a/projects/pnr-cicd-automation.yml +++ b/projects/pnr-cicd-automation.yml @@ -1,5 +1,5 @@ --- -name: CI/CD and Automation of Manual Operations +name: CI/CD and Automation of Manual Operations postdate: 2024-01-30 categories: - Computing @@ -35,4 +35,4 @@ contacts: - name: Zhangqier Wang email: wangzqe@mit.edu - name: Dmytro Kovalskyi - email: kdv@mit.edu \ No newline at end of file + email: kdv@mit.edu diff --git a/projects/pnr-smart-job-retries.yml b/projects/pnr-smart-job-retries.yml index 1356c7b..3707bf2 100644 --- a/projects/pnr-smart-job-retries.yml +++ b/projects/pnr-smart-job-retries.yml @@ -21,7 +21,7 @@ program: - IRIS-HEP fellow shortdescription: Develop a tool to monitor and make smart decisions on how to retry CMS grid jobs. description: > - The CMS experiment runs its data processing and simulation jobs on the Worldwide LHC Computing Grid in the scale of ~100k jobs in parallel. It’s inevitable to avoid job failures on this scale, and thus it’s crucial to have an effective failure recovery system. The existing algorithm is agnostic to the information of other jobs which run at the same site or belong to the same physics class. The objective of this project is to develop a tool which will monitor all the CMS grid jobs and make smart decisions on how to retry them by aggregating the data coming from different jobs across the globe. Such decisions can potentially be: reducing the job submission to computing sites experiencing particular failures, changing the job configuration in case of inaccurate configurations, and not retrying potentially ill-configured jobs. This project has the potential to significantly improve efficiency of the whole CMS computing grid, reducing the wasted cpu cycles and increasing the overall throughput. + The CMS experiment runs its data processing and simulation jobs on the Worldwide LHC Computing Grid in the scale of ~100k jobs in parallel. It’s inevitable to avoid job failures on this scale, and thus it’s crucial to have an effective failure recovery system. The existing algorithm is agnostic to the information of other jobs which run at the same site or belong to the same physics class. The objective of this project is to develop a tool which will monitor all the CMS grid jobs and make smart decisions on how to retry them by aggregating the data coming from different jobs across the globe. Such decisions can potentially be: reducing the job submission to computing sites experiencing particular failures, changing the job configuration in case of inaccurate configurations, and not retrying potentially ill-configured jobs. This project has the potential to significantly improve efficiency of the whole CMS computing grid, reducing the wasted cpu cycles and increasing the overall throughput. contacts: - name: Hassan Ahmed email: m.hassan@cern.ch @@ -34,4 +34,4 @@ contacts: - name: Zhangqier Wang email: wangzqe@mit.edu - name: Dmytro Kovalskyi - email: kdv@mit.edu \ No newline at end of file + email: kdv@mit.edu From 842822977a0f3897c92787817c21b8d99a5028d0 Mon Sep 17 00:00:00 2001 From: lucalavezzo Date: Wed, 31 Jan 2024 10:20:45 -0500 Subject: [PATCH 3/3] fix status --- projects/pnr-cicd-automation.yml | 2 +- projects/pnr-smart-job-retries.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/projects/pnr-cicd-automation.yml b/projects/pnr-cicd-automation.yml index 2d4e29f..f9fdebb 100644 --- a/projects/pnr-cicd-automation.yml +++ b/projects/pnr-cicd-automation.yml @@ -11,7 +11,7 @@ skillset: - Python - CI/CD status: - - In progress + - Available project: - IRIS-HEP location: diff --git a/projects/pnr-smart-job-retries.yml b/projects/pnr-smart-job-retries.yml index 1356c7b..2aa6611 100644 --- a/projects/pnr-smart-job-retries.yml +++ b/projects/pnr-smart-job-retries.yml @@ -10,7 +10,7 @@ experiments: skillset: - Python status: - - In progress + - Available project: - IRIS-HEP location: