Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

♻️ [CI] Improved Kapua CI workflow artifacts and dependency caching #4179

Merged
merged 10 commits into from
Jan 24, 2025

Conversation

Coduz
Copy link
Contributor

@Coduz Coduz commented Jan 24, 2025

Improved the way and what gets cached as dependencies and built artifacts in used GitHub Actions workflows

Related Issue
None

Description of the solution adopted
Instead of having a cache per-build, which means that for each run we are downloading all dependencies for a job (unless restarting the same job), we switched to have 2 caches:

Cache for external dependencies

Named Linux-maven-{branchName}-dependencies, which will cache all .m2/repository but the artifacts under .m2/repository/org/eclipse/kapua.

This cache will be per release branch/develop and can be reused by all workflows and for all branches that start from a release branch or develop.

A TODO is to have this cache have a time reference, like {year}-{month}, appended to it to make it rebuilt from time to time with updated dependencies.

Since filtering with exclude paths for action/cache is not working properly (see actions/toolkit#713), exclusion is made by moving the folder outside of the maven repository.

Cache for built artifacts
Named Linux-maven-{github_action_run_id}-kapua-artifacts, which will cache Kapua artifacts built during the run. This cache will be reused by all job from the same run test and other tasks

This cache is restored and then moved to .m2/repository/org/eclipse/kapua directory so it can be detected and reused by Maven

Benefit of this are:

  • External dependencies are not download for every build.
  • Cache size for each build is ~500MB for each run for built artifact and one that gets reused by all workflows of 400MB with external dependencies.
    • Before, cache entry for each build :
      Screenshot 2025-01-24 at 11 27 43
    • After, cache for each build + common external dependencies cache:
      Screenshot 2025-01-24 at 11 28 04
  • All workflow have a unique cache for external depedendencies. Before we each workflow had its own cache key:
    • Sonar: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }}
    • Owasp: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }} but JDK setup was enabling cache, duplicating it
    • KapuaCI: {{ github.run_id }}-${{ github.run_number }}-maven-cache

Screenshots
None

Any side note on the changes made
Some clean up and optimization of jobs and action has been done to simplify management and avoid duplication of common parts

Coduz added 10 commits January 23, 2025 16:14
Signed-off-by: Alberto Codutti <alberto.codutti@eurotech.com>
Signed-off-by: Alberto Codutti <alberto.codutti@eurotech.com>
Signed-off-by: Alberto Codutti <alberto.codutti@eurotech.com>
Signed-off-by: Alberto Codutti <alberto.codutti@eurotech.com>
Signed-off-by: Alberto Codutti <alberto.codutti@eurotech.com>
Signed-off-by: Alberto Codutti <alberto.codutti@eurotech.com>
Signed-off-by: Alberto Codutti <alberto.codutti@eurotech.com>
Signed-off-by: Alberto Codutti <alberto.codutti@eurotech.com>
Signed-off-by: Alberto Codutti <alberto.codutti@eurotech.com>
Signed-off-by: Alberto Codutti <alberto.codutti@eurotech.com>
@Coduz Coduz force-pushed the feat-improvedGitHubActionCaching branch from cc06fb7 to 3d876da Compare January 24, 2025 10:23
@Coduz Coduz merged commit 4a6c79b into eclipse-kapua:develop Jan 24, 2025
26 of 27 checks passed
@Coduz Coduz added the Enhancement This PR/Issue improves an part of Kapua label Jan 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement This PR/Issue improves an part of Kapua
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant