diff --git a/docs/Process_CI.md b/docs/Process_CI.md index 49779b7f..5b59c6f3 100644 --- a/docs/Process_CI.md +++ b/docs/Process_CI.md @@ -1,113 +1,219 @@ -# Automatic or Manual Firmware Compilation +# Build actions for developers and maintainers -[![Build train](https://github.com/armbian/build/actions/workflows/build-train.yml/badge.svg)](https://github.com/armbian/build/actions/workflows/build-train.yml) +**Note**: Add ideas for process improvements as comments to the ticket: +https://armbian.atlassian.net/browse/AR-2429 -Generates kernels at code push if the code, patches or config were changed in any way. It is also triggered via cron in the middle of Central European Time (CET) night. +Manual executing permissions are tied to [release manager](https://github.com/orgs/armbian/teams/release-manager) role within Armbian organization. Do you [want to help and take this role](https://calendly.com/armbian/office-hours)? -![Build](images/build-train.png) +## Prepare Standard Support images for release (release manager) -The build train is executed only if there are changed kernels. When this happens, it also generates armbian-firmware, desktop and u-boot packages. If the build succeeds it pushes packages to the package repository and increments trunk build version. +[![Build Standard Support Images](https://github.com/armbian/os/actions/workflows/complete-artifact-matrix-standard-support.yml/badge.svg)](https://github.com/armbian/os/actions/workflows/complete-artifact-matrix-standard-support.yml) -- generates all changed kernels, -- generate all boot loaders for all supported hardware, -- generate desktop pacakages, -- generates armbian-firmware, armbian-zsh, armbian-config. +This build workflow is executed manually when making: -You can change source repository and you can change destination package repository to https://beta.armbian.com (default) or https://apt.armbian.com +- a set of images for specific device +- a set of images for specific maintainer +- a full set of stable release images (default) -Manual Executing rights: [Armbian project member](https://github.com/orgs/armbian/people) +**Notes**: +- this process prepares images for release without pushing them to the download pages +- you can only generate images that are defined in [targets-release-standard-support.yaml](https://github.com/armbian/os/blob/main/userpatches/targets-release-standard-support.yaml) build lists! +- images generation workflows are compiled and are pretty much the same, just with different defaults -# Official Images Compilation +### 1. Open [workflow](https://github.com/armbian/os/actions/workflows/complete-artifact-matrix-standard-support.yml) and click -[![Build Official Images](https://github.com/armbian/os/actions/workflows/build-images.yml/badge.svg)](https://github.com/armbian/os/actions/workflows/build-images.yml) +![Run Workflow](images/run-workflow.png) +### 2. Select board -![Build](images/build-images-ci.png) +![Workflow](images/complete-artifact-matrix-standard-support.png) -Regenerate predefined stable images with incrementing patch version for selected board. +### 3. Run workflow -Manual executing rights: [Armbian release manager](https://forum.armbian.com/staffapplications/application/11-release-manager/) +![Build](images/run-worflow-button.png) -# Smoke tests on hardware devices +**(Workflow takes around 15 minutes to complete)** -Smoke testing is preliminary testing to reveal simple failures severe enough to, for example, reject a prospective software release. Our test case is constructed of three steps: +Generated images are uploaded to incoming folder: https://rsync.armbian.com/incoming/ under **your GitHub username** and once they are confirmed working, please notify @igorpecovnik to move them to official download pages. Once images are moved to [main download section](https://www.armbian.com/download/), automation refreshes download pages index within 15-30 minutes. -![Smoke](images/smoke-tests.png) +### Aditional options -- powering test equipment, consistent from several network switches, power supplies and dozens of hardware platforms -- running upgrade, reboot, repository switch, reboot, ... tests in parallel -- uploading a test report as build artefact following by powering the devices off. +Generates stable images defined in [targets-release-standard-support.yaml](https://github.com/armbian/os/blob/main/userpatches/targets-release-standard-support.yaml). This file is [autogenerated](https://github.com/armbian/os/blob/main/.github/workflows/recreate-matrix.yml#L147-L438) from [targets-release-standard-support.template](https://github.com/armbian/os/blob/main/userpatches/targets-release-standard-support.template). -Manual Executing rights: [Armbian project member](https://github.com/orgs/armbian/people) +Recommended images are defined via [RegEx mapping file](https://github.com/armbian/os/blob/main/exposed.map). -# Automatic Pull Requests Labeler +![Standard support images](images/standard-support-images.png) -[![Automatic Labeler](https://github.com/armbian/build/actions/workflows/labeler.yml/badge.svg)](https://github.com/armbian/build/actions/workflows/labeler.yml) +We are generating several images for each download / hardware target. They are automatically sorted by sections: -Automatically label new pull request based on the paths of files which are being changed. Configuration file can be found in: +- Desktop releases +- Server and IOT releases +- Dedicated applications - .github/labeler.yml +Images generation can be customized: + +- Framework build branch + - main (make images from trunk) + - v24.5 (previous stable release) +- Version override (leave empty for automatic bump) +- Board (make images for one board only) +- Maintainer (make images for selected maintainer) + +## Prepare application images for release (release manager) + +[![Build Dedicated Application Images](https://github.com/armbian/os/actions/workflows/complete-artifact-matrix-apps.yml/badge.svg)](https://github.com/armbian/os/actions/workflows/complete-artifact-matrix-apps.yml) + +This build workflow is executed manually when making: + +- a set of application images for specific device +- a set of application images for specific maintainer +- a full set of application images (default) + +**Notes**: +- **application images are released 10-15 minutes after build finishes succesfully** +- you can only generate images for applications that are defined in [targets-release-apps.yaml](https://github.com/armbian/os/blob/main/userpatches/targets-release-apps.yaml) build lists! +- images generation workflows are compiled and are pretty much the same, just with different defaults + +### 1. Open [workflow](https://github.com/armbian/os/actions/workflows/complete-artifact-matrix-apps.yml) and click + +![Run Workflow](images/run-workflow.png) + +### 2. Select board + +![Workflow](images/complete-artifact-matrix-standard-support.png) + +### 3. Run workflow + +![Build](images/run-worflow-button.png) + +**(Workflow takes around 15 minutes to complete)** + +Generated images are hosted at GitHub and released at once. Automation refreshes download pages within 15-30 minutes after/if workflow finished succesfully. + +![Dedicated Application Images](images/dedicated-applications.png) + +### Aditional options + +Generates dedicated application images defined in [targets-release-apps.yaml](https://github.com/armbian/os/blob/main/userpatches/targets-release-apps.yaml). This file is [autogenerated](https://github.com/armbian/os/blob/main/.github/workflows/recreate-matrix.yml#L147-L438) from [targets-release-apps.template](https://github.com/armbian/os/blob/main/userpatches/targets-release-apps.template). (You always edit template) + +Images generation can be customized: + +- framework build branch + - main (make images from trunk) + - v24.5 (previous stable release) +- version override (use latest release number or leave empty for automatic bump) +- board (make images only for one board) +- maintainer (make images for selected maintainer) + +## Repository update (cronjob/release manager) + +This pulls packages from build framework OCI cache located at GitHub and from [various 3rd party repositories](https://github.com/armbian/os/wiki/Import-3rd-party-packages) such as Chrome, Chromium, Code, Discord, (latest) ZFS, Thunderbird, Zoom, ... and pushes them to: -# Manual Pull Requests rebase +- `apt.armbian.com` (only new packages are added) +- `beta.armbian.com` (whole repository is recreated from scratch) -[![Automatic Rebase](https://github.com/armbian/build/actions/workflows/rebase.yml/badge.svg)](https://github.com/armbian/build/actions/workflows/rebase.yml) +### 1. Open [workflow](https://github.com/armbian/os/actions/workflows/repository-update.yml) and click -Pull most recent code from master branch and put your work on top of your pull request. +![Run Workflow](images/run-workflow.png) -How to use it? Simply comment +Action is executed automatically when artifact generations completes. Or manually. - /rebase +### 2. Include [artifacts from generated image(s)](https://netcup.armbian.com/partial/) -to trigger the action. +When +- [ ] Add https://netcup.armbian.com/partial/ to stable repo + +is selected. -- [Advantages of Git Rebase](https://itnext.io/advantages-of-git-rebase-af3b5f5448c6), -- [Automatic Rebase Action origin](https://github.com/marketplace/actions/automatic-rebase). +### 3. Run workflow -# Automatic or Manual Desktops Test Builds +![Build](images/run-worflow-button.png) -[![Build All Desktops](https://github.com/armbian/build/actions/workflows/build-all-desktops.yml/badge.svg)](https://github.com/armbian/build/actions/workflows/build-all-desktops.yml) +**(Workflow takes around 60 minutes to complete)** -Generates all desktops for arm64 and x86 arhitecture to verify if they build correctly. Build is triggered every day, manually (by [any member of Armbian project](https://github.com/orgs/armbian/people)) or in pull requests if label "Desktop" is set. Aim of this test case is to find out if there are troubles in packages relations. +## Build all artifacts (cronjob) -- releases: bullseye, bookworm, jammy, -- desktop environments: xfce, gnome, mate, cinnamon, budgie, kde-plasma, -- builds are not using cached rootfs to force packages assembly, -- included applications paths are "3dsupport browsers", -- builds are done with [Docker image](https://github.com/orgs/armbian/packages?repo_name=build) on public runners. +[![Build All Artifacts](https://github.com/armbian/os/actions/workflows/complete-artifact-matrix-all.yml/badge.svg)](https://github.com/armbian/os/actions/workflows/complete-artifact-matrix-all.yml) -# Automatic Kernel Build at Pull Requests +Generates all build artifacts cache for targets defined in [targets-all-not-eos.yaml](https://github.com/armbian/os/blob/main/userpatches/targets-all-not-eos.yaml). This build job runs **every 8 hours** and can also be run manually when needed. -Generates kernels at Pull Requests if their code, patches or config was changed. Build starts when label of Pull Request is set to "Ready to review" +This build job **needs to be successfully completed** in order to proceed generating any OS images! -# Integrity testings +## Build Rolling Release Images (cronjob) -This action tests package integrity from all stable images at download section. +[![Build Nightly Images](https://github.com/armbian/os/actions/workflows/complete-artifact-matrix-nightly.yml/badge.svg)](https://github.com/armbian/os/actions/workflows/complete-artifact-matrix-nightly.yml) -Manual Executing rights: [Armbian project member](https://github.com/orgs/armbian/people) +Generates all nighly (Rolling Release) images defined in [targets-release-nightly.yaml](https://github.com/armbian/os/blob/main/userpatches/targets-release-nightly.yaml). This file is [autogenerated](https://github.com/armbian/os/blob/main/.github/workflows/recreate-matrix.yml#L147-L438) from [targets-release-nightly.template](https://github.com/armbian/os/blob/main/userpatches/targets-release-nightly.template) -# Forked Helper +This build job runs every day at 9 a.m. UTC and can also be run manually when needed. Download pages are refreshed [automatically](https://github.com/armbian/os/actions/workflows/webindex-update.yml) after successful build. -- Run repository dispatch to default fork branch -- Dispatch event on forked repostitory +![Build](images/rolling-releases.png) -# Lint On Scripts +## Watchdog (cronjob) + +Runs every 15 minutes and re-trigger [failed builds](https://github.com/armbian/os/blob/main/.github/workflows/watchdog.yml#L26) six (6) times before finally gives out. This addresses various instabilities when building many artifacts on different hardware: + +- network timeouts +- artifact download failure +- loop devices allocation failure +- runner running low on space + +## Smoke tests on hardware devices (release manager) + +Smoke testing is preliminary testing to reveal simple failures severe enough to, for example, reject a prospective software release. Our test case is constructed of three steps: + +![Smoke](images/smoke-tests.png) + +- powering test equipment, consistent from several network switches, power supplies and dozens of hardware platforms +- running upgrade, reboot, repository switch, reboot, ... tests in parallel +- uploading a test report as build artifact following by powering the devices off. + +## Automatic Pull Requests Labeler (PR) + +[![Automatic Labeler](https://github.com/armbian/build/actions/workflows/labeler.yml/badge.svg)](https://github.com/armbian/build/actions/workflows/labeler.yml) + +Automatically label new pull request based on the paths of files which are being changed. Configuration file can be found in: + + .github/labeler.yml + +## Full distro test builds (cronjob/release manager) + +[![Build Nightly Images](https://github.com/armbian/os/actions/workflows/full-distro-build-and-test.yml/badge.svg)](https://github.com/armbian/os/actions/workflows/full-distro-build-and-test.yml) + +Generates all supported build combinations (minimal, cli, desktops) for x86 architecture to check package level changes inconsistency and dependencies. + +Options: + +- Framework build branch + - **main** + - testing_branch (string) + +## Build all artifacts (admin/PR) + +Generates artifacts at Pull Requests code. Build starts when label of Pull Request is set to "Build". Requires administration privileges. + +## Lint on shell scripts (PR) + +[![Lint On Shell Scripts](https://github.com/armbian/build/actions/workflows/pr-lint-scripts.yml/badge.svg)](https://github.com/armbian/build/actions/workflows/pr-lint-scripts.yml) ![Lint](images/linterror.png) -Run [ShellCheck](https://github.com/koalaman/shellcheck) on changed shell scripts and report problems within. Since our scripts are full of shellcheck problems we don't block merging on those errors. Not yet. +Run [ShellCheck](https://github.com/koalaman/shellcheck) on changed shell scripts and report problems within. Linting runs automatically on pull requests. + +## Update tools in build scripts (cronjob/admin) -Linting is run automatically on pull requests change. +[![Update Tools in Scripts](https://github.com/armbian/build/actions/workflows/update-tools.yml/badge.svg)](https://github.com/armbian/build/actions/workflows/update-tools.yml) -# Scorecards Security Scan +Some of our scripts download tools from a repo. These cannot be bumped by Dependabot, so this workflow is a self-created Dependabot to bump versions of those tools to stay up-to-date. This workflow only creates a PR if the version was actually updated. To add a new tool, it just needs to be added to the matrix [in the script](https://github.com/armbian/build/blob/main/.github/workflows/update-tools.yml) by filling out all the variables. -Scorecards is an automated tool that assesses a number of important heuristics ("checks") associated with software security and assigns each check a score of 0-10. You can use these scores to understand specific areas to improve in order to strengthen the security posture of your project. You can also assess the risks that dependencies introduce, and make informed decisions about accepting these risks, evaluating alternative solutions, or working with the maintainers to make improvements. +## Scorecards security scan (PR) -https://github.com/ossf/scorecard#what-is-scorecards +[![Scorecards Security Scan](https://github.com/armbian/build/actions/workflows/scorecard.yml/badge.svg)](https://github.com/armbian/build/actions/workflows/scorecard.yml) -# Kernel hardening analysis +[Scorecards](https://github.com/ossf/scorecard#what-is-scorecards) is an automated tool that assesses a number of important heuristics ("checks") associated with software security and assigns each check a score of 0-10. You can use these scores to understand specific areas to improve in order to strengthen the security posture of your project. You can also assess the risks that dependencies introduce, and make informed decisions about accepting these risks, evaluating alternative solutions, or working with the maintainers to make improvements. -This analysis checks kernel config if changed. +## Kernel hardening analysis (PR) -There are plenty of security hardening options for the Linux kernel. A lot of them are not enabled by the major distros. We have to enable these options ourselves to make our systems more secure. +[![Kernel Hardening Analysis](https://github.com/armbian/build/actions/workflows/pr-kernel-security-analysis.yml/badge.svg)](https://github.com/armbian/build/actions/workflows/pr-kernel-security-analysis.yml) -https://github.com/a13xp0p0v/kconfig-hardened-check/blob/master/README.md +This [analysis](https://github.com/a13xp0p0v/kconfig-hardened-check/blob/master/README.md) checks kernel configs and run if changed. There are plenty of security hardening options for the Linux kernel. A lot of them are not enabled by the major distros. We have to enable these options ourselves to make our systems more secure. diff --git a/docs/images/complete-artifact-matrix-standard-support.png b/docs/images/complete-artifact-matrix-standard-support.png new file mode 100644 index 00000000..cad33284 Binary files /dev/null and b/docs/images/complete-artifact-matrix-standard-support.png differ diff --git a/docs/images/dedicated-applications.png b/docs/images/dedicated-applications.png new file mode 100644 index 00000000..10e3fb0e Binary files /dev/null and b/docs/images/dedicated-applications.png differ diff --git a/docs/images/rolling-releases.png b/docs/images/rolling-releases.png new file mode 100644 index 00000000..2ebcc281 Binary files /dev/null and b/docs/images/rolling-releases.png differ diff --git a/docs/images/run-worflow-button.png b/docs/images/run-worflow-button.png new file mode 100644 index 00000000..b69aa195 Binary files /dev/null and b/docs/images/run-worflow-button.png differ diff --git a/docs/images/run-workflow.png b/docs/images/run-workflow.png new file mode 100644 index 00000000..bf5d2225 Binary files /dev/null and b/docs/images/run-workflow.png differ diff --git a/docs/images/standard-support-images.png b/docs/images/standard-support-images.png new file mode 100644 index 00000000..659276c3 Binary files /dev/null and b/docs/images/standard-support-images.png differ