Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add upgrade tests to test main against previous release #6690

Closed
wants to merge 1 commit into from
Closed

Add upgrade tests to test main against previous release #6690

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions test/e2e-common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,3 +169,10 @@ function delete_pipeline_resources() {
function delete_resolvers_resources() {
kubectl delete --ignore-not-found=true resolutionrequests.resolution.tekton.dev --all
}

function get_tests_from_release() {
previous_release=$1
PREVIOUS_BRANCH=release-${previous_release%.*}.x
echo ">> Retrieving tests from PREVIOUS_BRANCH"
git checkout $PREVIOUS_BRANCH
}
36 changes: 35 additions & 1 deletion test/e2e-tests-upgrade.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,19 +16,40 @@

# This script calls out to scripts in tektoncd/plumbing to setup a cluster
# and deploy Tekton Pipelines to it for running upgrading tests. There are
# two scenarios we need to cover in this script:
# three scenarios we need to cover in this script:

# Scenario 1: install the previous release, upgrade to the current release, and
# validate whether the Tekton pipeline works.

# Scenario 2: install the previous release, create the pipelines and tasks, upgrade
# to the current release, and validate whether the Tekton pipeline works.

# Scenario 3: install the current pipelines, test the current main devel against
# the integration tests from the previous release(s) to prevent regression. Incompatible
# changes made in the current release cycle are expected to fail and will be skipped after
# failing once.

source $(git rev-parse --show-toplevel)/test/e2e-common.sh
RELEASES='https://github.com/tektoncd/pipeline/releases/latest'
PREVIOUS_PIPELINE_VERSION=$(curl -L -s -H 'Accept: application/json' $RELEASES |
sed -e 's/.*"tag_name":"\([^"]*\)".*/\1/')

# BACKWARD_INCOMPATIBLE_TESTS_FILES, TESTS and EXAMPLES are manually maintained list of e2e tests that contains
# known breaking changes made in the current devel cycle. They are updated to prevent the upgrade test job from
# keeping failing after known breaking changes are made.
BACKWARD_INCOMPATIBLE_EXAMPLES=()
# BACKWARD_INCOMPATIBLE_TESTS is a flag pattern for 'go test e2e' command to skip known breacking changes
# ie: BACKWARD_INCOMPATIBLE_TESTS="TestDuplicatePodTaskRun|TestPipelineRunCustomTaskTimeout"
BACKWARD_INCOMPATIBLE_TESTS=""

function skip_breaking_examples {
for TEST in "${BACKWARD_INCOMPATIBLE_EXAMPLES[@]}"; do
echo "Remove $TEST since it is a breaking change at devel"
rm ./examples/v1/"$TEST".yaml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if instead of removing the examples tests from the local copy of the repo, would it instead make sense to update the examples tests to accept a flag for which tests should be skipped?

rm ./examples/v1beta1/"$TEST".yaml
done
}

# Script entry point.

if [ "${SKIP_INITIALIZE}" != "true" ]; then
Expand Down Expand Up @@ -76,6 +97,19 @@ go_test_e2e -timeout=20m ./test || failed=1
# Run the post-integration tests. We do not need to install the resources again, since
# they are installed before the upgrade. We verify if they still work, after going through
# the upgrade.

# Testing main against the latest release to detect regression.
header "Testing main against the latest release to detect regression"
install_pipeline_crd

get_tests_from_release $PREVIOUS_PIPELINE_VERSION

skip_breaking_examples

# Run the integration tests.
go_test_e2e -timeout=20m -skip=$BACKWARD_INCOMPATIBLE_TESTS ./test || failed=1

# Run the post-integration tests.
go_test_e2e -tags=examples -timeout=20m ./test/ || failed=1

(( failed )) && fail_test
Expand Down