fix: check ephemeral metadata is set before delete

[com6056]

We're seeing issues with rollout sync failing due to errors like this:

roCtx.reconcile err Operation cannot be fulfilled on pods \"REDACTED\": the object has been modified; please apply your changes to the latest version and try again"

It seems that we're trying to delete the metadata on every pod every sync, even if the pod doesn't have the keys. This means that in environments with high pod churn, this loop can get stuck for a very long time. This simply adds a check to make sure the key is actually present in the metadata before we delete it and mark it as modified.

Before the fix with the new test:

go test ./utils/replicaset/...
--- FAIL: TestSyncEphemeralPodMetadata (0.00s)
    canary_test.go:1353:
        	Error Trace:	/Users/jrodgers/argo-rollouts/utils/replicaset/canary_test.go:1353
        	Error:      	Should be false
        	Test:       	TestSyncEphemeralPodMetadata
time="2025-01-30T13:43:28-08:00" level=warning msg="Failed to determine existing ephemeral metadata from annotation: foo"
time="2025-01-30T13:43:28-08:00" level=info msg="ComputePodTemplateHash hash changed (new hash: 6c8776dbbb, old hash: fcb6c6ff9)" namespace=default rollout=red
time="2025-01-30T13:43:28-08:00" level=info msg="Pod template change detected (new: 8657b54cf, old: different-hash)" namespace=default rollout=nginx
time="2025-01-30T13:43:28-08:00" level=info msg="Canary steps change detected (new: 5ffbfbbd64, old: different-hash)" namespace=default rollout=nginx
time="2025-01-30T13:43:28-08:00" level=warning msg="Unable to convert ReplicaSet revision to int: strconv.Atoi: parsing \"not-an-int\": invalid syntax" ReplicaSet= namespace= rollout=ro
time="2025-01-30T13:43:28-08:00" level=warning msg="Unable to convert ReplicaSet revision to int: strconv.Atoi: parsing \"not-an-int\": invalid syntax" ReplicaSet= namespace= rollout=ro
FAIL
FAIL	github.com/argoproj/argo-rollouts/utils/replicaset	0.685s
FAIL

After the fix with the new test:

go test ./utils/replicaset/...
ok  	github.com/argoproj/argo-rollouts/utils/replicaset	(cached)

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this is a chore.
• The title of the PR is (a) conventional with a list of types and scopes found here, (b) states what changed, and (c) suffixes the related issues number. E.g. "fix(controller): Updates such and such. Fixes #1234".
• I've signed my commits with DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My builds are green. Try syncing with master if they are not.
• My organization is added to USERS.md.

[github-actions]

Published E2E Test Results

  4 files    4 suites   3h 8m 18s ⏱️
113 tests 106 ✅  7 💤 0 ❌
452 runs  424 ✅ 28 💤 0 ❌

Results for commit 289a3eb.

♻️ This comment has been updated with latest results.

[github-actions]

Published Unit Test Results

2 294 tests   2 294 ✅  2m 59s ⏱️
  128 suites      0 💤
    1 files        0 ❌

Results for commit 289a3eb.

♻️ This comment has been updated with latest results.

[agaudreault]

Code can be clearer if the lookup check that the variable exists

Suggested change

	if _, ok := metadata.Annotations[k]; !ok {
	continue
	}
	delete(metadata.Annotations, k)
	modified = true
	if _, ok := metadata.Annotations[k]; ok {
	delete(metadata.Annotations, k)
	modified = true
	}

[com6056]

I prefer to keep the code left and return/continue early, but happy to change it if you think it's better https://medium.com/@matryer/line-of-sight-in-code-186dd7cdea88

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[aneff-figma]

awesome