Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update k8schain #4488

Merged
merged 1 commit into from
Jan 21, 2022
Merged

Update k8schain #4488

merged 1 commit into from
Jan 21, 2022

Conversation

imjasonh
Copy link
Member

@imjasonh imjasonh commented Jan 14, 2022

Changes

This picks up a change to k8schain that drops a dependency on forked K8s
cred providers nominally maintained by Vincent.

See google/go-containerregistry#1234

* use go 1.17 semantics in go.mod
go get -u github.com/google/go-containerregistry/pkg/authn/k8schain@main
* revert k8s upgade that breaks knative/pkg
* undo go 1.17 change in go.mod
./hack/update-deps.sh

/kind cleanup
/assign vdemeester
/assign sbwsg

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

  • Docs included if any changes are user facing
  • Tests included if any functionality added or changed
  • Follows the commit message standard
  • Meets the Tekton contributor standards (including
    functionality, content, code)
  • Release notes block below has been filled in or deleted (only if no user facing changes)

Release Notes

Drops dependency on k8s cred providers, statically link GCR/ECR/ACR cred helpers' Go code instead

@tekton-robot tekton-robot assigned ghost Jan 14, 2022
@tekton-robot tekton-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. labels Jan 14, 2022
@tekton-robot tekton-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jan 14, 2022
@tekton-robot tekton-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 17, 2022
@imjasonh
Copy link
Member Author

Something weird is happening with k8schain hanging trying to pull from a local registry.

I added some test code to just pull the pipeline image:

        log.Println("------ pulling image with k8schain...")
        if _, err := remote.Image(r, remote.WithAuthFromKeychain(kc)); err != nil {
                t.Fatalf("Pulling %s: %v", r, err)
        }
        log.Println("------ PULLED IMAGE WITH K8SCHAIN")

...and it blocks for minutes before eventually failing and proceeding with anonymous auth (which succeeds):

2022/01/17 14:57:15 ------ pulling image with k8schain...
time="2022-01-17T14:57:15-05:00" level=error msg="Error parsing the serverURL" error="docker-credential-ecr-login can only be used with Amazon Elastic Container Registry." serverURL="127.0.0.1:59503/testreconcile_remotepipelineref"
2022/01/17 15:04:09 GET /v2/
2022/01/17 15:04:09 GET /v2/testreconcile_remotepipelineref/manifests/latest
2022/01/17 15:04:09 ------ PULLED IMAGE WITH K8SCHAIN

The log line is a bit of a red herring, it's just telling us that 127.0.0.1:59503 isn't ECR (which it isn't!), the mystery wait happens after that line is logged.

@ghost
Copy link

ghost commented Jan 18, 2022

...and it blocks for minutes before eventually failing and proceeding with anonymous auth (which succeeds):

This sounds a lot like the behaviour from #4087

If you try wrapping it in a runtime trace I am curious if you will see a similar thing as we did in that issue - a series of very long dialConnFor calls that eventually all seem to time out.

@imjasonh
Copy link
Member Author

Tracked the problematic behavior to the ACR cred helper, with a fix proposed in chrismellard/docker-credential-acr-env#2

With this change, the test passes as expected: imjasonh/pipeline@k8schain...imjasonh:k8schain-with-fix

@imjasonh
Copy link
Member Author

Tracked the problematic behavior to the ACR cred helper, with a fix proposed in chrismellard/docker-credential-acr-env#2

With this change, the test passes as expected: imjasonh/pipeline@k8schain...imjasonh:k8schain-with-fix

Update: the ACR cred helper PR has merged 🎉 , and google/go-containerregistry#1247 is now open to bring it in to k8schain -- as before, with that PR replaced in go.mod, the test passes: imjasonh/pipeline@k8schain...imjasonh:k8schain-with-fix

When that merges I'll remove the replace, go get -u, and push to this branch, which should pass all tests 🤞

@tekton-robot tekton-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 20, 2022
```
* use go 1.17 semantics in go.mod
go get -u github.com/google/go-containerregistry/pkg/authn/k8schain@main
* revert k8s upgade that breaks knative/pkg
* undo go 1.17 change in go.mod
./hack/update-deps.sh
```
@tekton-robot tekton-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 20, 2022
@imjasonh
Copy link
Member Author

/test pull-tekton-pipeline-integration-tests

@imjasonh
Copy link
Member Author

All the dependent PRs are merged, this should be ready to go now. 🚀

@ghost
Copy link

ghost commented Jan 20, 2022

/lgtm

@tekton-robot tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 20, 2022
@ghost
Copy link

ghost commented Jan 20, 2022

Tested locally without the AWS_ env vars in the controller config and didn't see a ten minute slowdown on the first TaskRun execution. Nice work! I think we can safely remove those again. For reference I used the examples/v1beta1/taskruns/no-ci/tekton-bundles.yaml example to test with.

One thing I did notice is the following log line printed with each taskrun execution. It would be awesome if it didnt print this but I don't think it's a blocker at all:

{"level":"info","ts":"2022-01-20T20:46:53.450Z","logger":"tekton-pipelines-controller.event-broadcaster","caller":"record/event.go:282","msg":"Event(v1.ObjectReference{Kind:\"TaskRun\", Namespace:\"default\", Name:\"remote-task-reference\", UID:\"602c57e3-2b4f-4032-80d2-af9f6cc0848
e\", APIVersion:\"tekton.dev/v1beta1\", ResourceVersion:\"13457267\", FieldPath:\"\"}): type: 'Normal' reason: 'Started' ","commit":"ccdfb72"}
time="2022-01-20T20:46:53Z" level=error msg="Error parsing the serverURL" error="docker-credential-ecr-login can only be used with Amazon Elastic Container Registry." serverURL=index.docker.io/ptasci67/example-oci

@imjasonh
Copy link
Member Author

One thing I did notice is the following log line printed with each taskrun execution. It would be awesome if it didnt print this but I don't think it's a blocker at all:

Yeah I've also been a bit annoyed by that. I might try to fix this either in ggcr or upstream in the ecr-login cred helper. Until then it's just noisy and not (usually!) indicative of a real error.

@tekton-robot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vdemeester

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tekton-robot tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 21, 2022
@tekton-robot tekton-robot merged commit adf0305 into tektoncd:main Jan 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lgtm Indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants