Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bootstrap tokens can expire before they can be used if the bootstrap cluster's time is sufficiently behind the workload cluster #6029

Closed
joejulian opened this issue Feb 1, 2022 · 8 comments · Fixed by #7717
Closed

bootstrap tokens can expire before they can be used if the bootstrap cluster's time is sufficiently behind the workload cluster #6029

joejulian opened this issue Feb 1, 2022 · 8 comments · Fixed by #7717
Labels
area/bootstrap Issues or PRs related to bootstrap providers kind/bug Categorizes issue or PR as related to a bug. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale.

Comments

@joejulian
Copy link
Contributor

What steps did you take and what happened:

  1. Set the time on the bootstrap cluster 1 hour in the past. 
    date -s $(date -d "1 hour ago")
clusterctl init ...
  2. Create a cluster (any cluster that uses the kubeadm bootstrap provider)

What did you expect to happen:

I expected the bootstrap controller to emit an error event and log entry because it cannot create a token that won't be expired before it can be used. With the expiration time in the past, Kubernetes will immediately clean it up.

Anything else you would like to add:

A secret, eg. bootstrap-token-zuzhz4 is created in the workload cluster. Because the expiration time is in the past, TokenCleaner deletes the token immediately after creation. One side-effect of this is that the capi-kubeadm-bootstrap-controller-manager logs will fill with lines like:

I0131 22:37:39.210898       1 kubeadmconfig_controller.go:285] controller/kubeadmconfig "msg"="Refreshing token until the infrastructure has a chance to consume it" "name"="cluster-ibs-ng-control-plane-vctx7" "namespace"="default" "reconciler group"="bootstrap.cluster.x-k8s.io" "reconciler kind"="KubeadmConfig" 
E0131 22:37:39.291993       1 controller.go:304] controller/kubeadmconfig "msg"="Reconciler error" "error"="failed to refresh bootstrap token: secrets \"bootstrap-token-zuzhz4\" not found" "name"="cluster-ibs-ng-control-plane-vctx7" "namespace"="default" "reconciler group"="bootstrap.cluster.x-k8s.io" "reconciler kind"="KubeadmConfig"

Environment:

  • Cluster-api version: v1.0.2
  • Minikube/KIND version: kind v0.11.1
  • Kubernetes version: (use kubectl version): 1.22.6

/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]
@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Feb 1, 2022
@fabriziopandini
Copy link
Member

For my better understanding, where the capbk provider is being deployed, on the local machine or on a remote infrastructure?

@joejulian
Copy link
Contributor Author

The capbk provider controller is deployed into the bootstrap cluster ("bootstrap"). The cluster that's being created ("mycluster") is not on the same machine, but is on machines that have the correct time. What matters is that the capbk controller (on "bootstrap") that's creating the join token be behind the cluster that kubeadm join is targeting ("mycluster"), so that the token is already expired when it's installed onto that cluster.

@fabriziopandini
Copy link
Member

/milestone v1.2
wondering how we can get cluster "local" time via the API server...

@k8s-ci-robot k8s-ci-robot added this to the v1.2 milestone Feb 1, 2022
@joejulian
Copy link
Contributor Author

Not sure, but we can at least check the creation time when we get the created resource back and error if it's within TTL/<some number (4?)>

@fabriziopandini fabriziopandini added the area/bootstrap Issues or PRs related to bootstrap providers label Feb 3, 2022
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 4, 2022
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jun 3, 2022
@joejulian
Copy link
Contributor Author

/remove-lifecycle rotten

@k8s-ci-robot k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Jun 9, 2022
@fabriziopandini fabriziopandini added the triage/accepted Indicates an issue or PR is ready to be actively worked on. label Jul 29, 2022
@fabriziopandini fabriziopandini removed this from the v1.2 milestone Jul 29, 2022
@fabriziopandini fabriziopandini removed the triage/accepted Indicates an issue or PR is ready to be actively worked on. label Jul 29, 2022
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 27, 2022
@karansinghneu karansinghneu mentioned this issue Nov 10, 2022
Closed
@batistein batistein mentioned this issue Nov 22, 2022
Closed
@bdehri bdehri mentioned this issue Dec 9, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/bootstrap Issues or PRs related to bootstrap providers kind/bug Categorizes issue or PR as related to a bug. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

🐛 Fix to enable bootstrap secret rotation if the secret itself missing bdehri/cluster-api
4 participants
@joejulian @fabriziopandini @k8s-ci-robot @k8s-triage-robot