Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1926484: UPSTREAM: <carry>: kube-apiserver: ignore SIGTERM/INT after the first one #558

Merged

Conversation

sttts
Copy link

@sttts sttts commented Feb 8, 2021

Normally kubelet sends SIGTERM to root process of container on graceful shutdown. But during node shutdown, systemd is the actor sending the signal instead. And systemd sends SIGTERM to every process. As we have the watch-termination wrapper, both systemd sends SIGTERM and watch-termination forwards => 2 SIGTERMs arrive at kube-apiserver, which makes it shutdown hard (os.Exit(1)).

This PR make kube-apiserver ignore a second SIGTERM.

@openshift-ci-robot openshift-ci-robot added backports/unvalidated-commits Indicates that not all commits come to merged upstream PRs. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Feb 8, 2021
@openshift-ci-robot
Copy link

@sttts: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

@openshift-ci-robot openshift-ci-robot added the vendor-update Touching vendor dir or related files label Feb 8, 2021
@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 8, 2021
@rphillips
Copy link

+++ Running case: verify.typecheck-dockerless 
+++ working dir: /go/src/k8s.io/kubernetes
+++ command: bash "hack/make-rules/../../hack/verify-typecheck-dockerless.sh"
type-checking windows/amd64
type-checking linux/arm
type-checking windows/386
ERROR(linux/arm): /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:129:40: too few arguments in call to server.SetupSignalHandler
ERROR(linux/arm): /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:127:39: too few arguments in call to server.SetupSignalHandler
type-checking darwin/amd64
ERROR(windows/amd64): /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:129:40: too few arguments in call to server.SetupSignalHandler
type-checking linux/386
ERROR(windows/amd64): /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:127:39: too few arguments in call to server.SetupSignalHandler
ERROR(windows/386): /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:129:40: too few arguments in call to server.SetupSignalHandler
type-checking linux/arm64

@sttts sttts force-pushed the sttts-ignore-second-signal branch from c86e461 to 2003be5 Compare February 8, 2021 18:21
@openshift-ci-robot
Copy link

@sttts: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

@sttts
Copy link
Author

sttts commented Feb 8, 2021

Fixed two compilation errors.

@mrunalp
Copy link
Member

mrunalp commented Feb 8, 2021

Opened https://bugzilla.redhat.com/show_bug.cgi?id=1926484 to use as a bugzilla for this.

@smarterclayton
Copy link

@@ -47,8 +49,15 @@ func SetupSignalContext() context.Context {
go func() {
<-shutdownHandler
cancel()
<-shutdownHandler
os.Exit(1) // second signal. Exit directly.
if exitOnSecondSignal {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this a temporary fix?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Temporary in the sense of a downstream carry.

See description for background.

@sttts sttts changed the title WIP: UPSTREAM: <carry>: kube-apiserver: ignore SIGTERM/INT after the first one Bug 1926484: UPSTREAM: <carry>: kube-apiserver: ignore SIGTERM/INT after the first one Feb 9, 2021
@openshift-ci-robot openshift-ci-robot added bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Feb 9, 2021
@openshift-ci-robot
Copy link

@sttts: This pull request references Bugzilla bug 1926484, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.8.0) matches configured target release for branch (4.8.0)
  • bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1926484: UPSTREAM: : kube-apiserver: ignore SIGTERM/INT after the first one

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot
Copy link

@sttts: This pull request references Bugzilla bug 1926484, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.8.0) matches configured target release for branch (4.8.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1926484: UPSTREAM: : kube-apiserver: ignore SIGTERM/INT after the first one

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@sttts
Copy link
Author

sttts commented Feb 9, 2021

/tetest

@@ -38,7 +38,7 @@ func main() {
logs.InitLogs()
defer logs.FlushLogs()

stopCh := genericapiserver.SetupSignalHandler()
stopCh := genericapiserver.SetupSignalHandler(false)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this be set to true? I think this is a standalone binary.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

true. Nobody uses it, but yes. Fixing.

@sttts sttts force-pushed the sttts-ignore-second-signal branch from 2003be5 to c4adb51 Compare February 9, 2021 11:17
@openshift-ci-robot
Copy link

@sttts: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

@rsevilla87
Copy link
Member

/test configmap-scale

@openshift-ci-robot
Copy link

@rsevilla87: The specified target(s) for /test were not found.
The following commands are available to trigger jobs:

  • /test artifacts
  • /test e2e-agnostic-cmd
  • /test e2e-aws
  • /test e2e-aws-csi
  • /test e2e-aws-disruptive
  • /test e2e-aws-fips
  • /test e2e-aws-jenkins
  • /test e2e-aws-multitenant
  • /test e2e-aws-ovn
  • /test e2e-aws-selfupgrade
  • /test e2e-aws-serial
  • /test e2e-aws-upgrade
  • /test e2e-azure
  • /test e2e-azure-upgrade
  • /test e2e-gcp
  • /test e2e-gcp-upgrade
  • /test e2e-metal-ipi
  • /test e2e-metal-ipi-ovn-dualstack
  • /test e2e-metal-ipi-ovn-ipv6
  • /test e2e-openstack
  • /test e2e-vsphere
  • /test images
  • /test integration
  • /test k8s-e2e-conformance-aws
  • /test k8s-e2e-gcp
  • /test k8s-e2e-gcp-serial
  • /test unit
  • /test verify
  • /test verify-commits

Use /test all to run the following jobs:

  • pull-ci-openshift-kubernetes-master-e2e-agnostic-cmd
  • pull-ci-openshift-kubernetes-master-e2e-aws-csi
  • pull-ci-openshift-kubernetes-master-e2e-aws-fips
  • pull-ci-openshift-kubernetes-master-e2e-aws-serial
  • pull-ci-openshift-kubernetes-master-e2e-gcp
  • pull-ci-openshift-kubernetes-master-e2e-gcp-upgrade
  • pull-ci-openshift-kubernetes-master-images
  • pull-ci-openshift-kubernetes-master-integration
  • pull-ci-openshift-kubernetes-master-k8s-e2e-gcp
  • pull-ci-openshift-kubernetes-master-k8s-e2e-gcp-serial
  • pull-ci-openshift-kubernetes-master-unit
  • pull-ci-openshift-kubernetes-master-verify
  • pull-ci-openshift-kubernetes-master-verify-commits

In response to this:

/test configmap-scale

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@sttts
Copy link
Author

sttts commented Feb 9, 2021

/hold

until we have proof it helps cc @mrunalp @smarterclayton

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 9, 2021
@droslean
Copy link
Member

droslean commented Feb 9, 2021

/test configmap-scale

atiratree pushed a commit to atiratree/kubernetes that referenced this pull request Sep 13, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
atiratree pushed a commit to atiratree/kubernetes that referenced this pull request Sep 20, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
atiratree pushed a commit to atiratree/kubernetes that referenced this pull request Sep 20, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
atiratree pushed a commit to atiratree/kubernetes that referenced this pull request Sep 25, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
atiratree pushed a commit to atiratree/kubernetes that referenced this pull request Oct 1, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Nov 26, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Nov 26, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Nov 27, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Nov 27, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Nov 28, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Nov 28, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Nov 29, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Nov 29, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Dec 2, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Dec 4, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Dec 4, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Dec 4, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Dec 5, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Dec 6, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Dec 7, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Dec 9, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Dec 10, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Dec 11, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Dec 11, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Dec 11, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Dec 12, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Dec 16, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Dec 17, 2024
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Jan 16, 2025
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
bertinatto pushed a commit to bertinatto/kubernetes that referenced this pull request Jan 16, 2025
… one

UPSTREAM: <carry>: kube-apiserver: set up separate signal handler functions to ignore further signals

This patches the changes from openshift#558 to provide
these new functions without changing the behavior for other repos that depend on them, such
as library-go.

OpenShift-Rebase-Source: 63ed200
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged. vendor-update Touching vendor dir or related files
Projects
None yet
Development

Successfully merging this pull request may close these issues.