OCPBUGS-46010: wait for at least 3 kube-apiserver instances #9302

tkashem · 2024-12-10T19:21:26Z

No description provided.

openshift-ci-robot · 2024-12-10T19:21:32Z

@tkashem: This pull request references Jira Issue OCPBUGS-45924, which is invalid:

expected the bug to target the "4.19.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

tkashem · 2024-12-10T19:50:57Z

/cc @benluddy

benluddy · 2024-12-10T20:19:50Z

This wouldn't resolve whatever is allowing concurrent etcd installers in 45924, so I opened a separate bug to track.

/retitle [WIP] OCPBUGS-46010: wait for at least 3 kube-apiserver instances

openshift-ci-robot · 2024-12-10T20:20:25Z

@tkashem: This pull request references Jira Issue OCPBUGS-46010, which is invalid:

expected the bug to target the "4.19.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

benluddy · 2024-12-10T21:23:31Z

data/data/bootstrap/files/usr/local/bin/wait-for-ha-api.sh

@@ -57,7 +57,7 @@ is_topology_ha() {

 ##
 ## for HA cluster, we mark the bootstrap process as complete when there
-## are at least two IP addresses available to the endpoints


It's worth updating to mention here that the bootstrap instance is itself one of the endpoints. We don't want to tear down the bootstrap instance until there are enough permanent instances to tolerate 1 unavailable instance.

benluddy · 2024-12-12T14:46:16Z

/lgtm

tkashem · 2024-12-13T14:26:12Z

/cc @patrickdillon

patrickdillon · 2024-12-13T16:55:18Z

/approve

patrickdillon · 2024-12-13T18:31:05Z

/test ?

openshift-ci · 2024-12-13T18:31:09Z

@patrickdillon: The following commands are available to trigger required jobs:

/test altinfra-images

/test aro-unit

/test artifacts-images

/test e2e-agent-compact-ipv4

/test e2e-aws-ovn

/test e2e-aws-ovn-edge-zones-manifest-validation

/test e2e-aws-ovn-upi

/test e2e-azure-ovn

/test e2e-azure-ovn-upi

/test e2e-gcp-ovn

/test e2e-gcp-ovn-upi

/test e2e-metal-ipi-ovn-ipv6

/test e2e-openstack-ovn

/test e2e-vsphere-ovn

/test e2e-vsphere-ovn-upi

/test gofmt

/test golint

/test govet

/test images

/test integration-tests

/test integration-tests-nodejoiner

/test openstack-manifests

/test shellcheck

/test terraform-images

/test terraform-verify-vendor

/test tf-lint

/test unit

/test verify-codegen

/test verify-vendor

/test yaml-lint

The following commands are available to trigger optional jobs:

/test altinfra-e2e-aws-custom-security-groups

/test altinfra-e2e-aws-ovn

/test altinfra-e2e-aws-ovn-fips

/test altinfra-e2e-aws-ovn-imdsv2

/test altinfra-e2e-aws-ovn-localzones

/test altinfra-e2e-aws-ovn-proxy

/test altinfra-e2e-aws-ovn-shared-vpc

/test altinfra-e2e-aws-ovn-shared-vpc-local-zones

/test altinfra-e2e-aws-ovn-shared-vpc-wavelength-zones

/test altinfra-e2e-aws-ovn-single-node

/test altinfra-e2e-aws-ovn-wavelengthzones

/test altinfra-e2e-azure-capi-ovn

/test altinfra-e2e-azure-ovn-shared-vpc

/test altinfra-e2e-gcp-capi-ovn

/test altinfra-e2e-gcp-ovn-byo-network-capi

/test altinfra-e2e-gcp-ovn-secureboot-capi

/test altinfra-e2e-gcp-ovn-xpn-capi

/test altinfra-e2e-ibmcloud-capi-ovn

/test altinfra-e2e-nutanix-capi-ovn

/test altinfra-e2e-openstack-capi-ccpmso

/test altinfra-e2e-openstack-capi-ccpmso-zone

/test altinfra-e2e-openstack-capi-dualstack

/test altinfra-e2e-openstack-capi-dualstack-upi

/test altinfra-e2e-openstack-capi-dualstack-v6primary

/test altinfra-e2e-openstack-capi-externallb

/test altinfra-e2e-openstack-capi-nfv-intel

/test altinfra-e2e-openstack-capi-ovn

/test altinfra-e2e-openstack-capi-proxy

/test altinfra-e2e-vsphere-capi-multi-vcenter-ovn

/test altinfra-e2e-vsphere-capi-ovn

/test altinfra-e2e-vsphere-capi-static-ovn

/test altinfra-e2e-vsphere-capi-zones

/test azure-ovn-marketplace-images

/test e2e-agent-4control-ipv4

/test e2e-agent-5control-ipv4

/test e2e-agent-compact-ipv4-appliance-diskimage

/test e2e-agent-compact-ipv4-none-platform

/test e2e-agent-compact-ipv6-minimaliso

/test e2e-agent-ha-dualstack

/test e2e-agent-sno-ipv4-pxe

/test e2e-agent-sno-ipv6

/test e2e-aws-default-config

/test e2e-aws-overlay-mtu-ovn-1200

/test e2e-aws-ovn-custom-iam-profile

/test e2e-aws-ovn-edge-zones

/test e2e-aws-ovn-fips

/test e2e-aws-ovn-heterogeneous

/test e2e-aws-ovn-imdsv2

/test e2e-aws-ovn-proxy

/test e2e-aws-ovn-public-ipv4-pool

/test e2e-aws-ovn-public-ipv4-pool-disabled

/test e2e-aws-ovn-public-subnets

/test e2e-aws-ovn-shared-vpc-custom-security-groups

/test e2e-aws-ovn-shared-vpc-edge-zones

/test e2e-aws-ovn-single-node

/test e2e-aws-ovn-techpreview

/test e2e-aws-ovn-upgrade

/test e2e-aws-ovn-workers-rhel8

/test e2e-aws-upi-proxy

/test e2e-azure-default-config

/test e2e-azure-ovn-resourcegroup

/test e2e-azure-ovn-shared-vpc

/test e2e-azure-ovn-techpreview

/test e2e-azurestack

/test e2e-azurestack-upi

/test e2e-crc

/test e2e-external-aws

/test e2e-external-aws-ccm

/test e2e-gcp-default-config

/test e2e-gcp-ovn-byo-vpc

/test e2e-gcp-ovn-heterogeneous

/test e2e-gcp-ovn-techpreview

/test e2e-gcp-ovn-xpn

/test e2e-gcp-secureboot

/test e2e-gcp-upgrade

/test e2e-gcp-upi-xpn

/test e2e-gcp-user-provisioned-dns

/test e2e-ibmcloud-ovn

/test e2e-metal-assisted

/test e2e-metal-ipi-ovn

/test e2e-metal-ipi-ovn-dualstack

/test e2e-metal-ipi-ovn-swapped-hosts

/test e2e-metal-ipi-ovn-virtualmedia

/test e2e-metal-single-node-live-iso

/test e2e-nutanix-ovn

/test e2e-openstack-ccpmso

/test e2e-openstack-ccpmso-zone

/test e2e-openstack-dualstack

/test e2e-openstack-dualstack-upi

/test e2e-openstack-externallb

/test e2e-openstack-nfv-intel

/test e2e-openstack-proxy

/test e2e-openstack-singlestackv6

/test e2e-powervs-capi-ovn

/test e2e-vsphere-multi-vcenter-ovn

/test e2e-vsphere-ovn-multi-network

/test e2e-vsphere-ovn-techpreview

/test e2e-vsphere-ovn-upi-zones

/test e2e-vsphere-ovn-zones

/test e2e-vsphere-ovn-zones-techpreview

/test e2e-vsphere-static-ovn

/test okd-scos-e2e-aws-ovn

/test okd-scos-images

/test tf-fmt

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-installer-master-altinfra-images

pull-ci-openshift-installer-master-aro-unit

pull-ci-openshift-installer-master-artifacts-images

pull-ci-openshift-installer-master-e2e-aws-ovn

pull-ci-openshift-installer-master-gofmt

pull-ci-openshift-installer-master-golint

pull-ci-openshift-installer-master-govet

pull-ci-openshift-installer-master-images

pull-ci-openshift-installer-master-okd-scos-e2e-aws-ovn

pull-ci-openshift-installer-master-shellcheck

pull-ci-openshift-installer-master-tf-fmt

pull-ci-openshift-installer-master-tf-lint

pull-ci-openshift-installer-master-unit

pull-ci-openshift-installer-master-verify-codegen

pull-ci-openshift-installer-master-verify-vendor

pull-ci-openshift-installer-master-yaml-lint

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

benluddy · 2024-12-13T18:36:51Z

/jira refresh

openshift-ci-robot · 2024-12-13T18:36:58Z

@benluddy: This pull request references Jira Issue OCPBUGS-46010, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.19.0) matches configured target version for branch (4.19.0)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @wangke19

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

patrickdillon · 2024-12-13T19:20:43Z

/test e2e-agent-compact-ipv4

tkashem · 2024-12-13T19:21:23Z

/hold

(make sure agent based install works)

tkashem · 2025-01-13T20:16:04Z

tested the new command against a cluster

$ oc --kubeconfig="$KUBECONFIG" get kubeapiservers cluster -o jsonpath='{range @.status.nodeStatuses[?(@.currentRevision>0)]}{.nodeName}{" "}{end}'

ip-10-0-121-118.us-east-2.compute.internal ip-10-0-16-56.us-east-2.compute.internal ip-10-0-58-223.us-east-2.compute.internal

tkashem · 2025-01-13T20:19:33Z

$ oc --kubeconfig="$KUBECONFIG" get kubeapiservers cluster -o yaml

apiVersion: operator.openshift.io/v1
kind: KubeAPIServer
metadata:
  annotations:
    include.release.openshift.io/ibm-cloud-managed: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
    release.openshift.io/create-only: "true"
  creationTimestamp: "2025-01-13T19:31:19Z"
  generation: 8
  name: cluster
  ownerReferences:
  - apiVersion: config.openshift.io/v1
    kind: ClusterVersion
    name: version
    uid: 3ac2fb0f-db7b-48e5-8ef8-e32e8ebfb69a
  resourceVersion: "32678"
  uid: f651e87a-6378-4d76-9e78-16d8db8a79fc
spec:
  logLevel: Normal
  managementState: Managed
  observedConfig:
    admission:
      pluginConfig:
        PodSecurity:
          configuration:
            defaults:
              audit: restricted
              audit-version: latest
              enforce: restricted
              enforce-version: latest
              warn: restricted
              warn-version: latest
        network.openshift.io/ExternalIPRanger:
          configuration:
            allowIngressIP: false
            apiVersion: network.openshift.io/v1
            kind: ExternalIPRangerAdmissionConfig
        network.openshift.io/RestrictedEndpointsAdmission:
          configuration:
            apiVersion: network.openshift.io/v1
            kind: RestrictedEndpointsAdmissionConfig
            restrictedCIDRs:
            - 10.128.0.0/14
            - 172.30.0.0/16
    apiServerArguments:
      api-audiences:
      - https://kubernetes.default.svc
      authentication-token-webhook-config-file:
      - /etc/kubernetes/static-pod-resources/secrets/webhook-authenticator/kubeConfig
      authentication-token-webhook-version:
      - v1
      etcd-servers:
      - https://10.0.121.118:2379
      - https://10.0.16.56:2379
      - https://10.0.58.223:2379
      - https://localhost:2379
      feature-gates:
      - AWSEFSDriverVolumeMetrics=true
      - AdditionalRoutingCapabilities=true
      - AdminNetworkPolicy=true
      - AlibabaPlatform=true
      - AzureWorkloadIdentity=true
      - BareMetalLoadBalancer=true
      - BuildCSIVolumes=true
      - ChunkSizeMiB=true
      - CloudDualStackNodeIPs=true
      - DisableKubeletCloudCredentialProviders=true
      - GCPLabelsTags=true
      - HardwareSpeed=true
      - IngressControllerLBSubnetsAWS=true
      - KMSv1=true
      - ManagedBootImages=true
      - ManagedBootImagesAWS=true
      - MultiArchInstallAWS=true
      - MultiArchInstallGCP=true
      - NetworkDiagnosticsConfig=true
      - NetworkLiveMigration=true
      - NewOLM=true
      - NodeDisruptionPolicy=true
      - OpenShiftPodSecurityAdmission=true
      - PrivateHostedZoneAWS=true
      - SetEIPForNLBIngressController=true
      - VSphereControlPlaneMachineSet=true
      - VSphereDriverConfiguration=true
      - VSphereMultiVCenters=true
      - VSphereStaticIPs=true
      - ValidatingAdmissionPolicy=true
      - AWSClusterHostedDNS=false
      - AutomatedEtcdBackup=false
      - BootcNodeManagement=false
      - CPMSMachineNamePrefix=false
      - ClusterAPIInstall=false
      - ClusterAPIInstallIBMCloud=false
      - ClusterMonitoringConfig=false
      - ClusterVersionOperatorConfiguration=false
      - ConsolePluginContentSecurityPolicy=false
      - DNSNameResolver=false
      - DynamicResourceAllocation=false
      - EtcdBackendQuota=false
      - EventedPLEG=false
      - Example=false
      - ExternalOIDC=false
      - GCPClusterHostedDNS=false
      - GatewayAPI=false
      - HighlyAvailableArbiter=false
      - ImageStreamImportMode=false
      - IngressControllerDynamicConfigurationManager=false
      - InsightsConfig=false
      - InsightsConfigAPI=false
      - InsightsOnDemandDataGather=false
      - InsightsRuntimeExtractor=false
      - KMSEncryptionProvider=false
      - MachineAPIMigration=false
      - MachineAPIOperatorDisableMachineHealthCheckController=false
      - MachineAPIProviderOpenStack=false
      - MachineConfigNodes=false
      - MaxUnavailableStatefulSet=false
      - MetricsCollectionProfiles=false
      - MinimumKubeletVersion=false
      - MixedCPUsAllocation=false
      - MultiArchInstallAzure=false
      - NetworkSegmentation=false
      - NodeSwap=false
      - NutanixMultiSubnets=false
      - OVNObservability=false
      - OnClusterBuild=false
      - PersistentIPsForVirtualization=false
      - PinnedImages=false
      - PlatformOperators=false
      - ProcMountType=false
      - RouteAdvertisements=false
      - RouteExternalCertificate=false
      - ServiceAccountTokenNodeBinding=false
      - SignatureStores=false
      - SigstoreImageVerification=false
      - TranslateStreamCloseWebsocketRequests=false
      - UpgradeStatus=false
      - UserNamespacesPodSecurityStandards=false
      - UserNamespacesSupport=false
      - VSphereHostVMGroupZonal=false
      - VSphereMultiDisk=false
      - VSphereMultiNetworks=false
      - VolumeAttributesClass=false
      - VolumeGroupSnapshot=false
      runtime-config:
      - admissionregistration.k8s.io/v1beta1=true
      send-retry-after-while-not-ready-once:
      - "false"
      service-account-issuer:
      - https://kubernetes.default.svc
      service-account-jwks-uri:
      - https://api.ci-ln-wni5ym2-76ef8.aws-2.ci.openshift.org:6443/openid/v1/jwks
      shutdown-delay-duration:
      - 129s
    authConfig:
      oauthMetadataFile: /etc/kubernetes/static-pod-resources/configmaps/oauth-metadata/oauthMetadata
    corsAllowedOrigins:
    - //127\.0\.0\.1(:|$)
    - //localhost(:|$)
    gracefulTerminationDuration: "194"
    imagePolicyConfig:
      internalRegistryHostname: image-registry.openshift-image-registry.svc:5000
    servicesSubnet: 172.30.0.0/16
    servingInfo:
      bindAddress: 0.0.0.0:6443
      bindNetwork: tcp4
      cipherSuites:
      - TLS_AES_128_GCM_SHA256
      - TLS_AES_256_GCM_SHA384
      - TLS_CHACHA20_POLY1305_SHA256
      - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
      - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
      - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
      - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
      - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
      minTLSVersion: VersionTLS12
      namedCertificates:
      - certFile: /etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.crt
        keyFile: /etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.key
      - certFile: /etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt
        keyFile: /etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key
      - certFile: /etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.crt
        keyFile: /etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.key
      - certFile: /etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt
        keyFile: /etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key
      - certFile: /etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.crt
        keyFile: /etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.key
  operatorLogLevel: Normal
  unsupportedConfigOverrides: null
status:
  conditions:
  - lastTransitionTime: "2025-01-13T19:33:46Z"
    reason: NoUnsupportedConfigOverrides
    status: "True"
    type: UnsupportedConfigOverridesUpgradeable
  - lastTransitionTime: "2025-01-13T19:33:46Z"
    message: Kubelet and API server minor versions are synced.
    reason: KubeletMinorVersionsSynced
    status: "True"
    type: KubeletMinorVersionUpgradeable
  - lastTransitionTime: "2025-01-13T19:33:46Z"
    message: All master nodes are ready
    reason: MasterNodesReady
    status: "False"
    type: NodeControllerDegraded
  - lastTransitionTime: "2025-01-13T19:33:46Z"
    reason: ExpectedReason
    status: "False"
    type: PodSecurityCustomerEvaluationConditionsDetected
  - lastTransitionTime: "2025-01-13T19:33:46Z"
    reason: ExpectedReason
    status: "False"
    type: PodSecurityOpenshiftEvaluationConditionsDetected
  - lastTransitionTime: "2025-01-13T19:33:46Z"
    reason: ExpectedReason
    status: "False"
    type: PodSecurityRunLevelZeroEvaluationConditionsDetected
  - lastTransitionTime: "2025-01-13T19:33:46Z"
    reason: ExpectedReason
    status: "False"
    type: PodSecurityDisabledSyncerEvaluationConditionsDetected
  - lastTransitionTime: "2025-01-13T19:33:46Z"
    reason: DefaultCertRotationBase
    status: "True"
    type: CertRotationTimeUpgradeable
  - lastTransitionTime: "2025-01-13T19:35:16Z"
    status: "False"
    type: MutatingAdmissionWebhookConfigurationError
  - lastTransitionTime: "2025-01-13T19:36:06Z"
    status: "False"
    type: ValidatingAdmissionWebhookConfigurationError
  - lastTransitionTime: "2025-01-13T19:39:38Z"
    status: "False"
    type: CRDConversionWebhookConfigurationError
  - lastTransitionTime: "2025-01-13T19:33:46Z"
    status: "False"
    type: VirtualResourceAdmissionError
  - lastTransitionTime: "2025-01-13T19:33:47Z"
    status: "False"
    type: StartupMonitorPodContainerExcessiveRestartsDegraded
  - lastTransitionTime: "2025-01-13T19:33:47Z"
    status: "False"
    type: StartupMonitorPodDegraded
  - lastTransitionTime: "2025-01-13T19:33:47Z"
    status: "False"
    type: EncryptionStateControllerDegraded
  - lastTransitionTime: "2025-01-13T19:33:47Z"
    status: "False"
    type: Encrypted
  - lastTransitionTime: "2025-01-13T19:33:47Z"
    status: "False"
    type: EncryptionPruneControllerDegraded
  - lastTransitionTime: "2025-01-13T19:33:47Z"
    status: "False"
    type: EncryptionKeyControllerDegraded
  - lastTransitionTime: "2025-01-13T19:33:47Z"
    status: "False"
    type: EncryptionMigrationControllerDegraded
  - lastTransitionTime: "2025-01-13T19:33:47Z"
    status: "False"
    type: EncryptionMigrationControllerProgressing
  - lastTransitionTime: "2025-01-13T19:52:58Z"
    reason: AsExpected
    status: "False"
    type: GuardControllerDegraded
  - lastTransitionTime: "2025-01-13T19:33:47Z"
    status: "False"
    type: StaticPodFallbackRevisionDegraded
  - lastTransitionTime: "2025-01-13T19:42:39Z"
    status: "False"
    type: RevisionControllerDegraded
  - lastTransitionTime: "2025-01-13T19:33:48Z"
    message: latency profile not set on cluster
    reason: ProfileEmpty
    status: "True"
    type: WorkerLatencyProfileComplete
  - lastTransitionTime: "2025-01-13T19:33:48Z"
    reason: ProfileEmpty
    status: "False"
    type: WorkerLatencyProfileProgressing
  - lastTransitionTime: "2025-01-13T19:35:57Z"
    status: "False"
    type: InstallerControllerDegraded
  - lastTransitionTime: "2025-01-13T19:42:49Z"
    status: "False"
    type: NodeInstallerDegraded
  - lastTransitionTime: "2025-01-13T19:57:29Z"
    message: 3 nodes are at revision 7
    reason: AllNodesAtLatestRevision
    status: "False"
    type: NodeInstallerProgressing
  - lastTransitionTime: "2025-01-13T19:37:32Z"
    message: 3 nodes are active; 3 nodes are at revision 7
    reason: ""
    status: "True"
    type: StaticPodsAvailable
  - lastTransitionTime: "2025-01-13T19:33:48Z"
    reason: AsExpected
    status: "False"
    type: MissingStaticPodControllerDegraded
  - lastTransitionTime: "2025-01-13T19:33:49Z"
    reason: AsExpected
    status: "False"
    type: WorkerLatencyProfileDegraded
  - lastTransitionTime: "2025-01-13T19:35:42Z"
    reason: AsExpected
    status: "False"
    type: NodeKubeconfigControllerDegraded
  - lastTransitionTime: "2025-01-13T19:33:49Z"
    status: "False"
    type: InstallerPodContainerWaitingDegraded
  - lastTransitionTime: "2025-01-13T19:33:49Z"
    status: "False"
    type: InstallerPodNetworkingDegraded
  - lastTransitionTime: "2025-01-13T19:33:49Z"
    status: "False"
    type: InstallerPodPendingDegraded
  - lastTransitionTime: "2025-01-13T19:34:20Z"
    status: "False"
    type: CertRotation_ExternalLoadBalancerServing_Degraded
  - lastTransitionTime: "2025-01-13T19:34:26Z"
    status: "False"
    type: ConfigObservationDegraded
  - lastTransitionTime: "2025-01-13T19:34:04Z"
    status: "False"
    type: CertRotation_CheckEndpointsClient_Degraded
  - lastTransitionTime: "2025-01-13T19:34:24Z"
    status: "False"
    type: CertRotation_KubeSchedulerClient_Degraded
  - lastTransitionTime: "2025-01-13T19:33:53Z"
    status: "False"
    type: AuditPolicyDegraded
  - lastTransitionTime: "2025-01-13T19:33:54Z"
    status: "False"
    type: CertRotation_InternalLoadBalancerServing_Degraded
  - lastTransitionTime: "2025-01-13T19:42:48Z"
    reason: AsExpected
    status: "False"
    type: BackingResourceControllerDegraded
  - lastTransitionTime: "2025-01-13T19:33:56Z"
    status: "False"
    type: StaticPodsDegraded
  - lastTransitionTime: "2025-01-13T19:34:00Z"
    status: "False"
    type: ResourceSyncControllerDegraded
  - lastTransitionTime: "2025-01-13T19:43:12Z"
    reason: AsExpected
    status: "False"
    type: KubeAPIServerStaticResourcesDegraded
  - lastTransitionTime: "2025-01-13T19:34:09Z"
    status: "False"
    type: CertRotation_KubeAPIServerToKubeletClientCert_Degraded
  - lastTransitionTime: "2025-01-13T19:34:12Z"
    status: "False"
    type: CertRotation_AggregatorProxyClientCert_Degraded
  - lastTransitionTime: "2025-01-13T19:34:14Z"
    status: "False"
    type: CertRotation_LocalhostServing_Degraded
  - lastTransitionTime: "2025-01-13T19:34:16Z"
    status: "False"
    type: CertRotation_NodeSystemAdminClient_Degraded
  - lastTransitionTime: "2025-01-13T19:34:18Z"
    status: "False"
    type: CertRotation_LocalhostRecoveryServing_Degraded
  - lastTransitionTime: "2025-01-13T19:34:22Z"
    status: "False"
    type: CertRotation_ServiceNetworkServing_Degraded
  - lastTransitionTime: "2025-01-13T19:34:23Z"
    status: "False"
    type: CertRotation_KubeControllerManagerClient_Degraded
  - lastTransitionTime: "2025-01-13T19:34:26Z"
    status: "False"
    type: CertRotation_ControlPlaneNodeAdminClient_Degraded
  - lastTransitionTime: "2025-01-13T19:43:21Z"
    status: "False"
    type: TargetConfigControllerDegraded
  latestAvailableRevision: 7
  nodeStatuses:
  - currentRevision: 7
    lastFailedCount: 0
    lastFailedReason: ""
    lastFailedRevision: 0
    lastFallbackCount: 0
    nodeName: ip-10-0-121-118.us-east-2.compute.internal
    targetRevision: 0
  - currentRevision: 7
    lastFailedCount: 0
    lastFailedReason: ""
    lastFailedRevision: 0
    lastFallbackCount: 0
    nodeName: ip-10-0-16-56.us-east-2.compute.internal
    targetRevision: 0
  - currentRevision: 7
    lastFailedCount: 0
    lastFailedReason: ""
    lastFailedRevision: 0
    lastFailedRevisionErrors:
    - |
      installer: ing-cert-002",
        (string) (len=21) "user-serving-cert-003",
        (string) (len=21) "user-serving-cert-004",
        (string) (len=21) "user-serving-cert-005",
        (string) (len=21) "user-serving-cert-006",
        (string) (len=21) "user-serving-cert-007",
        (string) (len=21) "user-serving-cert-008",
        (string) (len=21) "user-serving-cert-009"
       },
       CertConfigMapNamePrefixes: ([]string) (len=4 cap=4) {
        (string) (len=20) "aggregator-client-ca",
        (string) (len=9) "client-ca",
        (string) (len=29) "control-plane-node-kubeconfig",
        (string) (len=26) "check-endpoints-kubeconfig"
       },
       OptionalCertConfigMapNamePrefixes: ([]string) (len=1 cap=1) {
        (string) (len=17) "trusted-ca-bundle"
       },
       CertDir: (string) (len=57) "/etc/kubernetes/static-pod-resources/kube-apiserver-certs",
       ResourceDir: (string) (len=36) "/etc/kubernetes/static-pod-resources",
       PodManifestDir: (string) (len=25) "/etc/kubernetes/manifests",
       Timeout: (time.Duration) 2m0s,
       StaticPodManifestsLockFile: (string) "",
       PodMutationFns: ([]installerpod.PodMutationFunc) <nil>,
       KubeletVersion: (string) ""
      })
      I0113 19:40:44.064637       1 cmd.go:413] Getting controller reference for node ip-10-0-58-223.us-east-2.compute.internal
      I0113 19:40:44.073268       1 cmd.go:426] Waiting for installer revisions to settle for node ip-10-0-58-223.us-east-2.compute.internal
      I0113 19:40:44.073308       1 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
      I0113 19:40:44.073321       1 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
      I0113 19:40:44.076651       1 cmd.go:518] Waiting additional period after revisions have settled for node ip-10-0-58-223.us-east-2.compute.internal
      I0113 19:41:14.077032       1 cmd.go:524] Getting installer pods for node ip-10-0-58-223.us-east-2.compute.internal
      F0113 19:41:28.080207       1 cmd.go:109] Get "https://172.30.0.1:443/api/v1/namespaces/openshift-kube-apiserver/pods?labelSelector=app%3Dinstaller": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
    lastFailedTime: "2025-01-13T19:42:44Z"
    lastFallbackCount: 0
    nodeName: ip-10-0-58-223.us-east-2.compute.internal
    targetRevision: 0
  readyReplicas: 0
  serviceAccountIssuers:
  - name: https://kubernetes.default.svc

tkashem · 2025-01-14T10:30:54Z

/retest

tkashem · 2025-01-14T12:04:06Z

/label acknowledge-critical-fixes-only

tkashem · 2025-01-15T03:03:38Z

/retest-required

tkashem · 2025-01-15T12:46:22Z

/test e2e-agent-compact-ipv4

benluddy · 2025-01-15T14:10:54Z

/test e2e-agent-ha-dualstack

benluddy · 2025-01-15T14:11:12Z

/test e2e-agent-sno-ipv6

openshift-ci · 2025-01-15T15:06:43Z

@tkashem: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-azure-ovn	`19e98f9`	link	true	`/test e2e-azure-ovn`
ci/prow/e2e-azure-ovn-upi	`19e98f9`	link	true	`/test e2e-azure-ovn-upi`
ci/prow/e2e-azure-ovn-resourcegroup	`ba82c38`	link	false	`/test e2e-azure-ovn-resourcegroup`
ci/prow/e2e-vsphere-static-ovn	`ba82c38`	link	false	`/test e2e-vsphere-static-ovn`
ci/prow/e2e-vsphere-ovn-multi-network	`ba82c38`	link	false	`/test e2e-vsphere-ovn-multi-network`
ci/prow/okd-scos-e2e-aws-ovn	`ba82c38`	link	false	`/test okd-scos-e2e-aws-ovn`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

benluddy · 2025-01-15T15:37:41Z

/test e2e-agent-sno-ipv6

The latest failure looks like a transient image registry error. The other two agent jobs are green.

Error: copying system image from manifest list: reading blob sha256:0d1baa9a1e65a2a5993ffaad35ff54cfb88cd20846309f34746d67cc421dbfe9: fetching blob: received unexpected HTTP status: 502 Bad Gateway

benluddy · 2025-01-15T15:40:58Z

/lgtm

(assuming the last agent job passes)

tkashem · 2025-01-15T16:31:54Z

/hold cancel

(e2e-agent-compact-ipv4 has passed)

openshift-ci-robot · 2025-01-15T19:32:41Z

@tkashem: Jira Issue OCPBUGS-46010: All pull requests linked via external trackers have merged:

openshift/installer#9302

Jira Issue OCPBUGS-46010 has been moved to the MODIFIED state.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

benluddy · 2025-01-15T19:38:58Z

/cherry-pick release-4.18

openshift-cherrypick-robot · 2025-01-15T19:39:55Z

@benluddy: new pull request created: #9372

In response to this:

/cherry-pick release-4.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-bot · 2025-01-16T01:02:26Z

[ART PR BUILD NOTIFIER]

Distgit: ose-installer-terraform-providers
This PR has been included in build ose-installer-terraform-providers-container-v4.19.0-202501152337.p0.gb2723a8.assembly.stream.el9.
All builds following this will include this PR.

openshift-bot · 2025-01-16T01:32:39Z

[ART PR BUILD NOTIFIER]

Distgit: ose-installer-altinfra
This PR has been included in build ose-installer-altinfra-container-v4.19.0-202501152337.p0.gb2723a8.assembly.stream.el9.
All builds following this will include this PR.

openshift-bot · 2025-01-16T01:38:32Z

[ART PR BUILD NOTIFIER]

Distgit: ose-baremetal-installer
This PR has been included in build ose-baremetal-installer-container-v4.19.0-202501152337.p0.gb2723a8.assembly.stream.el9.
All builds following this will include this PR.

openshift-bot · 2025-01-16T02:23:38Z

[ART PR BUILD NOTIFIER]

Distgit: ose-installer-artifacts
This PR has been included in build ose-installer-artifacts-container-v4.19.0-202501152337.p0.gb2723a8.assembly.stream.el9.
All builds following this will include this PR.

wangke19 · 2025-01-26T11:18:52Z

/jira refresh

openshift-ci-robot · 2025-01-26T11:18:55Z

@wangke19: Jira Issue OCPBUGS-46010 is in an unrecognized state (Verified) and will not be moved to the MODIFIED state.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Dec 10, 2024

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 10, 2024

openshift-ci bot requested review from barbacbd and patrickdillon December 10, 2024 19:22

openshift-ci bot requested a review from benluddy December 10, 2024 19:51

openshift-ci bot changed the title ~~[WIP] OCPBUGS-45924: wait for at least 3 kube-apiserver instances~~ [WIP] OCPBUGS-46010: wait for at least 3 kube-apiserver instances Dec 10, 2024

benluddy reviewed Dec 10, 2024

View reviewed changes

tkashem force-pushed the fix-wait-for-ha-api branch from 62abaf4 to 19e98f9 Compare December 11, 2024 14:19

openshift-ci bot assigned benluddy Dec 12, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 12, 2024

tkashem changed the title ~~[WIP] OCPBUGS-46010: wait for at least 3 kube-apiserver instances~~ OCPBUGS-46010: wait for at least 3 kube-apiserver instances Dec 12, 2024

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 12, 2024

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 13, 2024

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Dec 13, 2024

openshift-ci bot requested a review from wangke19 December 13, 2024 18:37

tkashem changed the title ~~[WIP] OCPBUGS-46010: wait for at least 3 kube-apiserver instances~~ OCPBUGS-46010: wait for at least 3 kube-apiserver instances Jan 13, 2025

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 13, 2025

openshift-ci bot added the acknowledge-critical-fixes-only Indicates if the issuer of the label is OK with the policy. label Jan 14, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jan 15, 2025

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 15, 2025

openshift-merge-bot bot merged commit b2723a8 into openshift:main Jan 15, 2025
17 of 21 checks passed

openshift-cherrypick-robot mentioned this pull request Jan 15, 2025

[release-4.18] OCPBUGS-48451: wait for at least 3 kube-apiserver instances #9372

Merged

openshift-ci-robot mentioned this pull request Jan 16, 2025

OCPBUGS-45924: Prevent the installer controller from stomping statuses it just applied openshift/library-go#1913

Merged

openshift-ci-robot mentioned this pull request Jan 29, 2025

OCPBUGS-45924: Fix "live get after write" in static pod installer controller. openshift/library-go#1929

Merged

OCPBUGS-46010: wait for at least 3 kube-apiserver instances #9302

OCPBUGS-46010: wait for at least 3 kube-apiserver instances #9302

Conversation

tkashem commented Dec 10, 2024

openshift-ci-robot commented Dec 10, 2024

tkashem commented Dec 10, 2024

benluddy commented Dec 10, 2024

openshift-ci-robot commented Dec 10, 2024

benluddy Dec 10, 2024

Choose a reason for hiding this comment

benluddy commented Dec 12, 2024

tkashem commented Dec 13, 2024

patrickdillon commented Dec 13, 2024

patrickdillon commented Dec 13, 2024

openshift-ci bot commented Dec 13, 2024

benluddy commented Dec 13, 2024

openshift-ci-robot commented Dec 13, 2024

patrickdillon commented Dec 13, 2024

tkashem commented Dec 13, 2024

tkashem commented Jan 13, 2025

tkashem commented Jan 13, 2025 • edited Loading

tkashem commented Jan 14, 2025

tkashem commented Jan 14, 2025

tkashem commented Jan 15, 2025

tkashem commented Jan 15, 2025

benluddy commented Jan 15, 2025

benluddy commented Jan 15, 2025

openshift-ci bot commented Jan 15, 2025 • edited Loading

benluddy commented Jan 15, 2025 • edited Loading

benluddy commented Jan 15, 2025

tkashem commented Jan 15, 2025

openshift-ci-robot commented Jan 15, 2025

benluddy commented Jan 15, 2025

openshift-cherrypick-robot commented Jan 15, 2025

openshift-bot commented Jan 16, 2025

openshift-bot commented Jan 16, 2025

openshift-bot commented Jan 16, 2025

openshift-bot commented Jan 16, 2025

wangke19 commented Jan 26, 2025

openshift-ci-robot commented Jan 26, 2025

tkashem commented Jan 13, 2025 •

edited

Loading

openshift-ci bot commented Jan 15, 2025 •

edited

Loading

benluddy commented Jan 15, 2025 •

edited

Loading