BackendConfig support for balancingMode

[wminshew]

Would be great if BackendConfig had support for balancing mode per https://cloud.google.com/load-balancing/docs/backend-service#traffic_distribution

issue I'm running into now is all the created backend-services automatically balance by rate and are added to the same instance group. If I try to update-backend to change balancingMode UTILIZATION, gcloud throws a validation error (all backends in the same instance group must have the same balancing_mode)

[in the meantime, if there's a good work around please let me know.. have spent a few hours trying to figure it out and am just about ready to give up, even though its quite suboptimal for one of my services]

[rln-storytel]

We're being bitten by a similar bug. Our BackendConfigs are Utilization in zone c & d and have been for a long time. We recently added nodes in zone b, and the BackendConfigs there used Rate.

Now that we try to add an ingress, it tries to use the Rate balancer_mode in all zones, leading to this behaviour:

Name:             reverse-proxy
Namespace:        default
Address:          RE.DAC.TE.D
Default backend:  default-http-backend:80 (10.0.10.14:8080)
TLS:
  redacted-net-tls-20180702 terminates
Rules:
  Host                        Path  Backends
  ----                        ----  --------
  api.redacted.net
                                 api-service:80 (<none>)
  proxy.redacted.net
                                 legacy-proxy-service:80 (<none>)
  something-admin.redacted.net
                                 something-admin:80 (<none>)
  tools.redacted.net
                                 apex:80 (<none>)
Annotations:
  kubernetes.io/ingress.global-static-ip-name:       redacted-net-reverse-proxy
  ingress.kubernetes.io/backends:                    {"k8s-be-30601--redacted":"HEALTHY","k8s-be-31238--redacted":"HEALTHY","k8s-be-32285--redacted":"HEALTHY","k8s-be-32561--redacted":"HEALTHY"}
  ingress.kubernetes.io/https-forwarding-rule:       k8s-fws-default-reverse-proxy--redacted
  ingress.kubernetes.io/https-target-proxy:          k8s-tps-default-reverse-proxy--redacted
  ingress.kubernetes.io/ssl-cert:                    k8s-ssl-f681a1eb7d70c684-aa3ea05972381fac--redacted
  ingress.kubernetes.io/url-map:                     k8s-um-default-reverse-proxy--redacted
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.allow-http":"false","kubernetes.io/ingress.global-static-ip-name":"redacted-net-reverse-proxy"},"name":"reverse-proxy","namespace":"default"},"spec":{"rules":[{"host":"api.redacted.net","http":{"paths":[{"backend":{"serviceName":"api-service","servicePort":80}}]}},{"host":"proxy.redacted.net","http":{"paths":[{"backend":{"serviceName":"legacy-proxy-service","servicePort":80}}]}},{"host":"something-admin.redacted.net","http":{"paths":[{"backend":{"serviceName":"something-admin","servicePort":80}}]}},{"host":"tools.redacted.net","http":{"paths":[{"backend":{"serviceName":"apex","servicePort":80}}]}}],"tls":[{"secretName":"redacted-net-tls-20180702"}]}}

  kubernetes.io/ingress.allow-http:  false
Events:
  Type     Reason  Age                 From                     Message
  ----     ------  ----                ----                     -------
  Warning  Sync    25m (x44 over 18h)  loadbalancer-controller  Error during sync: Error running backend syncing routine: received errors when updating backend service: googleapi: Error 400: INCOMPATIBLE_BACKEND_SERVICES - Validation failed for instance group 'projects/redacted-prod/zones/europe-west1-c/instanceGroups/k8s-ig--redacted': backend services 'projects/redacted-prod/global/backendServices/k8s-be-32285--redacted' and 'projects/redacted-prod/global/backendServices/k8s-be-31976--redacted' point to the same instance group but the backends have incompatible balancing_mode. Values should be the same.
googleapi: Error 400: INCOMPATIBLE_BACKEND_SERVICES - Validation failed for instance group 'projects/redacted-prod/zones/europe-west1-b/instanceGroups/k8s-ig--redacted': backend services 'projects/redacted-prod/global/backendServices/k8s-be-32561--redacted' and 'projects/redacted-prod/global/backendServices/k8s-be-31976--redacted' point to the same instance group but the backends have incompatible balancing_mode. Values should be the same.
  Warning  Sync  15m (x52 over 17h)  loadbalancer-controller  Error during sync: Error running backend syncing routine: received errors when updating backend service: googleapi: Error 400: INCOMPATIBLE_BACKEND_SERVICES - Validation failed for instance group 'projects/redacted-prod/zones/europe-west1-c/instanceGroups/k8s-ig--redacted': backend services 'projects/redacted-prod/global/backendServices/k8s-be-32561--redacted' and 'projects/redacted-prod/global/backendServices/k8s-be-31976--redacted' point to the same instance group but the backends have incompatible balancing_mode. Values should be the same.
googleapi: Error 400: INCOMPATIBLE_BACKEND_SERVICES - Validation failed for instance group 'projects/redacted-prod/zones/europe-west1-b/instanceGroups/k8s-ig--redacted': backend services 'projects/redacted-prod/global/backendServices/k8s-be-32561--redacted' and 'projects/redacted-prod/global/backendServices/k8s-be-31976--redacted' point to the same instance group but the backends have incompatible balancing_mode. Values should be the same.
  Warning  Sync  13m (x50 over 18h)  loadbalancer-controller  Error during sync: Error running backend syncing routine: received errors when updating backend service: googleapi: Error 400: INCOMPATIBLE_BACKEND_SERVICES - Validation failed for instance group 'projects/redacted-prod/zones/europe-west1-c/instanceGroups/k8s-ig--redacted': backend services 'projects/redacted-prod/global/backendServices/k8s-be-32285--redacted' and 'projects/redacted-prod/global/backendServices/k8s-be-31976--redacted' point to the same instance group but the backends have incompatible balancing_mode. Values should be the same.
googleapi: Error 400: INCOMPATIBLE_BACKEND_SERVICES - Validation failed for instance group 'projects/redacted-prod/zones/europe-west1-b/instanceGroups/k8s-ig--redacted': backend services 'projects/redacted-prod/global/backendServices/k8s-be-32285--redacted' and 'projects/redacted-prod/global/backendServices/k8s-be-31976--redacted' point to the same instance group but the backends have incompatible balancing_mode. Values should be the same.
  Warning  Sync  5m41s (x47 over 18h)  loadbalancer-controller  Error during sync: Error running backend syncing routine: received errors when updating backend service: googleapi: Error 400: INCOMPATIBLE_BACKEND_SERVICES - Validation failed for instance group 'projects/redacted-prod/zones/europe-west1-c/instanceGroups/k8s-ig--redacted': backend services 'projects/redacted-prod/global/backendServices/k8s-be-32561--redacted' and 'projects/redacted-prod/global/backendServices/k8s-be-31976--redacted' point to the same instance group but the backends have incompatible balancing_mode. Values should be the same.
googleapi: Error 400: INCOMPATIBLE_BACKEND_SERVICES - Validation failed for instance group 'projects/redacted-prod/zones/europe-west1-b/instanceGroups/k8s-ig--redacted': backend services 'projects/redacted-prod/global/backendServices/k8s-be-32285--redacted' and 'projects/redacted-prod/global/backendServices/k8s-be-31976--redacted' point to the same instance group but the backends have incompatible balancing_mode. Values should be the same.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[rln-storytel]

/remove-lifecycle stale

After discussing this with Google Support, the issue is that our cluster was created prior to Kubernetes 1.7.2. The issue is even described in the documentation.

The only current workaround is to not add new zones. The only current fix is to reroll the cluster from scratch.

Is it really impossible to trigger an update from 'UTILIZATION' to 'RATE' in all zones?

[bowei]

/lifecycle frozen

[bowei]

Unfortunately the validation restrictions today do not allow for an easy way to switch modes for instance groups that are actively attached to an LB. However, it should be adding additional zones with UTILIZATION mode if it detects problems with creation on RATE, and that should be a bug. Can you file a separate issue for that specifically?

[bowei]

Note that utilization does not work unless you are using NEGs as the load on the VM initially receiving traffic is not correlated at all with the backend pod that may be running on a different VM.

[swetharepakula]

Closing as there has been no activity since 2019.