google_container_cluster: cannot set logging and monitoring config at the same time

[hazcod]

Terraform Version

Terraform v1.1.2
on darwin_arm64
+ provider registry.terraform.io/hashicorp/google-beta v4.5.0

Affected Resource(s)

• google_container_cluster

Terraform Configuration Files

resource "google_container_cluster" "gke_cluster" {
  ...
  # use stackdriver GKE for system and workload logs
  logging_service = "logging.googleapis.com/kubernetes"
  logging_config {
    enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"]
  }

  # use stackdriver GKE native system monitoring for everything
  monitoring_service = "monitoring.googleapis.com/kubernetes"
  monitoring_config {
    enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"]
  }
}

Panic Output

╷
│ Error: googleapi: Error 400: Cannot specify logging_config or monitoring_config together with logging_service or monitoring_service., badRequest
│ 
│   with module.google.module.dev_cluster.google_container_cluster.gke_cluster,
│   on google/gke/gke.tf line 1, in resource "google_container_cluster" "gke_cluster":
│    1: resource "google_container_cluster" "gke_cluster" {

Expected Behavior

Apply works.

Actual Behavior

Apply fails.

Steps to Reproduce

1. terraform apply

References

• google-beta v3.87 crashes with monitoring_config on google_container_cluster #10280
• b/298512779

[renescheepers]

I'm also running into this bug, makes it currently impossible to use Managed Prometheus with logging & metrics disabled.

[giuliano-sider]

I'm just wondering: why do you need to specify logging_service and monitoring_service in the config above? Can't you just leave it out of the config? I believe the value you specified is the default anyway (in fact, the other ones -"monitoring.googleapis.com" and "logging.googleapis.com" respectively - aren't actually available since GKE 1.14): https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1/projects.locations.clusters.

[giuliano-sider]

It should work once you take out logging_service and monitoring_service (which in practice weren't having any effect anyway).

[giuliano-sider]

Just to clarify (there may be some grammatical parsing ambiguity in the 400 Bad Request error message returned by the GKE API): it's fine to set both logging_config and monitoring_config. However, the GKE API doesn't want you to both logging_service and logging_config, or both monitoring_service and monitoring_config. But since GKE 1.14, you shouldn't have to set {logging,monitoring}_service anyway.

[giuliano-sider]

To disable logging for the cluster, you should be able to set enable_components to an empty list (https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1/projects.locations.clusters#loggingcomponentconfig) instead of setting logging_service.

[giuliano-sider]

Note that the following works (to get workload metrics nowadays, you have to enable GMP):

resource "google_container_cluster" "cluster_with_logging_monitoring_config" {
  provider           = google
  name               = "cluster-with-logging-monitoring-config"
  location           = "us-central1-a"

  initial_node_count = 1

  # "Cannot specify logging_config or monitoring_config together with logging_service or monitoring_service."
  # logging_service = "logging.googleapis.com/kubernetes"

  logging_config {
    enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"]
  }

  # "Cannot specify logging_config or monitoring_config together with logging_service or monitoring_service."
  # monitoring_service = "monitoring.googleapis.com/kubernetes"

  monitoring_config {
    enable_components = ["SYSTEM_COMPONENTS"]
    managed_prometheus {
      enabled = true
    }
  }
}

[giuliano-sider]

The config for disabling logging and system metrics but enabling GMP is:

resource "google_container_cluster" "cluster_with_no_logging_but_with_gmp" {
  provider           = google
  name               = "cluster-with-no-logging-but-with-gmp"
  location           = "us-central1-a"

  initial_node_count = 1

  # "Cannot specify logging_config or monitoring_config together with logging_service or monitoring_service."
  # logging_service = "logging.googleapis.com/kubernetes"

  logging_config {
    enable_components = []
  }

  # "Cannot specify logging_config or monitoring_config together with logging_service or monitoring_service."
  # monitoring_service = "monitoring.googleapis.com/kubernetes"

  monitoring_config {
    enable_components = []
    managed_prometheus {
      enabled = true
    }
  }
}