feat: Added support for GCP Private Service Connect

README.md

@@ -67,7 +67,9 @@ resources that lack official modules.
 
 ## Providers
 
-No providers.
+| Name | Version |
+|------|---------|
+| <a name="provider_google"></a> [google](#provider\_google) | ~> 4.82 |
 
 ## Modules
 
@@ -79,6 +81,7 @@ No providers.
 | <a name="module_gke_app"></a> [gke\_app](#module\_gke\_app) | wandb/wandb/kubernetes | 1.14.1 |
 | <a name="module_kms"></a> [kms](#module\_kms) | ./modules/kms | n/a |
 | <a name="module_networking"></a> [networking](#module\_networking) | ./modules/networking | n/a |
+| <a name="module_private_link"></a> [private\_link](#module\_private\_link) | ./modules/private_link | n/a |
 | <a name="module_project_factory_project_services"></a> [project\_factory\_project\_services](#module\_project\_factory\_project\_services) | terraform-google-modules/project-factory/google//modules/project_services | ~> 13.0 |
 | <a name="module_redis"></a> [redis](#module\_redis) | ./modules/redis | n/a |
 | <a name="module_service_accounts"></a> [service\_accounts](#module\_service\_accounts) | ./modules/service_accounts | n/a |
@@ -87,15 +90,18 @@ No providers.
 
 ## Resources
 
-No resources.
+| Name | Type |
+|------|------|
 
 ## Inputs
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
 | <a name="input_allowed_inbound_cidrs"></a> [allowed\_inbound\_cidrs](#input\_allowed\_inbound\_cidrs) | Which IPv4 addresses/ranges to allow access. This must be explicitly provided, and by default is set to ["*"] | `list(string)` | <pre>[<br>  "*"<br>]</pre> | no |
+| <a name="input_allowed_projects"></a> [allowed\_projects](#input\_allowed\_projects) | A map of allowed projects where each key is a project number and the value is the connection limit. | `map(number)` | `{}` | no |
 | <a name="input_app_wandb_env"></a> [app\_wandb\_env](#input\_app\_wandb\_env) | Extra environment variables for W&B | `map(string)` | `{}` | no |
 | <a name="input_bucket_name"></a> [bucket\_name](#input\_bucket\_name) | Use an existing bucket. | `string` | `""` | no |
+| <a name="input_create_private_link"></a> [create\_private\_link](#input\_create\_private\_link) | Whether to create a private link service. | `bool` | `false` | no |
 | <a name="input_create_redis"></a> [create\_redis](#input\_create\_redis) | Boolean indicating whether to provision an redis instance (true) or not (false). | `bool` | `false` | no |
 | <a name="input_database_machine_type"></a> [database\_machine\_type](#input\_database\_machine\_type) | Specifies the machine type to be allocated for the database | `string` | `"db-n1-standard-2"` | no |
 | <a name="input_database_sort_buffer_size"></a> [database\_sort\_buffer\_size](#input\_database\_sort\_buffer\_size) | Specifies the sort\_buffer\_size value to set for the database | `number` | `67108864` | no |
@@ -106,6 +112,7 @@ No resources.
 | <a name="input_force_ssl"></a> [force\_ssl](#input\_force\_ssl) | Enforce SSL through the usage of the Cloud SQL Proxy (cloudsql://) in the DB connection string | `bool` | `false` | no |
 | <a name="input_gke_machine_type"></a> [gke\_machine\_type](#input\_gke\_machine\_type) | Specifies the machine type to be allocated for the database | `string` | `"n1-standard-4"` | no |
 | <a name="input_gke_node_count"></a> [gke\_node\_count](#input\_gke\_node\_count) | n/a | `number` | `2` | no |
+| <a name="input_ilb_proxynetwork_cidr"></a> [ilb\_proxynetwork\_cidr](#input\_ilb\_proxynetwork\_cidr) | Internal load balancer proxy subnetwork | `string` | `"10.127.0.0/24"` | no |
 | <a name="input_labels"></a> [labels](#input\_labels) | Labels to apply to resources | `map(string)` | `{}` | no |
 | <a name="input_license"></a> [license](#input\_license) | Your wandb/local license | `string` | n/a | yes |
 | <a name="input_local_restore"></a> [local\_restore](#input\_local\_restore) | Restores W&B to a stable state if needed | `bool` | `false` | no |
@@ -117,6 +124,8 @@ No resources.
 | <a name="input_oidc_secret"></a> [oidc\_secret](#input\_oidc\_secret) | The Client secret of application in your identity provider | `string` | `""` | no |
 | <a name="input_other_wandb_env"></a> [other\_wandb\_env](#input\_other\_wandb\_env) | Extra environment variables for W&B | `map(string)` | `{}` | no |
 | <a name="input_parquet_wandb_env"></a> [parquet\_wandb\_env](#input\_parquet\_wandb\_env) | Extra environment variables for W&B | `map(string)` | `{}` | no |
+| <a name="input_psc_subnetwork_cidr"></a> [psc\_subnetwork\_cidr](#input\_psc\_subnetwork\_cidr) | Private link service reserved subnetwork | `string` | `"192.168.0.0/24"` | no |
+| <a name="input_public_access"></a> [public\_access](#input\_public\_access) | Whether to create a public endpoint for wandb access. | `bool` | `true` | no |
 | <a name="input_redis_reserved_ip_range"></a> [redis\_reserved\_ip\_range](#input\_redis\_reserved\_ip\_range) | Reserved IP range for REDIS peering connection | `string` | `"10.30.0.0/16"` | no |
 | <a name="input_redis_tier"></a> [redis\_tier](#input\_redis\_tier) | Specifies the tier for this Redis instance | `string` | `"STANDARD_HA"` | no |
 | <a name="input_resource_limits"></a> [resource\_limits](#input\_resource\_limits) | Specifies the resource limits for the wandb deployment | `map(string)` | <pre>{<br>  "cpu": null,<br>  "memory": null<br>}</pre> | no |
@@ -150,6 +159,7 @@ No resources.
 | <a name="output_fqdn"></a> [fqdn](#output\_fqdn) | The FQDN to the W&B application |
 | <a name="output_gke_node_count"></a> [gke\_node\_count](#output\_gke\_node\_count) | n/a |
 | <a name="output_gke_node_instance_type"></a> [gke\_node\_instance\_type](#output\_gke\_node\_instance\_type) | n/a |
+| <a name="output_private_attachement_id"></a> [private\_attachement\_id](#output\_private\_attachement\_id) | n/a |
 | <a name="output_service_account"></a> [service\_account](#output\_service\_account) | Weights & Biases service account used to manage resources. |
 | <a name="output_standardized_size"></a> [standardized\_size](#output\_standardized\_size) | n/a |
 | <a name="output_url"></a> [url](#output\_url) | The URL to the W&B application |

main.tf

@@ -136,6 +136,29 @@ locals {
   secret_store_source     = "gcp-secretmanager://${local.project_id}?namespace=${var.namespace}"
 }
 
+
+resource "google_compute_address" "address" {
+  count        = var.create_private_link ? 1 : 0
+  name         = "${var.namespace}-ip-address"
+  subnetwork   = local.subnetwork.name
+  address_type = "INTERNAL"
+  purpose      = "GCE_ENDPOINT"
+  depends_on   = [module.app_gke]
+}
+
+module "private_link" {
+  count             = var.create_private_link ? 1 : 0
+  source            = "./modules/private_link"
+  namespace         = var.namespace
+  ingress_name      = "${var.namespace}-internal"
+  network           = local.network
+  subnetwork        = local.subnetwork
+  allowed_projects  = var.allowed_projects
+  psc_subnetwork    = var.psc_subnetwork_cidr
+  proxynetwork_cidr = var.ilb_proxynetwork_cidr
+  depends_on        = [module.app_gke, module.wandb]
+}
+
 module "gke_app" {
   source  = "wandb/wandb/kubernetes"
   version = "1.14.1"
@@ -235,14 +258,22 @@ module "wandb" {
       }
 
       ingress = {
+        create       = var.public_access # external ingress for public connection
         nameOverride = var.namespace
         annotations = {
           "kubernetes.io/ingress.class"                 = "gce"
           "kubernetes.io/ingress.global-static-ip-name" = module.app_lb.address_operator_name
           "ingress.gcp.kubernetes.io/pre-shared-cert"   = module.app_lb.certificate
         }
+        secondary = var.create_private_link ? {
+          create       = var.create_private_link # internal ingress for private link connections
+          nameOverride = "${var.namespace}-internal"
+          annotations = {
+            "kubernetes.io/ingress.class"                   = "gce-internal"
+            "kubernetes.io/ingress.regional-static-ip-name" = google_compute_address.address.0.name
+          }
+        } : null
       }
-
       redis = { install = false }
       mysql = { install = false }
 

modules/app_gke/outputs.tf

@@ -32,4 +32,6 @@ output "instance_group_url" {
 output "node_pool" {
   value = google_container_node_pool.default
 }
-
+output "mig_instance_group_id" {
+  value = google_container_node_pool.default.managed_instance_group_urls[0]
+}

modules/private_link/main.tf

@@ -0,0 +1,66 @@
+data "kubernetes_ingress_v1" "ingress" {
+  metadata {
+    name = var.ingress_name
+  }
+}
+
+locals {
+  lb_name = data.kubernetes_ingress_v1.ingress.metadata[0].annotations != null ? data.kubernetes_ingress_v1.ingress.metadata[0].annotations["ingress.kubernetes.io/forwarding-rule"] : ""
+}
+
+resource "google_compute_service_attachment" "attachment" {
+  name                  = "${var.namespace}-private-link"
+  enable_proxy_protocol = false
+  connection_preference = "ACCEPT_MANUAL"
+  nat_subnets           = [google_compute_subnetwork.subnet.id]
+  target_service        =  local.lb_name
+
+ dynamic "consumer_accept_lists" {
+    for_each = var.allowed_projects != {} ? var.allowed_projects : {}
+    content {
+      project_id_or_num = consumer_accept_lists.key
+      connection_limit  = consumer_accept_lists.value
+    }
+  }
+  depends_on = [ data.kubernetes_ingress_v1.ingress ]
+}
+
+resource "google_compute_subnetwork" "subnet" {
+  name          = "${var.namespace}-psc-ilb-subnet"
+  network       = var.network.id
+  purpose       = "PRIVATE_SERVICE_CONNECT"
+  ip_cidr_range = var.psc_subnetwork
+}
+
+# proxy-only subnet
+resource "google_compute_subnetwork" "proxy_subnet" {
+  name          = "${var.namespace}-proxy-subnet"
+  provider      = google-beta
+  ip_cidr_range = var.proxynetwork_cidr
+  purpose       = "REGIONAL_MANAGED_PROXY"
+  role          = "ACTIVE"
+  network       = var.network.id
+}
+# allow all access from IAP and health check ranges
+resource "google_compute_firewall" "fw_iap" {
+  name          = "${var.namespace}-internal-fw"
+  provider      = google-beta
+  direction     = "INGRESS"
+  network       = var.network.id
+  source_ranges = ["130.211.0.0/22", "35.191.0.0/16", "35.235.240.0/20"]
+  allow {
+    protocol = "tcp"
+  }
+}
+
+# allow tcp from proxy subnet to backends
+resource "google_compute_firewall" "rule" {
+  name          = "${var.namespace}-fw-allow-iap-hc"
+  provider      = google-beta
+  direction     = "INGRESS"
+  network       = var.network.id
+  source_ranges = [var.proxynetwork_cidr]
+  allow {
+    protocol = "tcp"
+  }
+}

modules/private_link/outputs.tf

@@ -0,0 +1,3 @@
+output "private_attachement_id" {
+  value = google_compute_service_attachment.attachment.id
+}

modules/private_link/variables.tf

@@ -0,0 +1,41 @@
+variable "namespace" {
+  type        = string
+  description = "The name prefix for all resources created."
+}
+
+variable "labels" {
+  description = "Labels which will be applied to all applicable resources."
+  type        = map(string)
+  default     = {}
+}
+
+variable "network" {
+  description = "Google Compute Engine network to which the cluster is connected."
+  type        = object({ id = string })
+}
+
+
+variable "subnetwork" {
+  type = object({
+    self_link = string
+  })
+  description = "The subnetwork object containing the self-link of the subnetwork."
+}
+
+variable "allowed_projects" {
+  type = map(number)
+  default = {}
+  description = "A map of allowed projects where each key is a project number and the value is the connection limit."
+}
+
+variable "ingress_name" {
+  description = "Name of the ingress resources which was created by wandb module"
+  type        = string
+}
+
+variable "psc_subnetwork" {
+  type        = string
+}
+variable "proxynetwork_cidr" {
+  type        = string
+}

outputs.tf

@@ -85,5 +85,6 @@ output "database_instance_type" {
   value = coalesce(try(local.deployment_size[var.size].db, null), var.database_machine_type)
 }
 
-
-
+output "private_attachement_id" {
+  value = var.create_private_link ? module.private_link[0].private_attachement_id : null
+}

variables.tf

@@ -253,3 +253,39 @@ variable "parquet_wandb_env" {
   description = "Extra environment variables for W&B"
   default     = {}
 }
+
+##########################################
+# private link                           #
+##########################################
+
+variable "create_private_link" {
+  type        = bool
+  description = "Whether to create a private link service."
+  default     = false
+}
+
+variable "public_access" {
+  type        = bool
+  description = "Whether to create a public endpoint for wandb access."
+  default = true
+}
+
+variable "allowed_projects" {
+  type = map(number)
+  default = {
+    # "482878270665" = 4
+  }
+  description = "A map of allowed projects where each key is a project number and the value is the connection limit."
+}
+
+variable "psc_subnetwork_cidr" {
+  default     = "192.168.0.0/24"
+  description = "Private link service reserved subnetwork"
+  type        = string
+}
+
+variable "ilb_proxynetwork_cidr" {
+  default     = "10.127.0.0/24"
+  description = "Internal load balancer proxy subnetwork"
+  type        = string
+}