Creating a resource group with IBMCLOUD_VISIBILITY=private seems to require a call to public IAM API

[l2fprod]

Template

terraform {
  required_providers {
    ibm = {
      source  = "IBM-Cloud/ibm"
      version = ">= 1.28"
    }
  }
}

provider "ibm" {
}

resource "random_string" "random" {
  length = 8
  special = false
}

resource "ibm_resource_group" "group" {
  name = "${random_string.random.result}-group"
}

Terraform Version

$ terraform version
Terraform v1.0.2
on linux_amd64
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/ibm-cloud/ibm v1.28.0

To reproduce

I have OpenVPN configured so that I can access private endpoints from my machine:

$ ping private.us-south.iam.cloud.ibm.com
PING private.us-south.iam.cloud.ibm.com (166.9.250.226): 56 data bytes
64 bytes from 166.9.250.226: icmp_seq=0 ttl=57 time=27.640 ms
64 bytes from 166.9.250.226: icmp_seq=1 ttl=57 time=27.626 ms

1. Clear DNS cache and monitor DNS calls

   sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder
   sudo tcpdump udp port 53
   

2. Apply

   export IBMCLOUD_VISIBILITY=private
   terraform init
   terraform apply
   

3. Apply works BUT tcpdump shows lookup to iam.cloud.ibm.com public IP address which is unexpected as we should be using the private endpoints:

   10:33:22.521226 IP 10.59.0.6.62827 > 7.00.1aa1.ip4.static.sl-reverse.com.domain: 15794+ AAAA? private.us-south.iam.cloud.ibm.com. (52)
   10:33:22.521254 IP 10.59.0.6.60111 > 7.00.1aa1.ip4.static.sl-reverse.com.domain: 26752+ A? private.us-south.iam.cloud.ibm.com. (52)
   10:33:22.554948 IP 7.00.1aa1.ip4.static.sl-reverse.com.domain > 10.59.0.6.60111: 26752 3/0/0 A 166.9.251.2, A 166.9.250.226, A 166.9.250.194 (202)
   10:33:22.554951 IP 7.00.1aa1.ip4.static.sl-reverse.com.domain > 10.59.0.6.62827: 15794 0/1/0 (164)
   10:33:22.559132 IP 10.59.0.6.62578 > 7.00.1aa1.ip4.static.sl-reverse.com.domain: 10282+ SOA? private.us-south.iam.cloud.ibm.com. (52)
   
   ...
   
   10:33:23.974838 IP 10.59.0.6.65282 > 7.00.1aa1.ip4.static.sl-reverse.com.domain: 10255+ A? iam.cloud.ibm.com. (35)
   10:33:23.974862 IP 10.59.0.6.63524 > 7.00.1aa1.ip4.static.sl-reverse.com.domain: 40379+ AAAA? iam.cloud.ibm.com. (35)
   10:33:24.004897 IP 7.00.1aa1.ip4.static.sl-reverse.com.domain > 10.59.0.6.63524: 40379 0/1/0 (130)
   10:33:24.010329 IP 10.59.0.6.55987 > 7.00.1aa1.ip4.static.sl-reverse.com.domain: 51571+ SOA? iam.cloud.ibm.com. (35)
   10:33:24.024289 IP 7.00.1aa1.ip4.static.sl-reverse.com.domain > 10.59.0.6.65282: 10255 1/0/0 A 184.84.80.38 (68)
   10:33:24.288847 IP 10.59.0.6.63550 > 7.00.1aa1.ip4.static.sl-reverse.com.domain: 38554+ AAAA? private.us-south.resource-controller.cloud.ibm.com. (68)
   10:33:24.292752 IP 10.59.0.6.55719 > 7.00.1aa1.ip4.static.sl-reverse.com.domain: 12911+ A? private.us-south.resource-controller.cloud.ibm.com. (68)
   10:33:24.319472 IP 7.00.1aa1.ip4.static.sl-reverse.com.domain > 10.59.0.6.63550: 38554 1/1/0 CNAME bss.us-south.proxy.serviceendpoint.cloud.ibm.com. (287)
   10:33:24.322391 IP 7.00.1aa1.ip4.static.sl-reverse.com.domain > 10.59.0.6.55719: 12911 3/0/0 CNAME bss.us-south.proxy.serviceendpoint.cloud.ibm.com., A 166.9.250.196, A 166.9.250.228 (308)
   10:33:24.324774 IP 10.59.0.6.58399 > 7.00.1aa1.ip4.static.sl-reverse.com.domain: 18765+ AAAA? bss.us-south.proxy.serviceendpoint.cloud.ibm.com. (66)
   10:33:24.325054 IP 10.59.0.6.51652 > 7.00.1aa1.ip4.static.sl-reverse.com.domain: 38760+ A? bss.us-south.proxy.serviceendpoint.cloud.ibm.com. (66)
   10:33:24.362347 IP 7.00.1aa1.ip4.static.sl-reverse.com.domain > 10.59.0.6.58399: 18765 0/1/0 (173)
   10:33:24.370216 IP 7.00.1aa1.ip4.static.sl-reverse.com.domain > 10.59.0.6.51652: 38760 2/0/0 A 166.9.250.228, A 166.9.250.196 (194)
   10:33:25.309721 IP 10.59.0.6.57911 > 7.00.1aa1.ip4.static.sl-reverse.com.domain: 47472+ SOA? bss.us-south.proxy.serviceendpoint.cloud.ibm.com. (66)
   

4. Destroy

   terraform destroy
   

5. If I add 127.0.0.1 iam.cloud.ibm.com to my /etc/hosts and run apply again:

   │ Error: Error creating resource group: An error occurred while performing the 'authenticate' step: Post "https://iam.cloud.ibm.com/identity/token": dial tcp 127.0.0.1:443: connect: connection refused with responce code  null
   │
   │
   │   with ibm_resource_group.group,
   │   on main.tf line 32, in resource "ibm_resource_group" "group":
   │   32: resource "ibm_resource_group" "group" {
   │

   It clearly shows a call to iam.cloud.ibm.com. I did not expect this when using private.

[l2fprod]

It works if I define export IBMCLOUD_IAM_API_ENDPOINT=https://private.us-south.iam.cloud.ibm.com. Likely the issue comes from this hardcoded path:

terraform-provider-ibm/ibm/config.go

Line 1139 in e94ae1a

if c.BluemixAPIKey != "" {