IBM Cloud Terraform Provider is unable to upgrade the "vpc-block-csi-driver" addon on VPC Gen 2 Clusters

[bemahone]

Terraform Version

Terraform 13 w/ v1.29.0 IBM Cloud provider (running under schematics)

Affected Resource(s)

ibm_container_addons

Description of Issue:

The IBM Cloud Terraform Provider is unable to upgrade the "vpc-block-csi-driver" addon on VPC Gen 2 clusters. The cloud team releases new versions of this addon over time. These could be bug fixes, feature enhancements or security updates.

If iks_cluster_addons (map) is set to {"vpc-block-csi-driver":"3.0.1"}, but the cluster is running the 3.0.0 level of the plugin, the terraform/schematics apply will fail with an error similar to the message below (this example is executing against a VPC Gen 2 cluster):

 2021/08/16 15:38:28 Terraform apply | Error: Request failed with status code: 400, ServerErrorResponse: {"incidentID":"58a21827-6dc7-4557-a38e-77304ceacad4,58a21827-6dc7-4557-a38e-77304ceacad4","code":"Ec98d","description":"The ''vpc-block-csi-driver'' addon is not supported for the given provider. Supported providers are 'vpc-classic', 'vpc-gen2'.","type":"Provisioning"}
 2021/08/16 15:38:28 Terraform apply | 
 2021/08/16 15:38:28 Terraform apply |   on cluster/addons.tf line 5, in resource "ibm_container_addons" "addons":
 2021/08/16 15:38:28 Terraform apply |    5: resource "ibm_container_addons" "addons" {

The only way we've found to circumvent the problem is to do the following:

• Disable vpc-block-csi-driver plugin on cluster manually:
  • ibmcloud ks cluster addon disable vpc-block-csi-driver --cluster <cluster name>
• Enable vpc-block-csi-driver plugin on cluster manually:
  • ibmcloud ks cluster addon enable vpc-block-csi-driver --cluster <cluster name>
• Verify new version is running via:
  • ibmcloud ks cluster addon ls --cluster <cluster name>
• Verify var.iks_cluster_addons is set in our terraform to match the current running version of the plugin on the cluster

Sample terraform for installing addons:

##############################################################################
# Create Cluster Addons in VPC
##############################################################################
resource "ibm_container_addons" "addons" {
  count  	= length(var.iks_cluster_addons) > 0 && (var.create_iks_cluster) ? var.iks_cluster_count : 0
  cluster 	= format("%s-%s", var.iks_cluster_name, (count.index + 1) )
  depends_on = [
	  ibm_container_vpc_cluster.cluster,
	  ibm_container_vpc_worker_pool.ingress
	  ]
  resource_group_id = var.resource_group_id
 
  ###### APPEND IN RULES HERE #########################
   	dynamic "addons" {
  	iterator = addon 
    for_each = var.iks_cluster_addons  
    
	    content {
                     name        = addon.key
                     version     = addon.value
			
	  }
   
	}

	
  	timeouts {
  		create = var.iks_cluster_create_or_update_timeout
		update = var.iks_cluster_create_or_update_timeout
  	}	 
}

Sample variables.json:

  {
      "name": "iks_cluster_addons",
      "type": "map(string)",
      "value": "{\"alb-oauth-proxy\":\"1.0.0\",\"vpc-block-csi-driver\":\"3.0.0\"}"
    },

Additional Notes:

• Setting the {"vpc-block-csi-driver":"3.0.1"} to null or "" does not appear to help things move along, similar error messages as per above are displayed during the schematics/terraform apply failure
• What we need is a way to specify a requirement to install the addon, but also a way to upgrade it successfully
  • Adding a feature to automatically disable the addon and enable the addon if a new version is available would resolve this issue, as long as the current version could be updated in the terraform or be excluded moving forward
• The IBM documentation appears to indicate disabling and enabling the plugin is the way to update it:
  • To update the Block Storage for VPC add-on in your cluster, disable the add-on and then re-enable the add-on. You might see a warning that resources or data might be deleted. For the IBM® Cloud Block Storage for Virtual Private Cloud add-on update, PVC creation and app deployment are not disrupted when the add-on is disabled and existing volumes are not impacted.
    • https://cloud.ibm.com/docs/containers?topic=containers-vpc_bs_changelog