Error setting aws_dms_endpoint with extra_connection_attributes setting cdcMaxBatchInterval

[mikebollandajw]

Terraform Version

0.11.11
"aws" (2.0.0)...

Affected Resource(s)

aws_dms_endpoint
extra_connection_attributes

Terraform Configuration Files

resource "aws_dms_endpoint" "dms-s3-endpoint-target" {
  endpoint_id                 = "s3-${var.be_region}-${var.bucket_suffix}-adsprocessor2"
  endpoint_type               = "target"
  engine_name                 = "s3"
  extra_connection_attributes = "cdcMaxBatchInterval=10;compressionType=NONE;csvDelimiter=,;csvRowDelimiter="

  s3_settings = {
    service_access_role_arn = "${aws_iam_role.dms-iam-role.arn}"
    bucket_name             = "s3-${var.be_region}-${var.shortenv}-${var.bucket_suffix}"
    bucket_folder           = "${var.target_bucket_folder}"
    compression_type        = "NONE"
  }
}

Debug Output

Terraform will perform the following actions:

      extra_connection_attributes: "compressionType=none;csvDelimiter=,;csvRowDelimiter=\n;" => "cdcMaxBatchInterval=10;compressionType=none;csvDelimiter=,;csvRowDelimiter=\n;"

Panic Output

StartReplicationTask operation: Test connection for replication instance dms-eu-west-2-s-adsprocessor and endpointname should be successful for starting the replication task

Endpoint test result: Test Endpoint failed: Application-Status: 1020912, Application-Message: Failed to connect to database.

Expected Behavior

The endpoint extra connection attribute cdcMaxBatchInterval should exist

The bucket name should be set in the endpoint

Actual Behavior

-The endpoint loses its bucket name (in the aws console) and test fails when attempted

-Setting the bucket name back into the endpoint and testing still fails test

-The endpoint must be destroyed to fix the issue

-You cannot destroy an endpoint that is in use by a task, therefore you must destroy everything else

-AWS support cannot resolve the issue without destroying the endpoint

-They say the issue is terraform

Steps to Reproduce

1. terraform apply

Important Factoids

No

[mikebollandajw]

Manually defining the bucketName in the extra connection attribute will get around the endpoint corruption

However cdcMaxBatchInterval will be missing from the endpoint if viewed via DMS task manager - the state however will show it as there...

extra_connection_attributes: "compressionType=none;csvDelimiter=,;csvRowDelimiter=\n;" => "bucketName=s3-eu-west-2-s-ads-staging-cdc;cdcMaxBatchInterval=10compressionType=none;csvDelimiter=,;csvRowDelimiter=\n;"

ie

resource "aws_dms_endpoint" "dms-s3-endpoint-target" {
  endpoint_id                 = "s3-${var.be_region}-${var.bucket_suffix}-adsprocessor2"
  endpoint_type               = "target"
  engine_name                 = "s3"
  extra_connection_attributes = "bucketName=s3-${var.be_region}-${var.shortenv}-${var.bucket_suffix};${var.target_extra_connection_attributes}"

  s3_settings = {
    service_access_role_arn = "${aws_iam_role.dms-iam-role.arn}"
    bucket_name             = "s3-${var.be_region}-${var.shortenv}-${var.bucket_suffix}"
    bucket_folder           = "${var.target_bucket_folder}"
    compression_type        = "NONE"
  }
}

So i believe the endpoint corruption issue is something to do with the bucketName not being automatically appended when you use the cdcMaxBatchInterval setting

And finally that the cdcMaxBatchInterval is not being applied to the endpoint perhaps due to it being an unknown property

[ewbankkit]

Related:

• S3 settings on aws_dms_endpoint conflict with "extra_connection_attributes" #8009
• DMS endpoint s3 target to support extra_connection_attributes #8000

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.