Skip to content

Commit

Permalink
Msk features (#451)
Browse files Browse the repository at this point in the history
  • Loading branch information
@bcarranza
bcarranza authored Dec 14, 2023
1 parent df1abfc commit c8193c7
Show file tree
Hide file tree
Showing 23 changed files with 2,244 additions and 0 deletions.
89 changes: 89 additions & 0 deletions terraform-modules/aws/msk_1.0.9/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
## Requirements

No requirements.

## Providers

| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | n/a |

## Modules

| Name | Source | Version |
|------|--------|---------|
| <a name="module_msk"></a> [msk](#module\_msk) | ./module | n/a |

## Resources

| Name | Type |
|------|------|
| [aws_acmpca_certificate.cert](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/acmpca_certificate) | resource |
| [aws_acmpca_certificate_authority.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/acmpca_certificate_authority) | resource |
| [aws_acmpca_certificate_authority_certificate.cacert](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/acmpca_certificate_authority_certificate) | resource |
| [aws_cloudwatch_log_group.msk_cloudwatch_log_group](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_group) | resource |
| [aws_kms_key.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kms_key) | resource |
| [aws_s3_bucket.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket) | resource |
| [aws_s3_bucket_policy.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_policy) | resource |
| [aws_iam_policy_document.acmpca_bucket_access](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws_partition.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/partition) | data source |

## Inputs

| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| <a name="input_allowed_security_group_ids"></a> [allowed\_security\_group\_ids](#input\_allowed\_security\_group\_ids) | The security\_group\_id\_list output from the security\_groups module | `list(string)` | n/a | yes |
| <a name="input_autoscaling_enabled"></a> [autoscaling\_enabled](#input\_autoscaling\_enabled) | To automatically expand your cluster's storage in response to increased usage, you can enable this. [More info](https://docs.aws.amazon.com/msk/latest/developerguide/msk-autoexpand.html) | `bool` | `true` | no |
| <a name="input_aws_region"></a> [aws\_region](#input\_aws\_region) | The AWS region you want to deploy to | `string` | n/a | yes |
| <a name="input_broker_instance_type"></a> [broker\_instance\_type](#input\_broker\_instance\_type) | The instance type to use for the Kafka brokers | `string` | `"kafka.t3.small"` | no |
| <a name="input_broker_per_zone"></a> [broker\_per\_zone](#input\_broker\_per\_zone) | Number of Kafka brokers per zone. | `number` | `1` | no |
| <a name="input_broker_volume_size"></a> [broker\_volume\_size](#input\_broker\_volume\_size) | The size in GiB of the EBS volume for the data drive on each broker node | `number` | n/a | yes |
| <a name="input_client_broker"></a> [client\_broker](#input\_client\_broker) | Encryption setting for data in transit between clients and brokers. Valid values: TLS, TLS\_PLAINTEXT, and PLAINTEXT | `string` | n/a | yes |
| <a name="input_client_sasl_iam_enabled"></a> [client\_sasl\_iam\_enabled](#input\_client\_sasl\_iam\_enabled) | Enables client authentication via IAM policies (cannot be set to true at the same time as client\_sasl\_*\_enabled). | `bool` | `false` | no |
| <a name="input_client_sasl_scram_enabled"></a> [client\_sasl\_scram\_enabled](#input\_client\_sasl\_scram\_enabled) | Enables SCRAM client authentication via AWS Secrets Manager (cannot be set to `true` at the same time as `client_tls_auth_enabled`). | `bool` | `false` | no |
| <a name="input_client_sasl_scram_secret_association_arns"></a> [client\_sasl\_scram\_secret\_association\_arns](#input\_client\_sasl\_scram\_secret\_association\_arns) | List of AWS Secrets Manager secret ARNs for scram authentication (cannot be set to `true` at the same time as `client_tls_auth_enabled`). | `list(string)` | `[]` | no |
| <a name="input_client_tls_auth_enabled"></a> [client\_tls\_auth\_enabled](#input\_client\_tls\_auth\_enabled) | Set true to enable the Client TLS Authentication | `bool` | n/a | yes |
| <a name="input_cloudwatch_logs_enabled"></a> [cloudwatch\_logs\_enabled](#input\_cloudwatch\_logs\_enabled) | Indicates whether you want to enable or disable streaming broker logs to Cloudwatch Logs | `bool` | n/a | yes |
| <a name="input_cloudwatch_logs_log_group"></a> [cloudwatch\_logs\_log\_group](#input\_cloudwatch\_logs\_log\_group) | Name of the Cloudwatch Log Group to deliver logs to | `string` | n/a | yes |
| <a name="input_common_name"></a> [common\_name](#input\_common\_name) | The common name for the CA | `string` | `"example.com"` | no |
| <a name="input_create_private_ca"></a> [create\_private\_ca](#input\_create\_private\_ca) | Boolean flag to control if the resources about CA Private should be created | `bool` | `false` | no |
| <a name="input_encryption_at_rest_kms_key_arn"></a> [encryption\_at\_rest\_kms\_key\_arn](#input\_encryption\_at\_rest\_kms\_key\_arn) | You may specify a KMS key short ID or ARN (it will always output an ARN) to use for encrypting your data at rest. If null the key created in this module will be used. | `string` | `null` | no |
| <a name="input_encryption_in_cluster"></a> [encryption\_in\_cluster](#input\_encryption\_in\_cluster) | Whether data communication among broker nodes is encrypted | `bool` | n/a | yes |
| <a name="input_enhanced_monitoring"></a> [enhanced\_monitoring](#input\_enhanced\_monitoring) | Specify the desired enhanced MSK CloudWatch monitoring level. Valid values: DEFAULT, PER\_BROKER, and PER\_TOPIC\_PER\_BROKER | `string` | n/a | yes |
| <a name="input_expiration_in_days"></a> [expiration\_in\_days](#input\_expiration\_in\_days) | The CA expiration in days | `number` | `7` | no |
| <a name="input_jmx_exporter_enabled"></a> [jmx\_exporter\_enabled](#input\_jmx\_exporter\_enabled) | Set true to enable the Prometheus JMX Exporter | `bool` | `false` | no |
| <a name="input_kafka_version"></a> [kafka\_version](#input\_kafka\_version) | The desired Kafka software version | `string` | `"2.8.1"` | no |
| <a name="input_key_algorithm"></a> [key\_algorithm](#input\_key\_algorithm) | The CA key algorithm | `string` | `"RSA_4096"` | no |
| <a name="input_multi_vpc_connectivity_enabled"></a> [multi\_vpc\_connectivity\_enabled](#input\_multi\_vpc\_connectivity\_enabled) | Turn on the vpc connectivitiy | `bool` | `false` | no |
| <a name="input_multi_vpc_connectivity_iam_enabled"></a> [multi\_vpc\_connectivity\_iam\_enabled](#input\_multi\_vpc\_connectivity\_iam\_enabled) | Turn on the vpc connectivitiy in IAM mode | `bool` | `false` | no |
| <a name="input_name"></a> [name](#input\_name) | Solution name | `string` | n/a | yes |
| <a name="input_namespace"></a> [namespace](#input\_namespace) | Namespace, which could be your organization name or abbreviation, | `string` | n/a | yes |
| <a name="input_node_exporter_enabled"></a> [node\_exporter\_enabled](#input\_node\_exporter\_enabled) | Set true to enable the Prometheus Node Exporter | `bool` | `false` | no |
| <a name="input_properties"></a> [properties](#input\_properties) | Contents of the server.properties file. Supported properties are documented in the [MSK Developer Guide](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-properties.html) | `map(string)` | `{}` | no |
| <a name="input_s3_bucket_create"></a> [s3\_bucket\_create](#input\_s3\_bucket\_create) | Set it to true if you want the process to create the S3 bucket for you, and false if you already have one. | `bool` | `false` | no |
| <a name="input_s3_logs_bucket"></a> [s3\_logs\_bucket](#input\_s3\_logs\_bucket) | Name of the S3 bucket to deliver logs to (only able if (s3\_bucket\_create = true), It is going to create a new resource by you | `string` | n/a | yes |
| <a name="input_s3_logs_enabled"></a> [s3\_logs\_enabled](#input\_s3\_logs\_enabled) | Indicates whether you want to enable or disable streaming broker logs to S3 | `bool` | n/a | yes |
| <a name="input_s3_logs_prefix"></a> [s3\_logs\_prefix](#input\_s3\_logs\_prefix) | Prefix to append to the S3 folder name logs are delivered to | `string` | `""` | no |
| <a name="input_security_groups"></a> [security\_groups](#input\_security\_groups) | DEPRECATED: Use `allowed_security_group_ids` instead.<br>List of security group IDs to be allowed to connect to the cluster | `list(string)` | `[]` | no |
| <a name="input_signing_algorithm"></a> [signing\_algorithm](#input\_signing\_algorithm) | The CA signing algorithm | `string` | `"SHA512WITHRSA"` | no |
| <a name="input_subnet_ids"></a> [subnet\_ids](#input\_subnet\_ids) | Subnet IDs for Client Broker | `list(string)` | n/a | yes |
| <a name="input_tags"></a> [tags](#input\_tags) | Additional tags | `map(any)` | n/a | yes |
| <a name="input_vpc_id"></a> [vpc\_id](#input\_vpc\_id) | The VPC id of where you want to provision MSK | `string` | n/a | yes |
| <a name="input_years_valid"></a> [years\_valid](#input\_years\_valid) | The Certificate will be valid for a period of x years from the time it is issued. | `number` | `10` | no |
| <a name="input_zone_id"></a> [zone\_id](#input\_zone\_id) | Route53 DNS Zone ID for MSK broker hostnames | `string` | n/a | yes |

## Outputs

| Name | Description |
|------|-------------|
| <a name="output_bootstrap_brokers"></a> [bootstrap\_brokers](#output\_bootstrap\_brokers) | A comma separated list of one or more hostname:port pairs of kafka brokers suitable to boostrap connectivity to the kafka cluster |
| <a name="output_bootstrap_brokers_tls"></a> [bootstrap\_brokers\_tls](#output\_bootstrap\_brokers\_tls) | A comma separated list of one or more DNS names (or IPs) and TLS port pairs kafka brokers suitable to boostrap connectivity to the kafka cluster |
| <a name="output_cluster_arn"></a> [cluster\_arn](#output\_cluster\_arn) | Amazon Resource Name (ARN) of the MSK cluster |
| <a name="output_cluster_name"></a> [cluster\_name](#output\_cluster\_name) | MSK Cluster name |
| <a name="output_cluster_uuid"></a> [cluster\_uuid](#output\_cluster\_uuid) | UUID of the MSK cluster, for use in IAM policies |
| <a name="output_config_arn"></a> [config\_arn](#output\_config\_arn) | Amazon Resource Name (ARN) of the configuration |
| <a name="output_current_version"></a> [current\_version](#output\_current\_version) | Current version of the MSK Cluster used for updates |
| <a name="output_latest_revision"></a> [latest\_revision](#output\_latest\_revision) | Latest revision of the configuration |
| <a name="output_security_group_id"></a> [security\_group\_id](#output\_security\_group\_id) | The ID of the security group rule |
| <a name="output_security_group_name"></a> [security\_group\_name](#output\_security\_group\_name) | The name of the security group rule |
| <a name="output_zookeeper_connect_string"></a> [zookeeper\_connect\_string](#output\_zookeeper\_connect\_string) | A comma separated list of one or more hostname:port pairs to use to connect to the Apache Zookeeper cluster |
152 changes: 152 additions & 0 deletions terraform-modules/aws/msk_1.0.9/main.tf
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
locals{
years_valid = var.years_valid
}

resource "aws_cloudwatch_log_group" "msk_cloudwatch_log_group" {
name = var.cloudwatch_logs_log_group
tags = var.tags
}

#######################################
# S3 bucket
#######################################
resource "aws_kms_key" "this" {
description = "This key is used to encrypt bucket objects"
deletion_window_in_days = 10
}

resource "aws_s3_bucket" "this" {
count = var.s3_bucket_create ? 1 : 0
bucket = var.s3_logs_bucket
tags = var.tags
}

data "aws_iam_policy_document" "acmpca_bucket_access" {
count = var.s3_bucket_create ? 1 : 0
statement {
actions = [
"s3:GetBucketAcl",
"s3:GetBucketLocation",
"s3:PutObject",
"s3:PutObjectAcl",
]

resources = [
aws_s3_bucket.this[0].arn,
"${aws_s3_bucket.this[0].arn}/*",
]

principals {
identifiers = ["acm-pca.amazonaws.com"]
type = "Service"
}
}
}

resource "aws_s3_bucket_policy" "this" {
count = var.s3_bucket_create ? 1 : 0
bucket = aws_s3_bucket.this[0].id
policy = data.aws_iam_policy_document.acmpca_bucket_access[0].json
}

#######################################
# Private CA
#######################################
data "aws_partition" "current" {
}

resource "aws_acmpca_certificate_authority_certificate" "cacert" {
count = var.create_private_ca ? 1 : 0
certificate_authority_arn = aws_acmpca_certificate_authority.this[0].arn

certificate = aws_acmpca_certificate.cert[0].certificate
certificate_chain = aws_acmpca_certificate.cert[0].certificate_chain
}

resource "aws_acmpca_certificate" "cert" {
count = var.create_private_ca ? 1 : 0
certificate_authority_arn = aws_acmpca_certificate_authority.this[0].arn
certificate_signing_request = aws_acmpca_certificate_authority.this[0].certificate_signing_request
signing_algorithm = "SHA512WITHRSA"

template_arn = "arn:${data.aws_partition.current.partition}:acm-pca:::template/RootCACertificate/V1"

validity {
type = "YEARS"
value = local.years_valid
}
}

resource "aws_acmpca_certificate_authority" "this" {
count = var.create_private_ca ? 1 : 0
certificate_authority_configuration {
key_algorithm = var.key_algorithm
signing_algorithm = var.signing_algorithm

subject {
common_name = var.common_name
}
}

type = "ROOT"

revocation_configuration {
crl_configuration {
custom_cname = "crl.${var.common_name}"
enabled = true
expiration_in_days = var.expiration_in_days
s3_bucket_name = var.s3_logs_bucket
}
}

tags = var.tags

depends_on = [aws_s3_bucket_policy.this]
}

#######################################
# MSK Cluster
#######################################
module "msk" {
source = "./module"
#version = "v2.0.0"
namespace = var.namespace
name = var.name
vpc_id = var.vpc_id
client_broker = var.client_broker
zone_id = var.zone_id
security_groups = var.security_groups
subnet_ids = var.subnet_ids
kafka_version = var.kafka_version
#it was deprecated , now the param is number_of_broker_nodes
# number_of_broker_nodes = var.number_of_broker_nodes
broker_per_zone = var.broker_per_zone
broker_instance_type = var.broker_instance_type
broker_volume_size = var.broker_volume_size
tags = var.tags
certificate_authority_arns = var.create_private_ca ? [aws_acmpca_certificate_authority.this[0].arn] : []
client_tls_auth_enabled = var.client_tls_auth_enabled
client_sasl_iam_enabled = var.client_sasl_iam_enabled
client_sasl_scram_enabled = var.client_sasl_scram_enabled
client_sasl_scram_secret_association_arns = var.client_sasl_scram_secret_association_arns
encryption_in_cluster = var.encryption_in_cluster
encryption_at_rest_kms_key_arn = var.encryption_at_rest_kms_key_arn != null ? var.encryption_at_rest_kms_key_arn : aws_kms_key.this.arn
cloudwatch_logs_enabled = var.cloudwatch_logs_enabled
cloudwatch_logs_log_group = var.cloudwatch_logs_enabled == true ? var.cloudwatch_logs_log_group : ""
enhanced_monitoring = var.enhanced_monitoring
node_exporter_enabled = var.node_exporter_enabled
jmx_exporter_enabled = var.jmx_exporter_enabled
s3_logs_bucket = var.s3_logs_enabled == true ? var.s3_logs_bucket : ""
s3_logs_enabled = var.s3_logs_enabled
s3_logs_prefix = var.s3_logs_enabled == true ? var.s3_logs_prefix : ""
autoscaling_enabled = var.autoscaling_enabled
multi_vpc_connectivity_enabled = var.multi_vpc_connectivity_enabled
multi_vpc_connectivity_iam_enabled = var.multi_vpc_connectivity_iam_enabled
properties = var.properties
allowed_security_group_ids = var.allowed_security_group_ids
depends_on = [
aws_cloudwatch_log_group.msk_cloudwatch_log_group,
aws_s3_bucket.this,
aws_acmpca_certificate.cert[0]
]
}
Loading

0 comments on commit c8193c7

Please sign in to comment.