feat(pipes-alpha): support for customer-managed KMS keys to encrypt pipe data #33546

badmintoncryer · 2025-02-21T14:28:37Z

Issue # (if applicable)

Closes #31453

Reason for this change

AWS Pipes supports for encrypting data by customer managed KMS key instead of Amazon managed key.

https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-encryption-pipes-cmkey.html

The L2 Pipe construct does not support this feature now.

Description of changes

Add kmsKey prop to PipeProps

Describe any new or updated permissions being added

Add KMS key policy which enables pipes to access to the key.

https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-encryption-key-policy.html#eb-encryption-key-policy-pipe

Description of how you validated changes

Add both unit and integ tests.

Checklist

My code adheres to the CONTRIBUTING GUIDE and DESIGN GUIDELINES

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

badmintoncryer · 2025-02-21T14:29:58Z

packages/@aws-cdk/aws-pipes-alpha/README.md

@@ -23,7 +23,7 @@ can be filtered, transformed and enriched.

 ![diagram of pipes](https://d1.awsstatic.com/product-marketing/EventBridge/Product-Page-Diagram_Amazon-EventBridge-Pipes.cd7961854be4432d63f6158ffd18271d6c9fa3ec.png)

-For more details see the [service documentation](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes.html). 


Removed several unnecessary spaces.

aws-cdk-automation

(This review is outdated)

codecov · 2025-02-21T14:37:53Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.21%. Comparing base (7ebb92c) to head (8ce477f).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #33546   +/-   ##
=======================================
  Coverage   82.21%   82.21%           
=======================================
  Files         119      119           
  Lines        6876     6876           
  Branches     1162     1162           
=======================================
  Hits         5653     5653           
  Misses       1120     1120           
  Partials      103      103

Flag	Coverage Δ
suite.unit	`82.21% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
packages/aws-cdk	`∅ <ø> (∅)`
packages/aws-cdk-lib/core	`82.21% <ø> (ø)`

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

mazyu36

Thanks for your contribution! I think it's mostly fine.
I've added one question.

Also, I previously opened an issue here: #31453.
Would it be possible to link this contribution to that issue?

mazyu36 · 2025-02-22T06:34:38Z

packages/@aws-cdk/aws-pipes-alpha/lib/pipe.ts

@@ -295,6 +303,7 @@ export class Pipe extends PipeBase {
      targetParameters: target.targetParameters,
      desiredState: props.desiredState,
      logConfiguration: logConfiguration,
+      kmsKeyIdentifier: props.kmsKey?.keyArn,


Just a question—did it work correctly even without setting a key policy?

When I previously attempted to contribute to this, I encountered an error without a key policy, but I couldn't come up with a proper way to configure it and had to give up at the time.

It might have been my misunderstanding back then, or something might have changed with an update. If it worked fine, please just ignore this question.

Ref: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-encryption-key-policy.html#eb-encryption-key-policy-pipe

You are absolutely right.. I misunderstood that this implementation would be work fine. Thank you for your great suggestion.

How about adding the restriction that makes pipeName mandatory when kmsKey is provided?
This idea is not ideal, but it would provide the minimum feature that enables us to configure CMK for Pipes.

Thank you for confirming. So that's how it is...
I think your suggested solution is good.

Personally, I've been considering the following three options:

Make name mandatory when kmsKey is specified. This is the same as the suggestion you provided.

Instead of setting name to undefined when it's not specified, automatically generate it using Names.uniquid. However, I understand this would be a breaking change.

Configure kmsKey using a custom resource. I think this option is not ideal.

Based on the above, I believe option 1 or option 2 would be good, and I think option 1 would be better to avoid a breaking change.

I hadn't thought of option 2. I think implementing option 2 with a feature flag is also a good idea.
For now, I'll proceed with option 1. Of course, I'll also add integration testing to verify the functionality.

I've just noticed that the feature flag is not essential because this is alpha module..
I think the option 2 might be preferable for alpha module. I'll leave the decision to the maintainer.

I also think Options2 is smarter, but since the impact is significant, I'd like to wait for the maintainer's opinion.

Thanks for the deep dive. I think Option 1 makes sense to me. Can you help me understand Option 2 + feature flag solution?
In my understanding, we can't create the policy without knowing the pipe name. If we add a feature flag, for new stacks, we will always create a default pipe name if not specified which would solve the issue. However, for existing stack (who did not enable this feature flag), we will still have the same issue right?

@GavinZZ

Thank you for the clarification.

However, for existing stack (who did not enable this feature flag), we will still have the same issue right?

You're absolutely right! When setting a CMK for stacks under an App where the function flag is not enabled, the pipeName cannot be obtained, resulting in the inability to set the correct key policy.

Thanks for the quick response. Given the above discussion, I'll remove the do-not-merge label and let's go with the solution you had here. Thank you!

badmintoncryer · 2025-02-26T13:58:05Z

@mazyu36 Thank you for your nice review!! I've addressed your comments.

mazyu36

Thanks! LGTM

GavinZZ

Great, thank you!

GavinZZ · 2025-02-26T22:28:39Z

packages/@aws-cdk/aws-pipes-alpha/lib/pipe.ts

@@ -295,6 +303,7 @@ export class Pipe extends PipeBase {
      targetParameters: target.targetParameters,
      desiredState: props.desiredState,
      logConfiguration: logConfiguration,
+      kmsKeyIdentifier: props.kmsKey?.keyArn,


Thanks for the deep dive. I think Option 1 makes sense to me. Can you help me understand Option 2 + feature flag solution?
In my understanding, we can't create the policy without knowing the pipe name. If we add a feature flag, for new stacks, we will always create a default pipe name if not specified which would solve the issue. However, for existing stack (who did not enable this feature flag), we will still have the same issue right?

GavinZZ · 2025-02-26T22:29:57Z

Adding a do-not-merge flag for this thread #33546 (comment)

mergify · 2025-02-27T00:22:49Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

aws-cdk-automation · 2025-02-27T00:53:07Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
Commit ID: 8ce477f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2025-02-27T00:53:26Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

github-actions · 2025-02-27T00:53:44Z

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

badmintoncryer added 3 commits February 21, 2025 22:51

cmk support

66a25cc

add integ

f417ef8

update readme

ba2c374

badmintoncryer changed the title ~~feat(aws-pipes-alpha): support customer managed key to encrypt pipe data~~ feat(aws-pipes-alpha): support customer managed kms key to encrypt pipe data Feb 21, 2025

aws-cdk-automation requested a review from a team February 21, 2025 14:28

github-actions bot added p2 distinguished-contributor [Pilot] contributed 50+ PRs to the CDK labels Feb 21, 2025

badmintoncryer commented Feb 21, 2025

View reviewed changes

aws-cdk-automation previously requested changes Feb 21, 2025

View reviewed changes

badmintoncryer changed the title ~~feat(aws-pipes-alpha): support customer managed kms key to encrypt pipe data~~ feat(pipes-alpha): support customer managed kms key to encrypt pipe data Feb 21, 2025

fix import order

eca5cbe

badmintoncryer changed the title ~~feat(pipes-alpha): support customer managed kms key to encrypt pipe data~~ feat(pipes-alpha): support for customer-managed KMS keys to encrypt pipe data Feb 22, 2025

badmintoncryer and others added 2 commits February 22, 2025 09:32

Merge branch 'main' into pipes-cmk

90bdaa0

Merge branch 'main' into pipes-cmk

9489730

aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Feb 22, 2025

mazyu36 reviewed Feb 22, 2025

View reviewed changes

github-actions bot added 2 commits February 26, 2025 21:57

update integ test

181b4ab

fix unit test

c053604

github-actions bot added effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. labels Feb 26, 2025

badmintoncryer added 3 commits February 26, 2025 22:21

Merge branch 'main' into pipes-cmk

7da16d5

fix lint

140ddfd

update readme

0a4c434

mazyu36 approved these changes Feb 26, 2025

View reviewed changes

aws-cdk-automation added pr/needs-maintainer-review This PR needs a review from a Core Team Member and removed pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. labels Feb 26, 2025

GavinZZ approved these changes Feb 26, 2025

View reviewed changes

GavinZZ added the pr/do-not-merge This PR should not be merged at this time. label Feb 26, 2025

GavinZZ self-assigned this Feb 26, 2025

aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Feb 26, 2025

GavinZZ removed the pr/do-not-merge This PR should not be merged at this time. label Feb 27, 2025

Merge branch 'main' into pipes-cmk

8ce477f

mergify bot merged commit dd0d62f into aws:main Feb 27, 2025
20 checks passed

github-actions bot locked as resolved and limited conversation to collaborators Feb 27, 2025

badmintoncryer deleted the pipes-cmk branch February 27, 2025 00:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(pipes-alpha): support for customer-managed KMS keys to encrypt pipe data #33546

feat(pipes-alpha): support for customer-managed KMS keys to encrypt pipe data #33546

badmintoncryer commented Feb 21, 2025 •

edited

Loading

badmintoncryer Feb 21, 2025

aws-cdk-automation left a comment •

edited

Loading

codecov bot commented Feb 21, 2025 •

edited

Loading

mazyu36 left a comment

mazyu36 Feb 22, 2025

badmintoncryer Feb 25, 2025 •

edited

Loading

mazyu36 Feb 25, 2025 •

edited

Loading

badmintoncryer Feb 26, 2025 •

edited

Loading

badmintoncryer Feb 26, 2025 •

edited

Loading

mazyu36 Feb 26, 2025

GavinZZ Feb 26, 2025

badmintoncryer Feb 26, 2025

GavinZZ Feb 27, 2025

badmintoncryer commented Feb 26, 2025

mazyu36 left a comment

GavinZZ left a comment

GavinZZ Feb 26, 2025

GavinZZ commented Feb 26, 2025

mergify bot commented Feb 27, 2025

aws-cdk-automation commented Feb 27, 2025

mergify bot commented Feb 27, 2025

github-actions bot commented Feb 27, 2025

		@@ -23,7 +23,7 @@ can be filtered, transformed and enriched.

		![diagram of pipes](https://d1.awsstatic.com/product-marketing/EventBridge/Product-Page-Diagram_Amazon-EventBridge-Pipes.cd7961854be4432d63f6158ffd18271d6c9fa3ec.png)

		For more details see the [service documentation](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes.html).

feat(pipes-alpha): support for customer-managed KMS keys to encrypt pipe data #33546

feat(pipes-alpha): support for customer-managed KMS keys to encrypt pipe data #33546

Conversation

badmintoncryer commented Feb 21, 2025 • edited Loading

Issue # (if applicable)

Reason for this change

Description of changes

Describe any new or updated permissions being added

Description of how you validated changes

Checklist

badmintoncryer Feb 21, 2025

Choose a reason for hiding this comment

aws-cdk-automation left a comment • edited Loading

Choose a reason for hiding this comment

codecov bot commented Feb 21, 2025 • edited Loading

Codecov Report

mazyu36 left a comment

Choose a reason for hiding this comment

mazyu36 Feb 22, 2025

Choose a reason for hiding this comment

badmintoncryer Feb 25, 2025 • edited Loading

Choose a reason for hiding this comment

mazyu36 Feb 25, 2025 • edited Loading

Choose a reason for hiding this comment

badmintoncryer Feb 26, 2025 • edited Loading

Choose a reason for hiding this comment

badmintoncryer Feb 26, 2025 • edited Loading

Choose a reason for hiding this comment

mazyu36 Feb 26, 2025

Choose a reason for hiding this comment

GavinZZ Feb 26, 2025

Choose a reason for hiding this comment

badmintoncryer Feb 26, 2025

Choose a reason for hiding this comment

GavinZZ Feb 27, 2025

Choose a reason for hiding this comment

badmintoncryer commented Feb 26, 2025

mazyu36 left a comment

Choose a reason for hiding this comment

GavinZZ left a comment

Choose a reason for hiding this comment

GavinZZ Feb 26, 2025

Choose a reason for hiding this comment

GavinZZ commented Feb 26, 2025

mergify bot commented Feb 27, 2025

aws-cdk-automation commented Feb 27, 2025

AWS CodeBuild CI Report

mergify bot commented Feb 27, 2025

github-actions bot commented Feb 27, 2025

badmintoncryer commented Feb 21, 2025 •

edited

Loading

aws-cdk-automation left a comment •

edited

Loading

codecov bot commented Feb 21, 2025 •

edited

Loading

badmintoncryer Feb 25, 2025 •

edited

Loading

mazyu36 Feb 25, 2025 •

edited

Loading

badmintoncryer Feb 26, 2025 •

edited

Loading

badmintoncryer Feb 26, 2025 •

edited

Loading