Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix for Issue #196. Changed S3 protocol to use s3a:// for storing transformed output data #197

Merged
merged 155 commits into from
May 28, 2021
Merged
Changes from 1 commit
Commits
Show all changes
155 commits
Select commit Hold shift + click to select a range
18b45fc
initial commit for construct
knihit Nov 7, 2020
15930ce
initial commit for construct
knihit Nov 7, 2020
309cce6
adding a new construct
knihit Nov 13, 2020
e024476
adding a new construct
knihit Nov 13, 2020
73062c8
Merge remote-tracking branch 'upstream/master'
knihit Nov 19, 2020
0263463
updates to construct default glue attributes
knihit Nov 30, 2020
5f77445
updating package.json
knihit Dec 1, 2020
ec1b1a6
updating ts files
knihit Dec 2, 2020
726ea01
adding unit tests and integ tests
knihit Dec 3, 2020
37b6ffd
adding unit tests and integ tests
knihit Dec 3, 2020
e931fdd
adding unit tests and helper methods to complete the construct
knihit Dec 4, 2020
8cb3e26
adding unit tests
knihit Dec 5, 2020
da90d53
Merge remote-tracking branch 'upstream/master'
knihit Dec 5, 2020
0f4d7e2
adding unit tests
knihit Dec 5, 2020
a8f03e2
adding unit tests
knihit Dec 5, 2020
395af14
fix for linting error
knihit Dec 9, 2020
97e8797
adding glue job example
knihit Dec 10, 2020
a7e3529
update readme.md
knihit Dec 10, 2020
2e53b39
update snapshots and kms arn
knihit Dec 10, 2020
47a0332
update description in package.json
knihit Dec 10, 2020
b10c28d
integ test cases
knihit Dec 10, 2020
fc9d3c3
added simulator for kinesis stream
knihit Dec 11, 2020
50af095
added simulator for kinesis stream
knihit Dec 11, 2020
67b5ad1
integ test cases
knihit Dec 11, 2020
27c0401
updating construct for scriptlocation
knihit Dec 11, 2020
b21836c
updating python generator file
knihit Dec 11, 2020
7a5709a
updating example code
knihit Dec 12, 2020
7050d22
updating example code
knihit Dec 12, 2020
f57b68f
updating example code
knihit Dec 12, 2020
3f05829
updating example code
knihit Dec 12, 2020
47676ef
updating example code
knihit Dec 12, 2020
5dcf45e
updating example code
knihit Dec 12, 2020
94f7175
updating example code
knihit Dec 12, 2020
b1b8640
update README and Architecture
knihit Dec 12, 2020
979c477
updating README
knihit Dec 12, 2020
854b0c6
updating README
knihit Dec 12, 2020
b07d151
updating construct
knihit Dec 14, 2020
a300ebe
updating construct
knihit Dec 14, 2020
f6c464e
updating construct
knihit Dec 14, 2020
9e9d4e1
updating construct
knihit Dec 15, 2020
76d1e41
updating construct
knihit Dec 15, 2020
3793c73
updating construct
knihit Dec 15, 2020
341a6bf
updating construct
knihit Dec 15, 2020
743d971
updating construct
knihit Dec 15, 2020
bab75a4
updating construct
knihit Dec 15, 2020
4cb8edc
updating sample
knihit Dec 15, 2020
56e1cca
updating construct
knihit Dec 16, 2020
4aad13b
updating construct
knihit Dec 16, 2020
2b4bea3
updating construct
knihit Dec 16, 2020
b8915a3
updating construct
knihit Dec 16, 2020
f735cac
updating sample
knihit Dec 16, 2020
6253179
merging changes
knihit Dec 16, 2020
9322de0
updating sample
knihit Dec 16, 2020
993bde4
updating construct kinesis policy
knihit Dec 16, 2020
b664909
updating construct kinesis policy
knihit Dec 17, 2020
4730152
updating construct kinesis policy
knihit Dec 17, 2020
58773c5
updating construct kinesis policy
knihit Dec 17, 2020
ef8436e
updating construct kinesis policy
knihit Dec 17, 2020
0a02004
updating construct kinesis policy
knihit Dec 17, 2020
872650f
updating construct kinesis policy
knihit Dec 17, 2020
4bc12b6
updating construct
knihit Dec 17, 2020
80fd800
updating construct kinesis policy
knihit Dec 17, 2020
a6f4d4e
updating construct kinesis policy
knihit Dec 17, 2020
9e7b780
updating construct kinesis policy
knihit Dec 17, 2020
4df7b77
updating construct kinesis policy
knihit Dec 17, 2020
0d71ad4
updating construct policy
knihit Dec 17, 2020
58d09b0
updating readme
knihit Dec 17, 2020
5e5f142
Merge remote-tracking branch 'upstream/master'
knihit Dec 29, 2020
a8c70e4
Merge remote-tracking branch 'upstream/master'
knihit Jan 1, 2021
9a4d955
updated README
knihit Jan 5, 2021
3f09960
updated README
knihit Jan 5, 2021
536dd03
updated README
knihit Jan 5, 2021
49502ab
updated README
knihit Jan 5, 2021
32ff18a
updated README
knihit Jan 5, 2021
190e9ce
updated README
knihit Jan 5, 2021
83d5486
updated README
knihit Jan 5, 2021
8325790
updated README
knihit Jan 5, 2021
e668827
updated README
knihit Jan 5, 2021
d57f3b1
updates based on review
knihit Jan 6, 2021
7a8c0df
incoporating review comments
knihit Jan 6, 2021
8a7cec9
incoporating review comments
knihit Jan 6, 2021
22bc12d
incoporating review comments
knihit Jan 6, 2021
aa0bccd
incoporating review comments
knihit Jan 6, 2021
5a96af4
incoporating review comments
knihit Jan 6, 2021
b04003d
update to construct
knihit Jan 7, 2021
b867dd7
update to construct
knihit Jan 8, 2021
d2b92ca
update to construct
knihit Jan 8, 2021
6a7f40a
update to construct
knihit Jan 8, 2021
a067304
update to construct
knihit Jan 8, 2021
97979c5
update to construct
knihit Jan 8, 2021
34eeae8
refactoring code to match construct patterns
knihit Jan 15, 2021
438d31e
refactoring code to match construct patterns
knihit Jan 15, 2021
8e89453
refactoring code to match construct patterns
knihit Jan 15, 2021
90c5cb5
refactoring code to match construct patterns
knihit Jan 15, 2021
fc15d89
refactoring code to match construct patterns
knihit Jan 15, 2021
b8691a5
refactoring code to match construct patterns
knihit Jan 15, 2021
d1c44cd
refactoring code to match construct patterns
knihit Jan 15, 2021
7d3ffe6
fix for readme file
knihit Jan 19, 2021
62068e6
Merge remote-tracking branch 'upstream/master'
knihit Jan 19, 2021
7f3f148
eslint fixes
knihit Jan 19, 2021
fde06ae
update sample after refactoring construct
knihit Jan 19, 2021
dc83095
update sample after refactoring construct
knihit Jan 20, 2021
6b0ef07
remove _ from variable names
knihit Jan 20, 2021
7ad1ca6
updating glue version 2.0 as recommended in Glue service documentation
knihit Jan 20, 2021
14ffdb0
removed snapshot (that was not required) to fix build failure
knihit Jan 21, 2021
1d84029
update viperlight to fix build issue
knihit Jan 21, 2021
5783f5e
update viperlight to fix build issue
knihit Jan 21, 2021
f40ef76
cfn_nag fix for build issues
knihit Jan 21, 2021
e64e742
merge 1.82.0
knihit Jan 21, 2021
8100d19
updating integ snapshots
knihit Jan 21, 2021
0fc781f
updating integ snapshots
knihit Jan 21, 2021
c3ab317
Merge branch 'master' into master
knihit Jan 21, 2021
6a7032e
Merge remote-tracking branch 'upstream/master'
knihit Jan 21, 2021
afb173f
Merge branch 'master' of github.com:knihit/aws-solutions-constructs
knihit Jan 21, 2021
8ff7c8c
update header
knihit Jan 21, 2021
342f3de
update header
knihit Jan 21, 2021
7b2264b
incorporate publisher review comments
knihit Feb 3, 2021
1830784
Merge remote-tracking branch 'upstream/master'
knihit Feb 3, 2021
5a82e15
incorporate publisher review comments
knihit Feb 3, 2021
844620b
incorporate publisher review comments
knihit Feb 3, 2021
0e2c9c3
reorganization unitt tests
knihit Feb 3, 2021
7211d7d
reorganization unitt tests
knihit Feb 4, 2021
24779df
reorganization unitt tests
knihit Feb 4, 2021
3291b5e
reorganization unitt tests
knihit Feb 4, 2021
6928f6f
reorganization unitt tests
knihit Feb 4, 2021
75a869f
reorganization unitt tests
knihit Feb 4, 2021
cfd9836
incorporate review comments
knihit Feb 5, 2021
8c12c6c
fix for output s3 bucket
knihit Feb 5, 2021
285dbfa
fix for output s3 bucket
knihit Feb 5, 2021
7a8a8d1
fix for output s3 bucket
knihit Feb 5, 2021
a489345
update README with the correct class name
knihit Feb 5, 2021
fd6c92d
Merge remote-tracking branch 'upstream/master'
knihit Feb 8, 2021
649e629
updating README
knihit Feb 8, 2021
711291c
updates to readme
knihit Feb 16, 2021
8627fa4
updates to readme
knihit Feb 16, 2021
3dfa8bc
updates to readme
knihit Feb 16, 2021
15f1945
updates to readme
knihit Feb 16, 2021
ba8968a
updates to readme
knihit Feb 16, 2021
70a45b4
Merge remote-tracking branch 'upstream/master'
knihit Feb 23, 2021
da4be4a
Merge branch 'master' into master
knihit Feb 25, 2021
5914bb8
Merge remote-tracking branch 'upstream/master'
knihit Feb 25, 2021
88bce47
Merge branch 'master' of github.com:knihit/aws-solutions-constructs
knihit Feb 25, 2021
e0f22eb
removing duplicate words from README
knihit Feb 25, 2021
1240701
Merge remote-tracking branch 'upstream/master'
knihit Mar 5, 2021
50a776e
Merge branch 'awslabs:main' into master
knihit May 12, 2021
b39b51a
Merge branch 'awslabs:main' into master
knihit May 19, 2021
84b8547
fix for bug #184
knihit May 19, 2021
e992baf
fix for bug #184
knihit May 19, 2021
5a78316
Merge branch 'main' into master
knihit May 19, 2021
a4af4cd
fix for bug #184
knihit May 19, 2021
1311500
Merge branch 'master' of github.com:knihit/aws-solutions-constructs
knihit May 19, 2021
0f2e248
Merge branch 'awslabs:main' into master
knihit May 26, 2021
aa93f9b
fix for bug #196
knihit May 26, 2021
95dd49b
fix for bug #199
knihit May 27, 2021
5baa256
fix for bug #199
knihit May 27, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
updated README
  • Loading branch information
knihit committed Jan 5, 2021
commit 3f09960962225ea1238c859871d763c132ac7eb6
Original file line number Diff line number Diff line change
Expand Up @@ -88,12 +88,28 @@ _Parameters_

## Pattern Construct Props

| **Name** | **Type** | **Description** |
| :------------------ | :------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| kinesisStreamProps? | [`kinesis.StreamProps`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-kinesis.StreamProps.html) | Optional user-provided props to override the default props for the Kinesis stream. |
| existingStreamObj? | [`kinesis.Stream`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-kinesis.Stream.html) | Existing instance of Kinesis Stream, if this is set then kinesisStreamProps is ignored. |
| glueJobProps? | [`cfnJob.CfnJobProps`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-glue.CfnJobProps.html) | User provided props to override the default props for the CfnJob. |
| existingGlueJob? | [`cfnJob.CfnJob`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-glue.CfnJob.html) | Existing CfnJob configuration, if this this set then glueJobProps is ignored. |
| **Name** | **Type** | **Description** |
| :------------------- | :-------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| kinesisStreamProps? | [`kinesis.StreamProps`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-kinesis.StreamProps.html) | Optional user-provided props to override the default props for the Kinesis stream. |
| existingStreamObj? | [`kinesis.Stream`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-kinesis.Stream.html) | Existing instance of Kinesis Stream, if this is set then kinesisStreamProps is ignored. |
| glueJobProps? | [`cfnJob.CfnJobProps`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-glue.CfnJobProps.html) | User provided props to override the default props for the CfnJob. |
| existingGlueJob? | [`cfnJob.CfnJob`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-glue.CfnJob.html) | Existing CfnJob configuration, if this this set then glueJobProps is ignored. |
| existingGlueJob? | [`cfnJob.CfnJob`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-glue.CfnJob.html) | Existing CfnJob configuration, if this this set then glueJobProps is ignored. |
| glueJobCommandProps? | ['GlueJobCommandProps'](#gluejobcommandprops) | @default - using this props the construct will create a default CfnGlueJobProps. This property will be ignored |
| fieldSchema? | ['CfnTable.ColumnProperty'](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-glue.CfnTable.ColumnProperty.html) | Glue Database for this construct. If not provided the construct will create a new Glue Database. The database is where the schema for the data in Kinesis Data Streams is stored |
| database? | ['CfnDatabase'](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-glue.CfnDatabase.html) | Glue Database for this construct. If not provided the construct will create a new Glue Database. The database is where the schema for the data in Kinesis Data Streams is stored |
| table? | ['CfnTable](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-glue.CfnTable.html) | Glue Table for this construct, If not provided the construct will create a new Table in the database. This table should define the schema for the records in the Kinesis Data Streams. Either @table or @fieldSchema is mandatory. If @table is provided then @fieldSchema is ignored |
| jobArgumentsList | {} | List of arguments that can be passed to a CfnJob. This arguments will be augmented with the default |
| glueVersion | string(https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-glue.CfnJob.html) | Glue version as supported by the AWS Glue service. The value defaults to 1.0 in the construct, since glue streaming is not supported by version 2.0 and 1.0 is the only version that supports python 3.0. This property will only be used if @glueJobCommandProps is set. This property will be ingored if @glueJobProps or @existingGlueJob is set |

## GlueJobCommandProps

| **Name** | **Type** | **Description** |
| :-------------------- | :------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| jobCommandName | string | The command to execute the Glue Job, The name of the job command. For an Apache Spark ETL job, this must be glueetl. For a Python shell job, it must be pythonshell. For streaming job it must be gluestreaming |
| pythonVersion | string | The Python version being used to execute a Python shell job. Allowed values are 2 or 3. |
| scriptPath? | string | The location of the ETL script in your locat directory. If the @s3ObjectUrlForScript parameter is provided, this parmaeter is ignored |
| s3ObjectUrlForScript? | string | The S3 URL for the ETL script. If this parameter is provided, the @scriptPath parameter is ignored. |

# Default settings

Expand Down