Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(glue): support partition index on tables #17998

Merged
merged 11 commits into from
Dec 29, 2021
Next Next commit
first attempt at custom resource for partition index
  • Loading branch information
kaizencc committed Dec 10, 2021
commit 374fc23d51d4402e5e0102d1f3f1fed035f2bcab
91 changes: 90 additions & 1 deletion packages/@aws-cdk/aws-glue/lib/table.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,31 @@
import * as iam from '@aws-cdk/aws-iam';
import * as kms from '@aws-cdk/aws-kms';
import * as s3 from '@aws-cdk/aws-s3';
import { ArnFormat, Fn, IResource, Resource, Stack } from '@aws-cdk/core';
import { ArnFormat, Fn, IResource, Names, Resource, Stack } from '@aws-cdk/core';
import * as cr from '@aws-cdk/custom-resources';
import { Construct } from 'constructs';
import { DataFormat } from './data-format';
import { IDatabase } from './database';
import { CfnTable } from './glue.generated';
import { Column } from './schema';

/**
* Properties of a Partition Index.
*/
export interface PartitionIndexProps {
/**
* The name of the partition index.
*
* @default - a name will be generated for you.
*/
readonly indexName: string;

/**
* The partition index keys. These keys
* must be a subet of the table's partition keys.
*/
readonly keys: string[];
}
export interface ITable extends IResource {
/**
* @attribute
Expand Down Expand Up @@ -230,6 +248,8 @@ export class Table extends Resource implements ITable {
*/
public readonly partitionKeys?: Column[];

private partitionIndecies: number = 0;

constructor(scope: Construct, id: string, props: TableProps) {
super(scope, id, {
physicalName: props.tableName,
Expand Down Expand Up @@ -289,6 +309,59 @@ export class Table extends Resource implements ITable {
this.node.defaultChild = tableResource;
}

/**
* Add a partition index to the table.
* @see https://docs.aws.amazon.com/glue/latest/dg/partition-indexes.html
*
* Partition index keys must be a subset of the tables partition keys.
*/
public addPartitionIndex(props: PartitionIndexProps) {
if (this.partitionIndecies >= 3) {
throw new Error('Table can have a maximum of 3 partition indecies');
}
this.partitionIndecies++;
this.validatePartitionIndex(props);
const partitionIndex = new cr.AwsCustomResource(this, 'table-partition-index', {
onCreate: {
service: 'Glue',
action: 'createPartitionIndex',
parameters: {
DatabaseName: this.database.databaseName,
TableName: this.tableName,
PartitionIndex: {
IndexName: props.indexName ?? this.generateName(),
Keys: props.keys,
},
},
physicalResourceId: cr.PhysicalResourceId.of(
'CreatePartitionIndex',
),
},
policy: cr.AwsCustomResourcePolicy.fromSdkCalls({
resources: cr.AwsCustomResourcePolicy.ANY_RESOURCE,
}),
});

this.grantToUnderlyingResources(partitionIndex, ['glue:UpdateTable']);
}

private generateName(): string {
return Names.uniqueId(this);
}

private validatePartitionIndex(props: PartitionIndexProps) {
if (props.indexName && !props.indexName.match(/^[A-Za-z0-9\_\-])/)) {
throw new Error(`Index name can only have letters, numbers, hyphens, or underscores but received ${props.indexName}`);
}
if (!this.partitionKeys || this.partitionKeys.length === 0) {
throw new Error('To create a partition index the table must have partition keys');
}
const keyNames = this.partitionKeys.map(pk => pk.name);
if (!props.keys.every(k => keyNames.includes(k))) {
throw new Error(`All index keys must also be partition keys. Got ${props.keys} but partition key names are ${keyNames}`);
}
}

/**
* Grant read permissions to the table and the underlying data stored in S3 to an IAM principal.
*
Expand Down Expand Up @@ -336,6 +409,22 @@ export class Table extends Resource implements ITable {
});
}

/**
* Grant the given identity custom permissions to ALL underlying resources of the table.
* Permissions will be granted to the catalog, the database, and the table.
*/
public grantToUnderlyingResources(grantee: iam.IGrantable, actions: string[]) {
return iam.Grant.addToPrincipal({
grantee,
resourceArns: [
this.tableArn,
this.database.catalogArn,
this.database.databaseArn,
Comment on lines +459 to +461
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have a source for these permissions? What does glue:UpdateTable mean for a table vs catalog vs database?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is mostly from my own investigations (nothing documented that I could find).

Here's what I get with no permissions:

Received response status [FAILED] from custom resource. Message returned: User:
arn:aws:sts::489318732371:assumed-role/GluePartitionStack-AWS679f53fac002430cb0
da5b7982bd-1GZIV4NJQYJJ1/GluePartitionStack-AWS679f53fac002430cb0da5b7982bd-89Z
BzvCbg7Oi is not authorized to perform: glue:UpdateTable on resource: arn:aws:g
lue:us-east-1:489318732371:catalog (RequestId: ab6c2467-9c78-4d23-be59-953e4ab2
3144)

Here's what I get after I add glue:UpdateTable with permissions to the catalog:

Received response status [FAILED] from custom resource. Message returned: User:
arn:aws:sts::489318732371:assumed-role/GluePartitionStack-AWS679f53fac002430cb0
da5b7982bd-7U8A8EMIKP4E/GluePartitionStack-AWS679f53fac002430cb0da5b7982bd-nlw0
ulljEHZq is not authorized to perform: glue:UpdateTable on resource: arn:aws:gl
ue:us-east-1:489318732371:database/my_database (RequestId: d4caefa8-30d2-4b02-9
9e4-6b1305ab5aea)

So I then add the same permission to the database and I get:

Received response status [FAILED] from custom resource. Message returned: User:
arn:aws:sts::489318732371:assumed-role/GluePartitionStack-AWS679f53fac002430cb0
da5b7982bd-1N8TEEWC845G7/GluePartitionStack-AWS679f53fac002430cb0da5b7982bd-z2X
SZ4Zjykso is not authorized to perform: glue:UpdateTable on resource: arn:aws:g
lue:us-east-1:489318732371:table/my_database/json_table (RequestId: aa73a7ff-0f
f8-4c20-8338-b0f587298d6e)

The only valid permission is for the catalog, database, and table.

],
actions,
});
}

private getS3PrefixForGrant() {
return this.s3Prefix + '*';
}
Expand Down
1 change: 1 addition & 0 deletions packages/@aws-cdk/aws-glue/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@
"@aws-cdk/aws-s3": "0.0.0",
"@aws-cdk/aws-s3-assets": "0.0.0",
"@aws-cdk/core": "0.0.0",
"@aws-cdk/custom-resources": "0.0.0",
"constructs": "^3.3.69"
},
"homepage": "https://github.com/aws/aws-cdk",
Expand Down