Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cli): garbage collect s3 assets (under --unstable flag) #31611

Merged
merged 85 commits into from
Oct 21, 2024
Merged
Changes from 1 commit
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
5e635b9
basic api surface area
kaizencc Sep 23, 2024
785d820
more scaffolding with sdks
kaizencc Sep 23, 2024
b12362a
add cfn api calls
kaizencc Sep 24, 2024
796e8d2
tagging api in use now
kaizencc Sep 26, 2024
52b6c10
new garbo collector
kaizencc Sep 27, 2024
71062e0
parallel delete
kaizencc Sep 27, 2024
6fbcce4
integ test
kaizencc Sep 30, 2024
574e8ef
integ tests mostly work
kaizencc Sep 30, 2024
8bf315d
working integ test
kaizencc Oct 1, 2024
689b283
documentation
kaizencc Oct 1, 2024
96fbb5a
delete parallel
kaizencc Oct 1, 2024
64fe5db
more docs
kaizencc Oct 1, 2024
6f2381d
tagging works now
kaizencc Oct 1, 2024
890daaf
delete logs and add more docs
kaizencc Oct 1, 2024
e85c622
unstable
kaizencc Oct 1, 2024
e3a708e
linter
kaizencc Oct 1, 2024
835e399
Merge branch 'main' into conroy/gc
kaizencc Oct 1, 2024
4a5fd50
yarnlock
kaizencc Oct 2, 2024
3141414
change name to rollbackBufferDays
kaizencc Oct 2, 2024
1ace9ea
Merge branch 'main' into conroy/gc
kaizencc Oct 2, 2024
a6412e9
Merge branch 'conroy/gc' of https://github.com/aws/aws-cdk into conro…
kaizencc Oct 2, 2024
30005d5
buncha renames
kaizencc Oct 2, 2024
778fff0
consolidate api options
kaizencc Oct 2, 2024
e4c5ebb
faciliate actions better
kaizencc Oct 2, 2024
73d8eda
fix refresh stacks
kaizencc Oct 2, 2024
8e936a0
parallel get tags
kaizencc Oct 3, 2024
232f186
remove parallelism
kaizencc Oct 3, 2024
4e6bf9a
Merge branch 'main' into conroy/gc
kaizencc Oct 3, 2024
bcf561f
Merge branch 'conroy/gc' of https://github.com/aws/aws-cdk into conro…
kaizencc Oct 3, 2024
7ff4273
small change
kaizencc Oct 3, 2024
9f7341e
readme
kaizencc Oct 3, 2024
eb204b3
stupid stupid stupid
kaizencc Oct 3, 2024
2250768
review in progress means waiting
kaizencc Oct 7, 2024
f942870
unit tests
kaizencc Oct 7, 2024
18376d8
more unit tests
kaizencc Oct 7, 2024
80440df
robust qualifier
kaizencc Oct 7, 2024
033fed4
update integs
kaizencc Oct 7, 2024
f65f09c
more unit tests
kaizencc Oct 8, 2024
2731756
Merge branch 'main' into conroy/gc
kaizencc Oct 8, 2024
0b58840
finish merge
kaizencc Oct 8, 2024
607fa33
remove dup dep
kaizencc Oct 8, 2024
e898d3f
argh linter
kaizencc Oct 8, 2024
e69fcfc
Merge branch 'main' into conroy/gc
kaizencc Oct 8, 2024
44edfc6
Merge branch 'main' into conroy/gc
kaizencc Oct 8, 2024
2e1b780
minor updates
kaizencc Oct 13, 2024
fd08da5
refactor how stack refresh is done
kaizencc Oct 13, 2024
796248e
add debugs
kaizencc Oct 14, 2024
6a406c3
progress printer
kaizencc Oct 14, 2024
551aa86
lint
kaizencc Oct 14, 2024
411df38
various pr comments
kaizencc Oct 14, 2024
a46aa35
add two missing unit tests
kaizencc Oct 14, 2024
b3c6b99
ignore assets that are changed after gc starts
kaizencc Oct 14, 2024
0f233c5
linters
kaizencc Oct 14, 2024
f76f494
untag
kaizencc Oct 14, 2024
d861414
add integ test
kaizencc Oct 14, 2024
d7ea819
Merge branch 'main' into conroy/gc
kaizencc Oct 14, 2024
d1ac1b1
make some calls synchronous to simplify code
kaizencc Oct 15, 2024
4bf31b6
pr comments
kaizencc Oct 15, 2024
4610ac4
unit tests for large # of objects
kaizencc Oct 15, 2024
8c1c376
Merge branch 'conroy/gc' of https://github.com/aws/aws-cdk into conro…
kaizencc Oct 15, 2024
770ab8d
stupid stupid linter
kaizencc Oct 15, 2024
1728677
Merge branch 'main' into conroy/gc
kaizencc Oct 16, 2024
ffc0e18
pr comment
kaizencc Oct 16, 2024
b9f92b5
unit tests for background refresh
kaizencc Oct 16, 2024
d9d9b41
add prompt before deletion
kaizencc Oct 17, 2024
5d7942f
createdAtBufferDays
kaizencc Oct 17, 2024
4ce3a1a
lint
kaizencc Oct 17, 2024
e4c290c
readme update
kaizencc Oct 17, 2024
22a5365
Merge branch 'main' into conroy/gc
kaizencc Oct 17, 2024
ba28cf2
typos
kaizencc Oct 17, 2024
099c327
pause
kaizencc Oct 17, 2024
b109823
integ test fix
kaizencc Oct 17, 2024
9dc66a2
Merge branch 'main' into conroy/gc
kaizencc Oct 17, 2024
09f212b
Update packages/aws-cdk/lib/api/garbage-collection/stack-refresh.ts
kaizencc Oct 21, 2024
36c83ae
Merge branch 'main' into conroy/gc
kaizencc Oct 21, 2024
533797a
Update packages/aws-cdk/lib/cli.ts
kaizencc Oct 21, 2024
6056bf6
Update packages/aws-cdk/lib/cli.ts
kaizencc Oct 21, 2024
3736e37
start is not async
kaizencc Oct 21, 2024
9deae15
rename cli options
kaizencc Oct 21, 2024
4fec4a6
global unstable
kaizencc Oct 21, 2024
ada0384
Apply suggestions from code review
kaizencc Oct 21, 2024
59edf16
remove console statement
kaizencc Oct 21, 2024
4a75ec4
Merge branch 'conroy/gc' of https://github.com/aws/aws-cdk into conro…
kaizencc Oct 21, 2024
60a8ec5
Merge branch 'main' into conroy/gc
kaizencc Oct 21, 2024
0c851ba
Merge branch 'main' into conroy/gc
mergify[bot] Oct 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
integ test
  • Loading branch information
kaizencc committed Sep 30, 2024
commit 6fbcce4a318217169c48594df70a1dd5b3be052e
25 changes: 25 additions & 0 deletions packages/@aws-cdk-testing/cli-integ/lib/with-cdk-app.ts
Original file line number Diff line number Diff line change
@@ -311,6 +311,18 @@ export interface CdkModernBootstrapCommandOptions extends CommonCdkBootstrapComm
readonly usePreviousParameters?: boolean;
}

export interface CdkGarbageCollectionCommandOptions {
/**
* @default 0
*/
readonly days?: number;

/**
* @default 'all'
*/
readonly type?: 'ecr' | 's3' | 'all';
}

export class TestFixture extends ShellHelper {
public readonly qualifier = this.randomString.slice(0, 10);
private readonly bucketsToDelete = new Array<string>();
@@ -444,6 +456,18 @@ export class TestFixture extends ShellHelper {
});
}

public async cdkGarbageCollect(options: CdkGarbageCollectionCommandOptions): Promise<string> {
const args = ['gc'];
if (options.days) {
args.push('--days', String(options.days));
}
if (options.type) {
args.push('--type', options.type);
}

return this.cdk(args);
}

public async cdkMigrate(language: string, stackName: string, inputPath?: string, options?: CdkCliOptions) {
return this.cdk([
'migrate',
@@ -610,6 +634,7 @@ async function ensureBootstrapped(fixture: TestFixture) {
CDK_NEW_BOOTSTRAP: '1',
},
});

ALREADY_BOOTSTRAPPED_IN_THIS_RUN.add(envSpecifier);
}

Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
import { ListObjectsV2Command } from '@aws-sdk/client-s3';
import { integTest, randomString, withoutBootstrap } from '../../lib';

jest.setTimeout(2 * 60 * 60_000); // Includes the time to acquire locks, worst-case single-threaded runtime

// this is to ensure that asset bundling for apps under a stage does not break
integTest(
'Basic Garbage Collection',
withoutBootstrap(async (fixture) => {
const toolkitStackName = fixture.bootstrapStackName;
const bootstrapBucketName = `aws-cdk-garbage-collect-integ-test-bckt-${randomString()}`;
fixture.rememberToDeleteBucket(bootstrapBucketName); // just in case

await fixture.cdkBootstrapModern({
toolkitStackName,
bootstrapBucketName,
});

await fixture.cdkDeploy('lambda');
fixture.log('Setup complete!');
await fixture.cdkDestroy('lambda');
fixture.log('Teardown complete!');
await fixture.cdkGarbageCollect({
days: 0,
type: 's3',
});
fixture.log('Garbage collection complete!');

// assert that the bootstrap bucket is empty
await fixture.aws.s3.send(new ListObjectsV2Command({ Bucket: bootstrapBucketName }))
.then((result) => {
expect(result.Contents).toHaveLength(0);
});
}),
);