eks-v2-alpha: can't delete fargate cluster

[pahud]

Describe the bug

can't destroy the eks fargate cluster

v32-stack | 4 | 6:36:12 PM | DELETE_FAILED | Custom::AWSCDK-EKS-KubernetesPatch | hello-eks/CoreDnsComputeTypePatch/Resource/Default (helloeksCoreDnsComputeTypePatch4E0013F7) Received response status [FAILED] from custom resource. Message returned: Error: b'E0207 23:36:05.317537 11 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"\nE0207 23:36:07.233292 11 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"\nE0207 23:36:09.114991 11 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"\nE0207 23:36:11.021248 11 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"\nerror: You must be logged in to the server (the server has asked for the client to provide credentials)\n'

Logs: /aws/lambda/v32-stack-helloeksKubectlProviderHandler788A3C17-CURPgTpq6yNx

at invokeUserFunction (/var/task/framework.js:129:19)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async onEvent (/var/task/framework.js:22:27)
at async Runtime.handler (/var/task/cfn-response.js:60:13) (RequestId: 036ed2d7-5122-4557-9785-99aa1f59680c)

4 Currently in progress: v32-stack, helloeksfargateprofiledefault2FADE994

LAMBDA_WARNING: Unhandled exception. The most likely cause is an issue in the function code. However, in rare cases, a Lambda runtime update can cause unexpected function behavior. For functions using managed runtimes, runtime updates can be triggered by a function change, or can be applied automatically. To determine if the runtime has been updated, check the runtime version in the INIT_START log entry. If this error correlates with a change in the runtime version, you may be able to mitigate this error by temporarily rolling back to the previous runtime version. For more information, see https://docs.aws.amazon.com/lambda/latest/dg/runtimes-update.html
[ERROR] Exception: b'E0207 23:36:05.317537 11 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"\nE0207 23:36:07.233292 11 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"\nE0207 23:36:09.114991 11 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"\nE0207 23:36:11.021248 11 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"\nerror: You must be logged in to the server (the server has asked for the client to provide credentials)\n'Traceback (most recent call last):  File "/var/task/index.py", line 20, in handler    return patch_handler(event, context)  File "/var/task/patch/init.py", line 48, in patch_handler    kubectl([ 'patch', resource_name, '-n', resource_namespace, '-p', patch_json, '--type', patch_type ])  File "/var/task/patch/init.py", line 64, in kubectl    raise Exception(output)
END RequestId: 20823de2-857b-4005-ab73-5eec0a91ffad
REPORT RequestId: 20823de2-857b-4005-ab73-5eec0a91ffad Duration: 12725.71 ms Billed Duration: 12726 ms Memory Size: 1024 MB Max Memory Used: 173 MB Init Duration: 241.81 ms
No newer events at this moment. Auto retry paused. Resume

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

can destroy the stack with the cluster

Current Behavior

can't destroy the stack

Reproduction Steps

cdk deploy & destroy

import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as iam from 'aws-cdk-lib/aws-iam';
import { App, Stack, StackProps } from 'aws-cdk-lib';
import { KubectlV32Layer } from '@aws-cdk/lambda-layer-kubectl-v32';
import * as eks from '../lib';
import { Construct } from 'constructs';

export class EksClusterLatestVersion extends Stack {
  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props);

    // const vpc = new ec2.Vpc(this, 'Vpc', { natGateways: 1 });
    const vpc = ec2.Vpc.fromLookup(this, 'Vpc', {
      isDefault: true,
    });
    const mastersRole = new iam.Role(this, 'Role', {
      assumedBy: new iam.AccountRootPrincipal(),
    });

    new eks.FargateCluster(this, 'hello-eks', {
      vpc,
      mastersRole,
      version: eks.KubernetesVersion.V1_32,
      kubectlProviderOptions: {
        kubectlLayer: new KubectlV32Layer(this, 'kubectl'),
      },
    });
  }
}

const app = new App();

new EksClusterLatestVersion(app, 'v32-stack', {
  env: {
    account: process.env.CDK_DEFAULT_ACCOUNT,
    region: process.env.CDK_DEFAULT_REGION,
  },
});

app.synth();

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.177.0

Framework Version

No response

Node.js Version

all

OS

linux

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

Looking at https://docs.aws.amazon.com/eks/latest/userguide/fargate-getting-started.html#fargate-gs-coredns

Specifically

Run the following command to remove the eks.amazonaws.com/compute-type : ec2 annotation from the CoreDNS Pods.

kubectl patch deployment coredns \
    -n kube-system \
    --type json \
    -p='[{"op": "remove", "path": "/spec/template/metadata/annotations/eks.amazonaws.com~1compute-type"}]'

This means for Fargate cluster, we probably should complete remove this annotation rather than patch it from EC2 to Fargate

aws-cdk/packages/@aws-cdk/aws-eks-v2-alpha/lib/cluster.ts

Lines 1564 to 1572 in 9e3123f

	new KubernetesPatch(this, 'CoreDnsComputeTypePatch', {
	cluster: this,
	resourceName: 'deployment/coredns',
	resourceNamespace: 'kube-system',
	applyPatch: renderPatch(CoreDnsComputeType.FARGATE),
	restorePatch: renderPatch(CoreDnsComputeType.EC2),
	});
	}
	}

I guess we need to confirm this with EKS team if it's recommended to completely remove the annotation.

[xazhao]

The issue is during the deletion, the Access Entry is deleted before deleting KubernetesPatch. Since the permission is deleted, it can't apply the restore patch resulting the error.

Will create a PR to fix it.

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.