aws-cdk-lib/aws-eks: waiter-state-machine is not authorized to access the Log Destination #33332

SeraphicRav · 2025-02-07T08:00:48Z

Describe the bug

Hello !

I am trying to create a cluster in an AWS Account I have admin permissions on.

The creation fails when a Provider I have no control on fails:

Failed resources:
Kubernetes-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePr-G3A1X5DKUF26 | 4:05:50 PM | CREATE_FAILED        | AWS::StepFunctions::StateMachine      | @aws-cdk--aws-eks.ClusterResourceProvider/Provider/waiter-state-machine (Providerwaiterstatemachine5D4A9DF0) Resource handler returned message: "The state machine IAM Role is not authorized to access the Log Destination (Service: AWSStepFunctions; Status Code: 400; Error Code: AccessDeniedException; Request ID: c86629f3-fe7d-4c89-b782-f73b6bc410ed; Proxy: null)" (RequestToken: b9a6f357-3e31-2d56-ebc3-7f1d150459d3, HandlerErrorCode: AccessDenied)
Kubernetes | 4:06:08 PM | CREATE_FAILED        | AWS::CloudFormation::Stack            | @aws-cdk--aws-eks.ClusterResourceProvider.NestedStack/@aws-cdk--aws-eks.ClusterResourceProvider.NestedStackResource (awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourceProviderNestedStackResource9827C454) Embedded stack arn:aws:cloudformation:ap-northeast-1:797698216904:stack/Kubernetes-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePr-G3A1X5DKUF26/b0fc7030-e521-11ef-a2f8-0ef66568cceb was not successfully created: The following resource(s) failed to create: [Providerwaiterstatemachine5D4A9DF0]. 
❌  Kubernetes failed: _ToolkitError: The stack named Kubernetes failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource handler returned message: "The state machine IAM Role is not authorized to access the Log Destination (Service: AWSStepFunctions; Status Code: 400; Error Code: AccessDeniedException; Request ID: c86629f3-fe7d-4c89-b782-f73b6bc410ed; Proxy: null)" (RequestToken: b9a6f357-3e31-2d56-ebc3-7f1d150459d3, HandlerErrorCode: AccessDenied), Embedded stack arn:aws:cloudformation:ap-northeast-1:797698216904:stack/Kubernetes-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePr-G3A1X5DKUF26/b0fc7030-e521-11ef-a2f8-0ef66568cceb was not successfully created: The following resource(s) failed to create: [Providerwaiterstatemachine5D4A9DF0].

How can I fix this as it seems to be out of my scope ?

Regression Issue

Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

EKS cluster is created without error.

Current Behavior

The creation fails when a Provider I have no control on fails:

Failed resources:
Kubernetes-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePr-G3A1X5DKUF26 | 4:05:50 PM | CREATE_FAILED        | AWS::StepFunctions::StateMachine      | @aws-cdk--aws-eks.ClusterResourceProvider/Provider/waiter-state-machine (Providerwaiterstatemachine5D4A9DF0) Resource handler returned message: "The state machine IAM Role is not authorized to access the Log Destination (Service: AWSStepFunctions; Status Code: 400; Error Code: AccessDeniedException; Request ID: c86629f3-fe7d-4c89-b782-f73b6bc410ed; Proxy: null)" (RequestToken: b9a6f357-3e31-2d56-ebc3-7f1d150459d3, HandlerErrorCode: AccessDenied)
Kubernetes | 4:06:08 PM | CREATE_FAILED        | AWS::CloudFormation::Stack            | @aws-cdk--aws-eks.ClusterResourceProvider.NestedStack/@aws-cdk--aws-eks.ClusterResourceProvider.NestedStackResource (awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourceProviderNestedStackResource9827C454) Embedded stack arn:aws:cloudformation:ap-northeast-1:797698216904:stack/Kubernetes-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePr-G3A1X5DKUF26/b0fc7030-e521-11ef-a2f8-0ef66568cceb was not successfully created: The following resource(s) failed to create: [Providerwaiterstatemachine5D4A9DF0]. 
❌  Kubernetes failed: _ToolkitError: The stack named Kubernetes failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource handler returned message: "The state machine IAM Role is not authorized to access the Log Destination (Service: AWSStepFunctions; Status Code: 400; Error Code: AccessDeniedException; Request ID: c86629f3-fe7d-4c89-b782-f73b6bc410ed; Proxy: null)" (RequestToken: b9a6f357-3e31-2d56-ebc3-7f1d150459d3, HandlerErrorCode: AccessDenied), Embedded stack arn:aws:cloudformation:ap-northeast-1:797698216904:stack/Kubernetes-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePr-G3A1X5DKUF26/b0fc7030-e521-11ef-a2f8-0ef66568cceb was not successfully created: The following resource(s) failed to create: [Providerwaiterstatemachine5D4A9DF0].

Reproduction Steps

I have the following package.json:

{
  "name": "cdk",
  "bin": {
    "cdk": "bin/cdk.js"
  },
  "scripts": {
    "build": "tsc",
    "watch": "tsc -w",
    "test": "jest",
    "cdk": "cdk"
  },
  "devDependencies": {
    "@types/jest": "^29.5.14",
    "@types/semver": "^7.5.8",
    "aws-cdk": "2.178.1",
    "jest": "^29.7.0",
    "ts-jest": "^29.2.5",
    "ts-node": "^10.9.2",
    "typescript": "~5.6.3"
  },
  "dependencies": {
    "@aws-cdk/lambda-layer-kubectl-v32": "2.0.0",
    "aws-cdk-lib": "2.178.1",
    "constructs": "^10.0.0",
    "semver": "^7.7.1"
  }
}

And here is a kind of simplified version of my Stack

import { KubectlV32Layer } from "@aws-cdk/lambda-layer-kubectl-v32";
import { Stack, StackProps } from "aws-cdk-lib";
import { IVpc, SubnetType } from "aws-cdk-lib/aws-ec2";
import {
  Cluster,
  ClusterLoggingTypes,
  EndpointAccess,
  KubectlProvider,
  KubernetesVersion,
} from "aws-cdk-lib/aws-eks";
import { Construct } from "constructs";
import {
  AccountRootPrincipal,
  Effect,
  Group,
  IRole,
  ManagedPolicy,
  PolicyStatement,
  Role,
} from "aws-cdk-lib/aws-iam";

const kubernetesVersion = "1.32";

export class EksCluster extends Stack {
  public readonly cluster: Cluster;
  public readonly kubernetesVersion: KubernetesVersion;

  constructor(
    scope: Construct,
    id: string,
    props: StackProps & {
      vpc: IVpc;
    }
  ) {
    super(scope, id, props);
    const { vpc, awsAccountProduction } = props;
    this.kubernetesVersion = KubernetesVersion.of(kubernetesVersion);
    const adminRole = this.makeIamRole("Admin");
    this.cluster = this.makeCluster({ vpc, mastersRole: adminRole });
  }

  private makeIamRole(level: "Admin") {
    const role = new Role(this, `${level}Role`, {
      assumedBy: new AccountRootPrincipal(),
      roleName: `eks-${level.toLowerCase()}`,
    });
    const policyStatement = new PolicyStatement({
      resources: [role.roleArn],
      actions: ["sts:AssumeRole"],
      effect: Effect.ALLOW,
    });
    const managedPolicy = new ManagedPolicy(this, `${level}Policy`, {
      managedPolicyName: `assume-eks-${level.toLocaleLowerCase()}-role`,
    });
    managedPolicy.addStatements(policyStatement);
    return role;
  }

  private makeCluster({
    vpc,
    mastersRole,
  }: {
    vpc: IVpc;
    mastersRole?: IRole;
  }): Cluster {
    const cluster = new Cluster(this, "Cluster", {
      clusterName: "eks-default",
      vpc,
      vpcSubnets: [{ subnetType: SubnetType.PRIVATE_WITH_EGRESS }],
      defaultCapacity: 0,
      version: this.kubernetesVersion,
      kubectlLayer: new KubectlV32Layer(this, "kubectl"),
      mastersRole,
      clusterLogging: [
        ClusterLoggingTypes.API,
        ClusterLoggingTypes.AUTHENTICATOR,
        ClusterLoggingTypes.CONTROLLER_MANAGER,
        ClusterLoggingTypes.SCHEDULER,
      ],
    });
    return cluster;
  }

Portion of the generated template:

  "ProviderwaiterstatemachineRoleDefaultPolicyD3C3DA1A": {
   "Type": "AWS::IAM::Policy",
   "Properties": {
    "PolicyDocument": {
     "Statement": [
      {
       "Action": "lambda:InvokeFunction",
       "Effect": "Allow",
       "Resource": [
        {
         "Fn::GetAtt": [
          "ProviderframeworkisComplete26D7B0CB",
          "Arn"
         ]
        },
        {
         "Fn::GetAtt": [
          "ProviderframeworkonTimeout0B47CA38",
          "Arn"
         ]
        },
        {
         "Fn::Join": [
          "",
          [
           {
            "Fn::GetAtt": [
             "ProviderframeworkisComplete26D7B0CB",
             "Arn"
            ]
           },
           ":*"
          ]
         ]
        },
        {
         "Fn::Join": [
          "",
          [
           {
            "Fn::GetAtt": [
             "ProviderframeworkonTimeout0B47CA38",
             "Arn"
            ]
           },
           ":*"
          ]
         ]
        }
       ]
      },
      {
       "Action": [
        "logs:CreateLogDelivery",
        "logs:CreateLogStream",
        "logs:DeleteLogDelivery",
        "logs:DescribeLogGroups",
        "logs:DescribeResourcePolicies",
        "logs:GetLogDelivery",
        "logs:ListLogDeliveries",
        "logs:PutLogEvents",
        "logs:PutResourcePolicy",
        "logs:UpdateLogDelivery"
       ],
       "Effect": "Allow",
       "Resource": "*"
      }
     ],
     "Version": "2012-10-17"
    },
    "PolicyName": "ProviderwaiterstatemachineRoleDefaultPolicyD3C3DA1A",
    "Roles": [
     {
      "Ref": "ProviderwaiterstatemachineRole0C7159F9"
     }
    ]
   },
   "Metadata": {
    "aws:cdk:path": "Kubernetes/@aws-cdk--aws-eks.ClusterResourceProvider/Provider/waiter-state-machine/Role/DefaultPolicy/Resource"
   }
  },
  "ProviderwaiterstatemachineLogGroupDD693A98": {
   "Type": "AWS::Logs::LogGroup",
   "Properties": {
    "LogGroupName": {
     "Fn::Join": [
      "",
      [
       "/aws/vendedlogs/states/waiter-state-machine-",
       {
        "Ref": "ProviderframeworkisComplete26D7B0CB"
       },
       "-c8100e0c94c82f842732ba88b108b9eee575b910e3"
      ]
     ]
    },
    "RetentionInDays": 731,
    "Tags": [
     {
      "Key": "CmBillingGroup",
      "Value": "CRM Shared"
     }
    ]
   },
   "UpdateReplacePolicy": "Retain",
   "DeletionPolicy": "Retain",
   "Metadata": {
    "aws:cdk:path": "Kubernetes/@aws-cdk--aws-eks.ClusterResourceProvider/Provider/waiter-state-machine/LogGroup/Resource"
   }
  },

Possible Solution

No response

Additional Information/Context

I saw people say that the resource policies for logs have a limited size but I am not sure if that is the issue as the logs are in /aws/vendedlogs/states/:

aws logs describe-resource-policies | wc
      54      85    4575

CDK CLI Version

2.178.1 (build ae342cb)

Framework Version

No response

Node.js Version

v22.11.0

OS

macOS Version 15.2 (24C101)

Language

TypeScript

Language Version

5.6.3

Other information

No response

The text was updated successfully, but these errors were encountered:

pahud · 2025-02-07T17:43:59Z

Hi

We are still working on the 1.32 support in #33339 and we can't guarantee if 1.32 is working now before that PR is merged.

Are you able to simplify your code and see if 1.31 is working?

For example

import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as iam from 'aws-cdk-lib/aws-iam';
import { App, Stack, StackProps } from 'aws-cdk-lib';
import { KubectlV31Layer } from '@aws-cdk/lambda-layer-kubectl-v31';
import * as eks from 'aws-cdk-lib/aws-eks';
import { Construct } from 'constructs';

export class EksClusterLatestVersion extends Stack {
  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props);

    const vpc = new ec2.Vpc(this, 'Vpc', { natGateways: 1 });
    const mastersRole = new iam.Role(this, 'Role', {
      assumedBy: new iam.AccountRootPrincipal(),
    });

    new eks.Cluster(this, 'Cluster', {
      vpc,
      mastersRole,
      version: eks.KubernetesVersion.V1_31,
      kubectlLayer: new KubectlV31Layer(this, 'KubectlLayer'),
      defaultCapacity: 1,
    });
  }
}

const app = new App();
new EksClusterLatestVersion(app, 'v31-stack', {
  env: {
    account: process.env.CDK_DEFAULT_ACCOUNT,
    region: process.env.CDK_DEFAULT_REGION,
  },
})

github-actions · 2025-02-09T20:04:56Z

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

SeraphicRav · 2025-02-10T00:45:14Z

Hi !

Thanks for your answer !

I tried the provided code (modified it slightly to use an existing VPC because the account reached its quota), updated the package JSON to use Kubernetes 1.31. I still get the same error:

v31-stack-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePro-1QUOTCF8XU9GU | 20/17 | 9:42:22 AM | CREATE_COMPLETE      | AWS::Logs::LogGroup                   | @aws-cdk--aws-eks.ClusterResourceProvider/Provider/waiter-state-machine/LogGroup (ProviderwaiterstatemachineLogGroupDD693A98) 
v31-stack-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePro-1QUOTCF8XU9GU | 21/17 | 9:42:32 AM | CREATE_COMPLETE      | AWS::IAM::Policy                      | @aws-cdk--aws-eks.ClusterResourceProvider/Provider/waiter-state-machine/Role/DefaultPolicy (ProviderwaiterstatemachineRoleDefaultPolicyD3C3DA1A) 
v31-stack-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePro-1QUOTCF8XU9GU | 21/17 | 9:42:34 AM | CREATE_IN_PROGRESS   | AWS::StepFunctions::StateMachine      | @aws-cdk--aws-eks.ClusterResourceProvider/Provider/waiter-state-machine (Providerwaiterstatemachine5D4A9DF0) 
v31-stack-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePro-1QUOTCF8XU9GU | 21/17 | 9:42:57 AM | CREATE_FAILED        | AWS::StepFunctions::StateMachine      | @aws-cdk--aws-eks.ClusterResourceProvider/Provider/waiter-state-machine (Providerwaiterstatemachine5D4A9DF0) Resource handler returned message: "The state machine IAM Role is not authorized to access the Log Destination (Service: AWSStepFunctions; Status Code: 400; Error Code: AccessDeniedException; Request ID: 9dab9fa2-211b-430e-9b10-80a821e3b120; Proxy: null)" (RequestToken: 0a2e6a33-e092-1fa1-b1b6-f205fbfa1e42, HandlerErrorCode: AccessDenied)
v31-stack-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePro-1QUOTCF8XU9GU | 21/17 | 9:42:58 AM | CREATE_FAILED        | AWS::CloudFormation::Stack            | v31-stack-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePro-1QUOTCF8XU9GU The following resource(s) failed to create: [Providerwaiterstatemachine5D4A9DF0]. 
v31-stack | 21/17 | 9:43:08 AM | CREATE_FAILED        | AWS::CloudFormation::Stack            | @aws-cdk--aws-eks.ClusterResourceProvider.NestedStack/@aws-cdk--aws-eks.ClusterResourceProvider.NestedStackResource (awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourceProviderNestedStackResource9827C454) Embedded stack arn:aws:cloudformation:ap-northeast-1:797698216904:stack/v31-stack-awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourcePro-1QUOTCF8XU9GU/bbceef10-e747-11ef-b4bc-069219310d87 was not successfully created: The following resource(s) failed to create: [Providerwaiterstatemachine5D4A9DF0]. 
v31-stack | 21/17 | 9:43:09 AM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack            | v31-stack The following resource(s) failed to create: [awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourceProviderNestedStackResource9827C454]. Rollback requested by user.

SeraphicRav added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Feb 7, 2025

github-actions bot added the @aws-cdk/aws-iam Related to AWS Identity and Access Management label Feb 7, 2025

pahud self-assigned this Feb 7, 2025

pahud added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Feb 7, 2025

pahud removed their assignment Feb 7, 2025

pahud added effort/medium Medium work item – several days of effort p2 and removed needs-triage This issue or PR still needs to be triaged. labels Feb 7, 2025

github-actions bot added the closing-soon This issue will automatically close in 4 days unless further comments are made. label Feb 9, 2025

github-actions bot removed closing-soon This issue will automatically close in 4 days unless further comments are made. response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. labels Feb 10, 2025

github-actions bot mentioned this issue Mar 1, 2025

Monthly issue metrics report #33660

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

aws-cdk-lib/aws-eks: waiter-state-machine is not authorized to access the Log Destination #33332

aws-cdk-lib/aws-eks: waiter-state-machine is not authorized to access the Log Destination #33332

SeraphicRav commented Feb 7, 2025

pahud commented Feb 7, 2025

github-actions bot commented Feb 9, 2025

SeraphicRav commented Feb 10, 2025

aws-cdk-lib/aws-eks: waiter-state-machine is not authorized to access the Log Destination #33332

aws-cdk-lib/aws-eks: waiter-state-machine is not authorized to access the Log Destination #33332

Comments

SeraphicRav commented Feb 7, 2025

Describe the bug

Regression Issue

Last Known Working CDK Version

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

CDK CLI Version

Framework Version

Node.js Version

OS

Language

Language Version

Other information

pahud commented Feb 7, 2025

github-actions bot commented Feb 9, 2025

SeraphicRav commented Feb 10, 2025