addons: Response object is too long.

[cyril94440]

Describe the bug

I can't deploy the blueprint when adding the Nginx addon (not sure it is related to Nginx...)

Expected Behavior

I expect the eks blueprint to be able to split the cloudformation template otherwise its usage is very limited

Current Behavior

Response object is too long. is thrown and stack is getting deleted.

Reproduction Steps

#!/usr/bin/env node
import "source-map-support/register";
import * as cdk from "aws-cdk-lib";
import * as blueprints from "@aws-quickstart/eks-blueprints";
import { ArgoCDAddOnProps } from "@aws-quickstart/eks-blueprints";
import { KubernetesVersion, NodegroupAmiType } from "aws-cdk-lib/aws-eks";
import { TeamDatasciencePreprod, TeamDatascienceProd } from "../lib/teams";
import { InstanceType } from "aws-cdk-lib/aws-ec2";

//Secrets required in Secrets Manager : "argocd/repo/ssh_key" "argocd/admin_password"

const app = new cdk.App();

// ---- CONSTANTS ----
const clusterName = "clustername";
export const account = "000000000";
const region = "eu-west-3";
const domainAWSID = "Route53ZoneID"
const subdomain = 'my.domain.com'

const boostrapRepoUrl =
  "git@github.com:team/repo.git";

// ---- KARPENTER ----
const karpenterAddonProps = {
  provisionerSpecs: {
    "kubernetes.io/arch": ["arm64"],
  },
  subnetTags: {
    Name: `${clusterName}/${clusterName}-vpc/Priv1`,
  },
  securityGroupTags: {
    [`kubernetes.io/cluster/${clusterName}`]: "owned",
  },
};

// ---- ArgoCD ----
const argoCDAddonProps: ArgoCDAddOnProps = {
  bootstrapRepo: {
    repoUrl: boostrapRepoUrl,
    credentialsSecretName: "argocd-repo-ssh_key", //Not working :(
    credentialsType: "SSH",
    path: "main",
  },
  adminPasswordSecretName: "argocd/admin_password",
};

// ---- Default Instance ----
const clusterProvider = new blueprints.MngClusterProvider({
  version: KubernetesVersion.V1_25,
  desiredSize: 1,
  instanceTypes: [new InstanceType("t4g.medium")],
  diskSize: 50,
  amiType: NodegroupAmiType.BOTTLEROCKET_ARM_64
});


// ---- Teams ----
const teams: Array<blueprints.Team> = [
  new TeamDatasciencePreprod(app),
  new TeamDatascienceProd(app),
];

// ---- Full EKS cluster ----
const addOns: Array<blueprints.ClusterAddOn> = [
  new blueprints.addons.SecretsStoreAddOn(),
  new blueprints.addons.AwsLoadBalancerControllerAddOn(), 
  new blueprints.addons.NginxAddOn({externalDnsHostname:subdomain, certificateResourceName: "kscert"}),
  new blueprints.addons.ArgoCDAddOn(argoCDAddonProps),
  new blueprints.addons.MetricsServerAddOn(),
  new blueprints.addons.VpcCniAddOn(),
  new blueprints.addons.KarpenterAddOn(karpenterAddonProps),
  new blueprints.addons.ExternalDnsAddOn({
    hostedZoneResources:["kszone"],
  })
];

blueprints.EksBlueprint.builder()
  .account(account)
  .region(region)
  .resourceProvider("kszone", new blueprints.ImportHostedZoneProvider(domainAWSID, subdomain))
  .resourceProvider("kscert", new blueprints.CreateCertificateProvider('wildcert', `*.${subdomain}`, "kszone"))
  .addOns(...addOns)
  .clusterProvider(clusterProvider)
  .teams(...teams)
  .build(app, clusterName);

lib/teams.ts

import { ApplicationTeam } from "@aws-quickstart/eks-blueprints";
import { App } from "aws-cdk-lib";
import { account } from "../bin/eks";
import { ArnPrincipal } from "aws-cdk-lib/aws-iam";

export class TeamDatasciencePreprod extends ApplicationTeam {
  constructor(app: App) {
    super({
      name: "datascience-preprod",
      users: [new ArnPrincipal(`arn:aws:iam::${account}:user/test`)],
    });
  }
}

export class TeamDatascienceProd extends ApplicationTeam {
  constructor(app: App) {
    super({
      name: "datascience-prod",
      users: [new ArnPrincipal(`arn:aws:iam::${account}:user/test`)],
    });
  }
}

Possible Solution

Split in multiple Stack ? No documentation about how to do that with the blueprint

Additional Information/Context

No response

CDK CLI Version

2.76.0 (build 78c411b)

EKS Blueprints Version

No response

Node.js Version

18.14.0

Environment details (OS name and version, etc.)

Mac OS

Other information

No response

[shapirov103]

@cyril94440 We have nginx pattern (running atm) with Route53 and ACM integration working in our patterns here.

Do you mind providing the full stack of the error?

The full blueprint that you have is referencing teams that we don't have locally to reproduce.

[cyril94440]

@shapirov103 I've added the teams definition.

From your pattern, does buildAsync help split the cloudformation template in multiple files?

The other issue I am encountering with Nginx is the following:
Error: values don\'t meet the specifications of the schema(s) in the following chart(s):\nnginx-ingress:\n- controller.service.httpPort.enable: Invalid type. Expected: boolean, given: string\n- controller.service.httpsPort.targetPort: Invalid type. Expected: integer, given: string\n\n'

And it looks like I am not the only one: https://stackoverflow.com/questions/75778139/aws-cdk-eks-blueprint-fails-with-error-invalid-type/76175704#76175704

I'll take a look at the pattern you provided and try to get as close as possible from its definition.

I really appreciate your help.

Thanks!

[shapirov103]

@cyril94440 thank you for bringing it up to our attention. Initial look: appears to be a regression issue from the chart upgrade. This will be investigated and solved shortly.

[cyril94440]

@shapirov103 ok, any way to choose a stable previous version?

Also what about the "Response object is too long." how to overcome the cloudformation template size issue with eks blueprint?

Thanks

[shapirov103]

I am reproducing the nginx issue now. I cannot pinpoint exactly the release, but I see that in 1.6.0 and 1.5.3 there was a version change in the nginx helm chart. You can track this through https://github.com/aws-quickstart/cdk-eks-blueprints/releases (generally there is a manifest with addon versions for each release).

With respect to the response object too long: I never saw this issue before, regardless of how big the pattern was. Are you sending large chunks of data with the blueprint with your teams?

[cyril94440]

@shapirov103 Ok, I will try with a previous blueprint version.

I am not sure to understand your question but as you can see the team definition is very simple and I am not doing anything else.

Here are some informations that might help you:

Here is the output of the cdk deploy:

The lambda function cluster-awscdkawseksClust-ProviderframeworkisCompl-ID is emitting an error:

2023-05-05T15:55:42.955Z	df0c9041-4e05-4fb9-8efb-43065d40d9e1	ERROR	Invoke Error 	{
    "errorType": "Error",
    "errorMessage": "{\"RequestType\":\"Create\",\"ServiceToken\":\"arn:aws:lambda:eu-west-1:009032792877:function:udiniv2-awscdkawseksClust-ProviderframeworkonEvent-Clq544FUaJ94\",\"ResponseURL\":\"https://cloudformation-custom-resource-response-euwest1.s3-eu-west-1.amazonaws.com/arn%3Aaws%3Acloudformation%3Aeu-west-1%3A009032792877%3Astack/udiniv2/cf4d3bd0-eb5b-11ed-b86f-0290e7d204db%7Cudiniv2F9EAC977%7Cf5858173-784d-4f65-8ca5-366e4e890b2b?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230505T155129Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKIAU7SEXKRM6LZ53QKY%2F20230505%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Signature=67a14852beb531707cbc723ee06b9f858e11a0f5f31b6895af9708d0068f7cff\",\"StackId\":\"arn:aws:cloudformation:eu-west-1:009032792877:stack/udiniv2/cf4d3bd0-eb5b-11ed-b86f-0290e7d204db\",\"RequestId\":\"f5858173-784d-4f65-8ca5-366e4e890b2b\",\"LogicalResourceId\":\"udiniv2F9EAC977\",\"ResourceType\":\"Custom::AWSCDK-EKS-Cluster\",\"ResourceProperties\":{\"ServiceToken\":\"arn:aws:lambda:eu-west-1:009032792877:function:udiniv2-awscdkawseksClust-ProviderframeworkonEvent-Clq544FUaJ94\",\"Config\":{\"encryptionConfig\":[{\"provider\":{\"keyArn\":\"arn:aws:kms:eu-west-1:009032792877:key/97c8a0ac-e464-4066-ae10-86f7738306c8\"},\"resources\":[\"secrets\"]}],\"resourcesVpcConfig\":{\"endpointPrivateAccess\":\"true\",\"securityGroupIds\":[\"sg-0d5d38a16d8efc7d7\"],\"endpointPublicAccess\":\"true\",\"subnetIds\":[\"subnet-05057a31990d956b2\",\"subnet-0a788b272fb5e0124\",\"subnet-033049c8dc53e52d9\",\"subnet-0ec6b39361635c6df\",\"subnet-053e6c0477130ee28\",\"subnet-08720f7c9be1919ed\"]},\"roleArn\":\"arn:aws:iam::009032792877:role/udiniv2-udiniv2Role23F9DEAA-TTWVXGVIEC1B\",\"name\":\"udiniv2\",\"version\":\"1.25\"},\"AssumeRoleArn\":\"arn:aws:iam::009032792877:role/udiniv2-udiniv2CreationRole3D2A06E4-EHFG5BBS1ORS\",\"AttributesRevision\":\"2\"},\"PhysicalResourceId\":\"udiniv2\"}",
    "stack": [
        "Error: {\"RequestType\":\"Create\",\"ServiceToken\":\"arn:aws:lambda:eu-west-1:009032792877:function:udiniv2-awscdkawseksClust-ProviderframeworkonEvent-Clq544FUaJ94\",\"ResponseURL\":\"https://cloudformation-custom-resource-response-euwest1.s3-eu-west-1.amazonaws.com/arn%3Aaws%3Acloudformation%3Aeu-west-1%3A009032792877%3Astack/udiniv2/cf4d3bd0-eb5b-11ed-b86f-0290e7d204db%7Cudiniv2F9EAC977%7Cf5858173-784d-4f65-8ca5-366e4e890b2b?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230505T155129Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKIAU7SEXKRM6LZ53QKY%2F20230505%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Signature=67a14852beb531707cbc723ee06b9f858e11a0f5f31b6895af9708d0068f7cff\",\"StackId\":\"arn:aws:cloudformation:eu-west-1:009032792877:stack/udiniv2/cf4d3bd0-eb5b-11ed-b86f-0290e7d204db\",\"RequestId\":\"f5858173-784d-4f65-8ca5-366e4e890b2b\",\"LogicalResourceId\":\"udiniv2F9EAC977\",\"ResourceType\":\"Custom::AWSCDK-EKS-Cluster\",\"ResourceProperties\":{\"ServiceToken\":\"arn:aws:lambda:eu-west-1:009032792877:function:udiniv2-awscdkawseksClust-ProviderframeworkonEvent-Clq544FUaJ94\",\"Config\":{\"encryptionConfig\":[{\"provider\":{\"keyArn\":\"arn:aws:kms:eu-west-1:009032792877:key/97c8a0ac-e464-4066-ae10-86f7738306c8\"},\"resources\":[\"secrets\"]}],\"resourcesVpcConfig\":{\"endpointPrivateAccess\":\"true\",\"securityGroupIds\":[\"sg-0d5d38a16d8efc7d7\"],\"endpointPublicAccess\":\"true\",\"subnetIds\":[\"subnet-05057a31990d956b2\",\"subnet-0a788b272fb5e0124\",\"subnet-033049c8dc53e52d9\",\"subnet-0ec6b39361635c6df\",\"subnet-053e6c0477130ee28\",\"subnet-08720f7c9be1919ed\"]},\"roleArn\":\"arn:aws:iam::009032792877:role/udiniv2-udiniv2Role23F9DEAA-TTWVXGVIEC1B\",\"name\":\"udiniv2\",\"version\":\"1.25\"},\"AssumeRoleArn\":\"arn:aws:iam::009032792877:role/udiniv2-udiniv2CreationRole3D2A06E4-EHFG5BBS1ORS\",\"AttributesRevision\":\"2\"},\"PhysicalResourceId\":\"udiniv2\"}",
        "    at isComplete (/var/task/framework.js:55:15)",
        "    at processTicksAndRejections (internal/process/task_queues.js:95:5)",
        "    at async Runtime.handler (/var/task/cfn-response.js:52:13)"
    ]
}

[shapirov103]

The CFN Object Too Long maybe a side effect of the error on the NGINX side. This will be out shortly with 1.7.3 (#677 ), I suggest you retest it then.
I saw similar issues against CDK, e.g. this one.

[shapirov103]

Please try 1.7.3 which is out. I anticipate it to fix the issues.

[cyril94440]

Hi @shapirov103 It is working perfectly now! Thank you for your help.