Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KeyError when running sam-beta-cdk build on projen typescript app #2849

Closed
mbuotidem opened this issue May 3, 2021 · 10 comments
Closed

KeyError when running sam-beta-cdk build on projen typescript app #2849

mbuotidem opened this issue May 3, 2021 · 10 comments

Comments

@mbuotidem
Copy link

Description:

I have a cdk TypeScript project created using projen. My project contains two stacks, a codepipeline stack and the application stack. In the application stack, I'm creating a lambda resource. When I run sam-beta-cdk build, I get the error below:

misaac:~/environment/guichet (main) $ sam-beta-cdk build
Synthesizing CDK App
Traceback (most recent call last):
  File "samcli/__main__.py", line 12, in <module>
  File "click/core.py", line 829, in __call__
  File "click/core.py", line 782, in main
  File "click/core.py", line 1259, in invoke
  File "click/core.py", line 1066, in invoke
  File "click/core.py", line 610, in invoke
  File "click/decorators.py", line 73, in new_func
  File "click/core.py", line 610, in invoke
  File "samcli/lib/telemetry/metric.py", line 174, in wrapped
  File "samcli/lib/telemetry/metric.py", line 141, in wrapped
  File "samcli/lib/iac/utils/helpers.py", line 30, in wrapper
  File "samcli/lib/iac/cdk/plugin.py", line 116, in get_project
  File "samcli/lib/iac/cdk/plugin.py", line 207, in _get_project_from_cloud_assembly
  File "samcli/lib/iac/cdk/plugin.py", line 207, in <listcomp>
  File "samcli/lib/iac/cdk/plugin.py", line 222, in _build_stack
  File "samcli/lib/iac/cdk/plugin.py", line 280, in _build_resources_section
KeyError: '/home/ec2-user/environment/guichet/lambda'
[9116] Failed to execute script __main__

Steps to reproduce:

Install projen and sam-cdk-beta

Create a projen TypeScript app - projen new awscdk-app-ts

Set "@aws-cdk/core:newStyleStackSynthesis": true in cdk.json

Create a codepipeline stack in main.ts

Create an application stack that creates a lambda resource using code: lambda.Code.fromAsset, eg code: lambda.Code.fromAsset('lambda') with the folder lambda placed at the root

Create your lambda and a package.json file in the lambda folder

Initialize your application stack in the codepipeline stack

Run sam-beta-cdk build.

You can also just use the repo here : https://github.com/mbuotidem/guichet/tree/sam-beta-cdk

Observed result:

misaac:~/environment/guichet (sam-beta-cdk) $ sam-beta-cdk build --debug
2021-05-03 16:01:22,491 | Determining project type...
2021-05-03 16:01:22,491 | The project is a CDK project.
2021-05-03 16:01:22,526 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-05-03 16:01:22,528 | CDK Toolkit found at ./node_modules/aws-cdk/bin/cdk
2021-05-03 16:01:22,529 | Synthesizing CDK App
2021-05-03 16:01:22,529 | command: ['./node_modules/aws-cdk/bin/cdk', 'synth', '--no-staging', '-o', '.aws-sam/.cdk-out']
2021-05-03 16:01:30,971 | Cloud assembly synthed at .aws-sam/.cdk-out
2021-05-03 16:01:30,972 | _validate_cloud_assembly: .aws-sam/.cdk-out
2021-05-03 16:01:30,976 | ca-stack_name: GuichetPipelineDev
2021-05-03 16:01:30,977 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '43e09bd1-cd43-44f4-9ceb-811953ff6ab7', 'installationId': '9bc63fd8-2d14-436d-a835-745df131378d', 'sessionId': '6c7f3a49-a0c5-4017-8655-f74a301a8106', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.7.10', 'samcliVersion': '1.22.0.dev202104291816', 'metricSpecificAttributes': {'projectType': 'CDK'}, 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 8449, 'exitReason': 'KeyError', 'exitCode': 255}}]}
2021-05-03 16:01:31,353 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Traceback (most recent call last):
  File "samcli/__main__.py", line 12, in <module>
  File "click/core.py", line 829, in __call__
  File "click/core.py", line 782, in main
  File "click/core.py", line 1259, in invoke
  File "click/core.py", line 1066, in invoke
  File "click/core.py", line 610, in invoke
  File "click/decorators.py", line 73, in new_func
  File "click/core.py", line 610, in invoke
  File "samcli/lib/telemetry/metric.py", line 174, in wrapped
  File "samcli/lib/telemetry/metric.py", line 141, in wrapped
  File "samcli/lib/iac/utils/helpers.py", line 30, in wrapper
  File "samcli/lib/iac/cdk/plugin.py", line 116, in get_project
  File "samcli/lib/iac/cdk/plugin.py", line 207, in _get_project_from_cloud_assembly
  File "samcli/lib/iac/cdk/plugin.py", line 207, in <listcomp>
  File "samcli/lib/iac/cdk/plugin.py", line 222, in _build_stack
  File "samcli/lib/iac/cdk/plugin.py", line 280, in _build_resources_section
KeyError: '/home/ec2-user/environment/guichet/lambda'
[11297] Failed to execute script __main__

Expected result:

sam-beta-cdk build should complete without any issues and I should be able to then run sam-cdk-beta local invoke to test the lambda function locally.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: Amazon Linux 2 on EC2, WSL2 Ubuntu 20.04 on Windows 10, Windows 10 64bit
  2. sam --version: SAM CLI, version 1.22.0.dev202104291816
  3. AWS region: us-east-1
@awood45 awood45 added area/build sam build command stage/needs-investigation Requires a deeper investigation area/cdk labels May 3, 2021
@moelasmar
Copy link
Contributor

I was not able to reproduce this issue
I executed the following steps in a cloud9 Amazon Linux 2 instance

Admin:~/environment $ git clone https://github.com/mbuotidem/guichet.git
Cloning into 'guichet'...
remote: Enumerating objects: 169, done.
remote: Counting objects: 100% (169/169), done.
remote: Compressing objects: 100% (103/103), done.
remote: Total 169 (delta 80), reused 137 (delta 53), pack-reused 0
Receiving objects: 100% (169/169), 163.48 KiB | 7.78 MiB/s, done.
Resolving deltas: 100% (80/80), done.
Admin:~/environment $ cd guichet/
Admin:~/environment/guichet (main) $ npm install 
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated request-promise-native@1.0.9: request-promise-native has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^2.1.2 (node_modules/jest-haste-map/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.3.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN guichet@0.0.0 No repository field.

added 1229 packages from 592 contributors and audited 1234 packages in 30.456s

89 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

Admin:~/environment/guichet (main) $ sam-beta-cdk build --debug
2021-05-03 20:39:17,513 | Determining project type...
2021-05-03 20:39:17,513 | The project is a CDK project.
2021-05-03 20:39:17,570 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-05-03 20:39:17,572 | CDK Toolkit found at ./node_modules/aws-cdk/bin/cdk
2021-05-03 20:39:17,573 | Synthesizing CDK App
2021-05-03 20:39:17,573 | command: ['./node_modules/aws-cdk/bin/cdk', 'synth', '--no-staging', '-o', '.aws-sam/.cdk-out']
2021-05-03 20:39:23,910 | Cloud assembly synthed at .aws-sam/.cdk-out
2021-05-03 20:39:23,911 | _validate_cloud_assembly: .aws-sam/.cdk-out
2021-05-03 20:39:23,914 | ca-stack_name: GuichetPipelineDev
2021-05-03 20:39:23,914 | Using config file: samconfig.toml, config environment: default
2021-05-03 20:39:23,914 | Expand command line arguments to:
2021-05-03 20:39:23,915 | --template_file=/home/ec2-user/environment/guichet/template.yml --build_dir=.aws-sam/build --cache_dir=.aws-sam/cache --project_type=CDK --iac=<samcli.lib.iac.cdk.plugin.CdkPlugin object at 0x7f8d05748f50> --project=<samcli.lib.iac.interface.Project object at 0x7f8d05697b10> 
2021-05-03 20:39:23,969 | 'build' command is called
2021-05-03 20:39:23,969 | Collected default values for parameters: {'gitOwnerParameter': 'guichet-repo-owner', 'gitRepoParameter': 'guichet-repo', 'gitBranchParameter': 'guichet-repo-branch', 'BootstrapVersion': '/cdk-bootstrap/hnb659fds/version'}
2021-05-03 20:39:24,045 | 20 stacks found in the template
2021-05-03 20:39:24,045 | Collected default values for parameters: {'gitOwnerParameter': 'guichet-repo-owner', 'gitRepoParameter': 'guichet-repo', 'gitBranchParameter': 'guichet-repo-branch', 'BootstrapVersion': '/cdk-bootstrap/hnb659fds/version'}
2021-05-03 20:39:24,116 | 20 resources found in the stack GuichetPipelineDev
2021-05-03 20:39:24,116 | Collected default values for parameters: {'gitOwnerParameter': 'guichet-repo-owner', 'gitRepoParameter': 'guichet-repo', 'gitBranchParameter': 'guichet-repo-branch', 'BootstrapVersion': '/cdk-bootstrap/hnb659fds/version'}
2021-05-03 20:39:24,187 | Collected default values for parameters: {'gitOwnerParameter': 'guichet-repo-owner', 'gitRepoParameter': 'guichet-repo', 'gitBranchParameter': 'guichet-repo-branch', 'BootstrapVersion': '/cdk-bootstrap/hnb659fds/version'}
2021-05-03 20:39:24,258 | Instantiating build definitions
2021-05-03 20:39:24,258 | No previous build graph found, generating new one

Build Succeeded

Built Artifacts  : .aws-sam/build

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: cdk deploy -a .aws-sam/build
    
2021-05-03 20:39:24,395 | Installed version 1.22.0.dev202104291816, current version 1.23.0

SAM CLI update available (1.23.0); (1.22.0.dev202104291816 installed)
To download: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html
2021-05-03 20:39:24,396 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '2acabea9-5c7e-4367-82ee-5f09361ad7ab', 'installationId': '59a07490-4ea0-42ac-bd99-98782ce41d9d', 'sessionId': 'ba454a77-e77f-46a0-a25b-bbca7369fbdf', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.7.10', 'samcliVersion': '1.22.0.dev202104291816', 'metricSpecificAttributes': {'projectType': 'CDK'}, 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 6823, 'exitReason': 'success', 'exitCode': 0}}]}
2021-05-03 20:39:24,693 | Telemetry response: 200

@mbuotidem
Copy link
Author

@moelasmar I believe that might be because you're in the main branch. Could you try switching to the sam-beta-cdk branch?

The main branch does not currently create the stack that has the reference to the lambda function.

@moelasmar
Copy link
Contributor

ok, thanks for the update

@moelasmar
Copy link
Contributor

I can reproduce the issue

@moelasmar moelasmar added type/bug and removed stage/needs-investigation Requires a deeper investigation labels May 3, 2021
@redbaron
Copy link

redbaron commented May 4, 2021

Workaround is to set "@aws-cdk/core:newStyleStackSynthesis": false , then it builds fine.

@brainstorm
Copy link

I can confirm and reproduce this in my CDK-SAM python project as well:

aws-samples/aws-sam-support-aws-cdk#1 (comment)

The workaround suggested by @redbaron seems to work great... I reckon that directive should be included in cdk init for CDKv2 to avoid this situation?:

aws/aws-cdk#14814

@teemuniiranen
Copy link

teemuniiranen commented Jul 15, 2021

I reproduced this also with TypeScript CDK 2.0.0-rc.11, and TypeScript Lambda function. After changing "@aws-cdk/core:newStyleStackSynthesis": false I still got error from NodejsNpmBuilder:NpmPack. Could this be related then to NodejsFunction I use in CDK code?

Running NodejsNpmBuilder:NpmPack

Build Failed
Error: NodejsNpmBuilder:NpmPack - NPM Failed: npm ERR! code ENOENT
...
npm ERR! enoent ENOENT: no such file or directory, open '/path/to/project/.aws-sam/.cdk-out/asset.87f5f1b13a1603a16408f67af2b0e053b00da1ccd68efc02f60775ed01a4f109/package.json'

Indeed the package.json is missing. There is only index.js file which contains my Lambda code as JavaScript. So dependencies, that package.json, does not get copied/generated?

EDIT:
index.js contains all the dependencies bundled in this case. If I just create an empty package.json, for example by running npm init in the asset directory /path/to/project/.aws-sam/.cdk-out/asset.87f5f1b13a1603a16408f67af2b0e053b00da1ccd68efc02f60775ed01a4f109, build succeeds and I can use sam-beta-cdk local invoke and sam-beta-cdk local start-api.

EDIT2:
There is a way to leave something unbundled with NodejsFunction (or underlying esbuild) and have it as a node_module. https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda_nodejs-readme.html#install-modules

Then package.json gets generated but this does not work either. The build then complains about the missing name and version:

Build Failed
Error: NodejsNpmBuilder:NpmPack - NPM Failed: npm ERR! Invalid package, must have name and version

And indeed those are missing. I tested this with lambda-log dependency and then package.json looks like this: {"dependencies":{"lambda-log":"^3.0.0"}}.

Proposed fix: optimal solution would to allow totally bundled JavaScript code. Fix could be "empty package.json" generation (if it is missing) after "Synthesizing CDK App" but before running "NodejsNpmBuilder:NpmPack" when running sam-beta-cdk build. Or can we then just skip the packing totally? It is already built and bundled with esbuild by the CDK in this case.

@pseudo-su
Copy link

This other issue seems related

#2968 (comment)

@SirIle
Copy link

SirIle commented Nov 4, 2021

Hi, I just ran into this while trying to get SAM and CDK to work. As a workaround I defined the NodejsFunction to copy a dummy package.json after the bundling to the out directory which seems to work. First I created a package.json file containing just the bare minimum:

{
  "name": "dummy",
  "version": "1.0.0"
}

For this test I put it into the tests folder that was created by scaffolding. After that I changed the function definition to contain the code that copies the file to the out directory:

    const helloFunction = new NodejsFunction(this, 'HelloHandler', {
      handler: 'lambdaHandler',
      entry: 'hello-world/app.js',
      bundling: {
        commandHooks: {
          beforeBundling(inputDir: string, outputDir: string): any {}, 
          afterBundling(inputDir: string, outputDir: string): string[] {
            return [`cp ${inputDir}/test/package.json ${outputDir}`];
          },
          beforeInstall(inputDir: string, outputDir: string): any {}
        }
      }
    });

I admit that's not the prettiest workaround, but it seems to work and running sam-beta-cdk local invoke and sam-beta-cdk local start-api work. Also when running the sam-beta-cdk buildin the background while the server is running takes the changes into use without restarting the server.

@mildaniel
Copy link
Contributor

Hey all, since AWS SAM CLI now supports CDK differently than in the beta, this issue shouldn't exist anymore since CDK will handle this type of building logic. Please try using the GA version and re-open if a similar problem exists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

10 participants