(core): Normalize hashing algorithm between different OSs

[TRANTANKHOA]

The Question

How can I have the same asset hash when building the stack from different OS?
I'm very reluctant to implement a custom hash for the source content but the snapshot test we have will fail when run in the CI container since the snapshot was created on Windows.

Environment

• CDK CLI Version: 1.100.0
• Module Version: 1.100.0
• Node.js Version: v14.16.0
• OS: Windows 10 Pro 19042.928
• Language (Version): ~3.9.7

Other information

For example I have a PythonFunction as belows

new PythonFunction(this, 'TaskMonitoringLambda', {
    entry: path.join(__dirname, '..', 'lambda'),
    index: 'failed_task_notification.py',
    handler: 'push',
    runtime: Runtime.PYTHON_3_7,
    retryAttempts: this.node.tryGetContext('max_retries'),
    memorySize: 128,
    functionName: `${machineID}TaskMonitoringLambda`,
    assetHashType: AssetHashType.SOURCE, // see https://github.com/aws/aws-cdk/pull/12984
    environment: {
        'FAILED_TASK_TOPIC': failedTaskTopic.topicArn,
        'REGION': this.node.tryGetContext('region'),
        'ENV': env
    }
})

and the asset hash for this is stable between repeated builds in Windows, despite cleaning out the caches and docker images etc..
However the hash produced in Centos container is different.

[jogold]

What's in your lambda folder? Can you check if the file sizes are identical on Windows and CentOS?

The size is part of the hash:

aws-cdk/packages/@aws-cdk/core/lib/fs/fingerprint.ts

Line 91 in 0c1acc3

return `${stat.size}:${hash.digest('hex')}`;

[peterwoodworth]

@TRANTANKHOA let me know if the advice given by jogold was helpful!

[TRANTANKHOA]

thanks @jogold and @peterwoodworth . I wasn't interested in finding out why they are different since there may be more than one reasons. Ideally for our use case, the hash should be based on the textual content of the files only, not os-related factors including the LF/CRLF stuffs. Apparently this is an issue across many hashing algos I've checked and people seem having the same trouble across many languages too.

For now what I've settle with is building a hash dictionary on my dev. machine and check that dictionary into git so that my cdk app can use it when in another os.

For anyone having the same problem, the gist of the hack is as follows

1. First build the docker/lambda directory hash as part of unit test

test('Update hash dictionary', async () => {
  const osNameString = osName();
  // console.log(`OS name: ${osNameString}`)
  if (osNameString.includes('Linux') == false) {
      // This snippet only run in dev machines
      const hashOptions = {
          folders: {
              exclude: ['.*', '**.*', '**node_modules', '**test', '**build', '**dist', '*env', 'bin', 'lib']
          },
          files: {
              exclude: ['**.d.ts'],
              include: ['**.ts', '**.json', '**.py', '**.sh', 'Dockerfile', 'requirements.txt']
          },
          algo: 'md5'
      }
      await hashElement('.', path.join(__dirname, '..'), hashOptions)
          .then(hash => {
              const fs = require('fs');
              const data = JSON.stringify(hash, null, 2);
              fs.writeFile("hash.json", data, function(err: NodeJS.ErrnoException) {
                  if (err) {
                      console.log(err);
                  }
              });
              // console.log(hash.children)
          })
          .catch(error => {
              return console.error('hashing failed:', error);
          });
  }
  })

2. Lambda already allows custom hash as of v1.100.0 but not DockerImageAsset so you'd need

interface CustomProps extends ecr.DockerImageAssetProps {
    assetHashType: AssetHashType; // AssetHashType.CUSTOM see https://github.com/aws/aws-cdk/pull/12984
    assetHash: string; // must be specified
}

class CustomDockerImage extends ecr.DockerImageAsset{
    private props: CustomProps
    constructor(scope: Construct, id: string, props: CustomProps){
        super(scope, id, props);
        this.props = props;
    }
}

so that you can declare the docker image as

new CustomDockerImage(scope, 'Image', {
    directory: dockerDir,
    ...
    assetHash: dockerHash,
    assetHashType: AssetHashType.CUSTOM
})

then you can use snapshot test across os platforms. This is a bit of a hack but I wonder if you can build something similar to the core library for this?

[jogold]

@eladb do you know why the file size is included in the hash?

[peterwoodworth]

@eladb I'd like to hear your take on this

[eladb]

I am not sure why we need the file size is included in the hash and I generally agree with @TRANTANKHOA that the hashing algorithm should normalize the inputs across OSs (for example, subdirectory paths come to mind as another thing that might be different, CRLF, etc).

We should revisit the hashing algorithm and remove/normalize any potential OS-specific elements.

@jogold is this something you'd be interested to look at?

[jogold]

@jogold is this something you'd be interested to look at?

I will have a look at this, yes.

Changing the hash algorithm will reupload all assets to S3/ECR and update all stacks. Not really breaking for S3 assets but potentially breaking for ECR/Docker assets that have FROM instructions that are not pinned to a specific version. Is this for v1?

[Hi-Fi]

Is this going to be addressed with V2 CDK? We are facing same issue with NodeJS CDK.

• Mac and Linux generate same asset IDs
• With Windows, Docker (with Windows cloned repo), Docker (with Docker cloned repo) and Docker (with Docker cloned repo to directory that's not mounted from Windows) each generate different ID than *nix (and also different with each other).

In latter cases npm install is made in Docker as Projen is not working with Windows.

[ayozemr]

Subscribed, interested on when this is addressed. Thanks for the work.

[peterwoodworth]

Hey @jogold, I'm curious if you've made any progress on this or if you've hit any roadblocks

[jogold]

Hi @peterwoodworth, not yet. I should have time to address this before the end of August.

[peterwoodworth]

@jogold should this issue be closed?

[jogold]

@jogold should this issue be closed?

Yes, I think so.

@TRANTANKHOA feel free to reopen if v1.121.0 doesn't solve your issue.

[jmhossler]

I have a similar issue, except I'm seeing different asset hashes on arch linux and ubuntu. I've updated to v1.121.0 but I am still seeing this issue, we do snapshot tests so I'm having to manually update the snapshots to get it working.

I've compared with a coworker, and the lambda resource file sizes are all the same. Any recommendations of other things to check to understand why the asset hashes are different between arch linux and ubuntu?

[jogold]

@jmhossler can you share a minimal repro? Are you pulling the same files from a git repo and getting different hashes?

[jmhossler]

@jogold I'll try to do that this evening, this is currently happening in a private repo that I can't share. It's a bit hard for me because I don't have an ubuntu machine to validate it with, but if I can try to make a sample with a git runner to replicate the issue.

The files are in the git repo, yes, but I'm not pulling them down. The lambda files are in the same git repo as the CDK application.

[alfianabdi]

I have similar issue, between Ubuntu 20.04 and AmazonLinux 2. I am using cdk 1.122.0 (build ae09c16) with node 14.17.1.
I tried to clean up and clone the repo in both Ubuntu and AmazonLinux2 then install dependecies again.
Both introduce new change in assets.
Old hash: asset.f35a2260b4975da
New hash in Ubuntu: asset.d3221d8a94463af
New hash in Amazon Linux: asset.0f326c9258aa679

[jogold]

@alfianabdi what kind of asset? can you share some code?

[alfianabdi]

The assets for lambda function, I am able to reproduce with this sample
https://github.com/alfianabdi/cdk-lambda.git

[mlippens]

We have the same issue with function resources NodejsFunction and PythonFunction.

[ayozemr]

I think I am seeing this also in DockerImageFunction. Doing diff in a mac throws asset change vs deployed with CI that uses ubuntu, when no code change is present. Code is a Java lambda

[jogold]

The assets for lambda function, I am able to reproduce with this sample alfianabdi/cdk-lambda.git

@alfianabdi I cannot reproduce this.

I'm doing the following both on Ubuntu and Amazon Linux (in Docker containers):

$ git clone https://github.com/alfianabdi/cdk-lambda.git
$ cd cdk-lambda/
$ npm i
$ cd functions/
$ npm i
$ cd ..
$ npm run cdk synth
$ ls cdk.out/

In both cases I'm getting asset.63c890dcd34187d75a5a75d0b691f96674cafe096cc9fe10f39168ff790a703d.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.