[core] "Full docker build" no longer works

[kellertk]

As detailed in Contributing.md, I should be able to initiate a full build of CDK under Docker with docker build -t aws-cdk .. This does not work because scripts/check-prerequisites.sh attempts to connect to the Docker daemon and run docker ps, which is impossible in a Docker container.

I've reproduced this in Docker on Windows and with Podman on Fedora 32.

Reproduction Steps

$ git clone https://github.com/aws/aws-cdk.git
$ cd aws-cdk
$ docker build -t aws-cdk . 

What did you expect to happen?

CDK gets built.

What actually happened?

[4/4] Building fresh packages...
Done in 143.03s.
git: 'secrets' is not a git command. See 'git --help'.
git-secrets scan ok
Checking if node is installed... Ok
Checking node version... Ok
Checking if yarn is installed... Ok
Checking yarn version... Ok
Checking if javac is installed... Ok
Checking javac version... Ok
Checking if mvn is installed... Ok
Checking mvn version... Ok
Checking if dotnet is installed... Ok
Checking dotnet version... Ok
Checking if python3 is installed... Ok
Checking python3 version... Ok
Checking if ruby is installed... Ok
Checking ruby version... Ok
Checking if docker is installed... Ok
Checking if docker is running... Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
The command '/bin/sh -c ./build.sh ${BUILD_ARGS} && ./link-all.sh' returned a non-zero code: 1

You can't connect to the Docker daemon because we're already in a Docker container.

Environment

• CLI Version : aws-cli/2.0.25 Python/3.7.7 Windows/10 botocore/2.0.0dev29
• Framework Version: d45a57c
• Node.js Version: v12.16.1
• OS : Windows 10, and also Fedora 32
• Language (Version): N/A

Other

I believe this was introduced when the scripts/check-prerequisites.sh script was added in #8929 on July 14, 2020. If I blank out this entire script with exit 0, the build succeeds.

---

This is 🐛 Bug Report

[cyuste]

Makes no sense to check for docker daemon running in the docker build process. With this change this check will only trigger a warning if it fails

[cyuste]

PR fails due to lack of tests but honestly I don't know if that condition applies here

[rix0rrr]

The CDK build actually uses docker as well (I believe for tests), so you do need to have docker-in-docker for it to work.

[cyuste]

I tried to run the full build and failed

yarn-cling: > ln -sf ../../package2 test/test-fixture/package1/node_modules/ && jest
yarn-cling: ln: target ‘test/test-fixture/package1/node_modules/’ is not a directory: No such file or directory
yarn-cling: npm ERR! Test failed.  See above for more details.
yarn-cling: error Command failed with exit code 1.
yarn-cling: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
lerna ERR! yarn run build+test exited 1 in 'yarn-cling'
lerna WARN complete Waiting for 1 child process to exit. CTRL-C to exit immediately.

real	1m56.571s
user	2m15.212s
sys	0m12.392s
❌  Last command failed. Scroll up to see errors in log (search for '!!!!!!!!').
The command '/bin/sh -c ./build.sh ${BUILD_ARGS} && ./link-all.sh' returned a non-zero code: 1

Does not look related to docker, nevertheless I will try to take a look at it in detail when I have some spare time

[cyuste]

Quick update:
As @rix0rrr mentioned, at least @aws-cdk/aws-s3-assets requires docker daemon to run its tests. My bad, I'll restore previous version of requirements.
I think that docker build does not work currently in master branch, so far this is my current situation:

• yarn-cling package failed to build (the error mentioned above): Some files (**/node_modules/package.json) were not copied in the docker image causing this failure. I modified .dockerignore to copy them to the image
• Docker daemon is not running in the jsii/superchain container. I tried to start it manually but got the following error:

WARN[2020-09-21T21:06:56.583985800Z] Running modprobe bridge br_netfilter failed with message: modprobe: WARNING: Module bridge not found in directory /lib/modules/4.19.76-linuxkit
modprobe: WARNING: Module br_netfilter not found in directory /lib/modules/4.19.76-linuxkit
, error: exit status 1 
WARN[2020-09-21T21:06:56.604847500Z] Running iptables --wait -t nat -L -n failed with message: `iptables v1.8.2 (legacy): can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.`, error: exit status 3 
INFO[2020-09-21T21:06:56.651233500Z] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
INFO[2020-09-21T21:06:56.653083100Z] stopping event stream following graceful shutdown  error="context canceled" module=libcontainerd namespace=plugins.moby
INFO[2020-09-21T21:06:56.653083100Z] stopping healthcheck following graceful shutdown  module=libcontainerd
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.2 (legacy): can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.
 (exit status 3)

I'm not a docker expert but this is the first time that I see this docker-in-docker scheme, in my experience when you need docker inside a container you publish the host machine docker unix socket port with a volume, but this is something that can only be done in execution time, not build. I goggled for this issue and it looks like at least this is not something usual.

Anyway, I can continue with this but first I would appreciate a confirmation that docker build is actually broken and I'm not making a big mistake in the docker build process.

[rix0rrr]

Honestly, I don't think we use docker build ourselves. What we ourselves do is "run" the build in the jsii/superchain image (using privileged: true on CodeBuild which gives us access to Docker from within the container), and then spirit the resulting files out another way.

At this point our build does require access to Docker. If docker build makes it hard to set that up, maybe docker --rm running the build with a cp command at the end makes more sense?

[cyuste]

Yeah, I think it makes more sense to use docker only to avoid installing lerna and all the development dependencies but always using the host files (using a volume or whatever is your favourite method).
In my personal experience I haven't been able to finish a full build using docker but neither I need it (I'm happy with partial builds) so I don't know, I think that maybe it would be best to forget this procedure and focus into something more useful, like building using jsii/superchain as you mention and creating a docker image with only the cdk binaries to allow using cdk this way.
I'm happy to help in whatever you decide is best :)

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.