Webpack broken on docker

[mekarpeles]

@tabshaikh can you please provide screenshots here of the error you're hitting so we can help you debug?

@cdrini I'm assigning to you as deputy of docker / dev; feel free to ask on #openlibrary-g if anyone can help debug and re-assign
cc: @hornc. Our job is to make sure this issue gets assigned to someone (if not @cdrini).

[mekarpeles]

@Batey96 is there any chance you would be willing to help us + @tabshaikh with this docker + webpack issue and see if you can reproduce?

The request is to build the latest Open Library docker (https://github.com/internetarchive/openlibrary/tree/master/docker), see if you hit any errors, and if so, document it here (with your summary or a screenshot)

This would be a huge help!

[hornc]

@tabshaikh @mekarpeles I suspect this may be caused by running make js as the incorrect user -- it should be run as openlibrary rather than root. I am trying to confirm and will provide the correct command here.

[hornc]

I have been able to reproduce this -- the images build correctly and uses webpack, but trying to run make js manually in the running container with docker-compose exec web make js fails with

mkdir -p static/build
bash static/js/vendor.jsh > static/build/vendor-v2.js
npm run build-assets:webpack

> openlibrary@1.0.0 build-assets:webpack /openlibrary
> NODE_ENV=production webpack --mode=production

sh: 1: webpack: not found
npm ERR! file sh
npm ERR! code ELIFECYCLE
npm ERR! errno ENOENT
npm ERR! syscall spawn
npm ERR! openlibrary@1.0.0 build-assets:webpack: `NODE_ENV=production webpack --mode=production`
npm ERR! spawn ENOENT
npm ERR! 
npm ERR! Failed at the openlibrary@1.0.0 build-assets:webpack script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2019-04-08T00_10_35_894Z-debug.log
Makefile:33: recipe for target 'js' failed
make: *** [js] Error 1

regardless of running as root or openlibrary.

I think the problem is that we are using a persistent volume to store node_modules, and this is not being updated when olbase is being updated. The reason for having this volume in the first place place was to allow the local dev /openlibrary source to be mounted in the container to have the code being worked on, but the node_modules sub-directory would have the correct node modules as installed by the image.

It looks like this volume isn't getting updated when the image is rebuilt. I'm not yet sure of the best way to fix this and have a working node_modules subdir that updates but doesn't get clobbered by mounting the dev dir. If any one has any ideas, please let me know.

The workaround will be to manually delete the ol-nodemodules volume before rebuilding the images.

Delete the volume with:
docker volume rm openlibrary_ol-nodemodules

There should be a better way to handle this automatically if the package.json file updates in future though. @tabshaikh, can you please let me know whether this workaround unblocks you?

[hornc]

UPDATE You shouldn't need to rebuild the images:
docker volume rm openlibrary_ol-nodemodules
then
docker-compose up
should have the same effect since the correct node modules are already in the images.

[hornc]

found by @cdrini : https://jdlm.info/articles/2016/03/06/lessons-building-node-app-docker.html a guide on how this problem was handled in 2016 on another project. They use the same kind of volume mount trick, but using non-named / non-persistent volumes. I will investigate whether moving back to non-named volumes will guarantee they are created fresh every docker-compose up with the correct and latest node_modules from the base image.

The article has a very good description of the node_modules problem in the section "The node_modules Volume Trick" and how mounting the local dev dir overwrites the installed image modules with the empty host node_modules. The blue / red filenames is a great way to illustrate a problem (I have found) difficult to describe clearly.

[supercontracts]

@hornc Thanks, the commands work perfectly