Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🔨 Fix docker dav1d shared library linking #408

Merged
merged 7 commits into from
Aug 21, 2024

Conversation

wolffshots
Copy link
Contributor

Adding explicit linking to the process appears to fix the error that I was experiencing in Docker builds that I mentioned in #406:

stump  | /app/stump: error while loading shared libraries: libdav1d.so.7: cannot open shared object file: No such file or directory
stump exited with code 0

The Dockerfile still does not build locally despite adding a step to generate prisma.rs because of this error but that seems to be a separate issue:

                                                      ^^^^^^^^^^^^ help: a static with a similar name exists: `DATAMODEL_STR`
175.6
175.6 error[E0425]: cannot find value `MIGRATIONS_DIR` in module `super`
175.6  --> core/src/prisma.rs:3:3446816
175.6   |
175.6 3 | ...igrate_deploy (super :: DATAMODEL_STR , super :: MIGRATIONS_DIR , & self . 0 . url ()) . await ; :: prisma_client_rust :: tokio :: tim...
175.6   |                                                     ^^^^^^^^^^^^^^ not found in `super`
175.6
175.6 error[E0425]: cannot find value `MIGRATIONS_DIR` in module `super`
175.6  --> core/src/prisma.rs:3:3447215
175.6   |
175.6 3 | ...e (migration , super :: DATAMODEL_STR , super :: MIGRATIONS_DIR , & self . 0 . url () ,) . await } pub fn _db_push (& self) -> :: pris...
175.6   |                                                     ^^^^^^^^^^^^^^ not found in `super`
175.6
175.6 For more information about this error, try `rustc --explain E0425`.
175.6 error: could not compile `stump_core` (lib) due to 5 previous errors

@@ -2,7 +2,7 @@
# Frontend Build Stage
# ------------------------------------------------------------------------------

FROM node:20.0.0-alpine3.16 as frontend
FROM node:20.0.0-alpine3.16 AS frontend
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just to appease the linting

Comment on lines 54 to 59
ninja install; \
ln -s /usr/local/lib/x86_64-linux-gnu/libdav1d.so.7.0.0 /usr/local/lib/libdav1d.so.7; \
ln -s /usr/local/lib/x86_64-linux-gnu/libdav1d.so.7.0.0 /usr/local/lib/libdav1d.so.7.0; \
ln -s /usr/local/lib/x86_64-linux-gnu/libdav1d.so.7.0.0 /usr/local/lib/libdav1d.so.7.0.0; \
echo "/usr/local/lib" >> /etc/ld.so.conf.d/mylibs.conf \
&& ldconfig
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could probably be done a lot more elegantly and generically but this seems to work so I'd be interested to see how I could do it better

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I won't have time to test this until after work, but this won't account for arm64 builds

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I'm happy to look at it again tomorrow to work out how to make it more generic (like using TARGETARCH or something). Defs just a hack for now to get it to work for me
If you have an idea of how to improve I'll get to it in the morning but it's late over here so gonna call it for this attempt

@wolffshots wolffshots changed the title 🔨 Fix docker dav1d shared library linking Draft: 🔨 Fix docker dav1d shared library linking Aug 20, 2024
Comment on lines +66 to +67
RUN cargo prisma generate --schema ./core/prisma/schema.prisma; \
./scripts/release/utils.sh -w; \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change allows me to build locally for testing but I'm happy to revert if needing

@wolffshots
Copy link
Contributor Author

wolffshots commented Aug 20, 2024

I had to overwrite the entrypoint on my docker compose to get this to successfully run: entrypoint: ["/bin/sh", "-c", "echo '/usr/local/lib' >> /etc/ld.so.conf.d/mylibs.conf && ldconfig && /entrypoint.sh"] but I did get it to work with this branch!
Just gonna try with a modified entrypoint.sh quickly before I call it for the night

@wolffshots
Copy link
Contributor Author

wolffshots commented Aug 20, 2024

The change that I just pushed allowed me to just run the image without changing the entrypoint in my compose.yml and it appears to be up and running a-okay now 🥳

@wolffshots wolffshots changed the title Draft: 🔨 Fix docker dav1d shared library linking 🔨 Fix docker dav1d shared library linking Aug 20, 2024
set -ex; \
./scripts/release/utils.sh -p; \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that I actually had to remove the prisma.rs text replacements because it was actually generated in the Dockerfile above so was already correct

Comment on lines 101 to 102
COPY --from=builder /usr/local/lib /usr/local/lib
COPY --from=builder /etc/ld.so.conf.d/mylibs.conf /etc/ld.so.conf.d/mylibs.conf
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

COPY --from=builder /usr/local/lib /usr/local/lib <- this is necessary
COPY --from=builder /etc/ld.so.conf.d/mylibs.conf  /etc/ld.so.conf.d/mylibs.conf <- this can probs be removed because I do it in `entrypoint.sh` now

@aaronleopold
Copy link
Collaborator

@wolffshots I made a few changes, since nightly is broken I wanted to make sure at the very least a working image goes out sooner rather than later. I appreciate your fixes! I translated them slightly to work for the arm64 build and then tweaked the ldconfig to the final layer in order to omit having to add it in the entrypoint exec. I left your change to generate the prisma client in the docker file, mostly because I am curious how it performs. If it winds up having to download the engines each time, I'll likely revert it. Best case scenario, though, it makes it easier for folks like you to build the images locally without needing a rust environment setup 🙂

After some discussions on Discord, once this fix is merged I will likely revert the AVIF-specific additions until another time when a more batteries-included solution can be put together.

@aaronleopold aaronleopold merged commit a8ef2a4 into stumpapp:develop Aug 21, 2024
6 checks passed
@wolffshots
Copy link
Contributor Author

Great stuff! Glad to have been able to help and thanks for all the hard work

@aaronleopold
Copy link
Collaborator

aaronleopold commented Aug 21, 2024

I left your change to generate the prisma client in the docker file, mostly because I am curious how it performs. If it winds up having to download the engines each time, I'll likely revert it. Best case scenario, though, it makes it easier for folks like you to build the images locally without needing a rust environment setup 🙂

I had to remove it, with prisma generation it stalled after 4+ hours on the arm64 build. The build without it completed both amd64 and arm64 in just under 2 😅

But I added some parameters you can use to generate it in the docker build. You can do something like:

PLATFORMS='linux/amd64' TAG='local' RUN_PRISMA_GENERATE=true ./docker/build.sh

@wolffshots
Copy link
Contributor Author

I had to remove it, with prisma generation it stalled after 4+ hours on the arm64 build. The build without it completed both amd64 and arm64 in just under 2 😅

Fair that makes sense

But I added some parameters you can use to generate it in the docker build. You can do something like:

PLATFORMS='linux/amd64' TAG='local' RUN_PRISMA_GENERATE=true ./docker/build.sh

Great stuff, that's all good by me

@wolffshots wolffshots deleted the fix/docker-with-dav1d branch August 21, 2024 16:36
@aaronleopold aaronleopold mentioned this pull request Sep 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants