Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support QE conda env in /opt/conda #784

Merged
merged 6 commits into from
Jul 23, 2024
+24 −22
Merged

Support QE conda env in /opt/conda #784

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4c44c4b
Move QE conda install into separate stage
danielhollas Jul 22, 2024
1b74658
Move after pseudos
danielhollas Jul 22, 2024
d2a75e4
Max cache mode
danielhollas Jul 22, 2024
983bd07
Support QE conda env in /opt/conda
danielhollas Jul 22, 2024
29bfefa
Merge branch 'main' into qe-env-dir
danielhollas Jul 23, 2024
7afc5f7
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .dockerignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
**/*.egg-info
**/__pycache__
**/.*_cache
**/*.pyc
**/*.tar.gz
*.code-workspace
**/.*.ipynb
**/.ipynb*
.venv/
build/
export/
.do-not-setup-on-localhost

# Sphinx documentation
docs/html
screenshots/
2 changes: 1 addition & 1 deletion .github/workflows/docker-build-test-upload.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ jobs:
context: .
platforms: linux/amd64
cache-to: |
type=gha,scope=${{ github.workflow }},mode=min
type=gha,scope=${{ github.workflow }},mode=max
cache-from: |
type=gha,scope=${{ github.workflow }}

Expand Down
47 changes: 26 additions & 21 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,26 @@
# syntax=docker/dockerfile:1
FROM ghcr.io/astral-sh/uv:0.2.18 AS uv
FROM ghcr.io/aiidalab/full-stack:2024.1019
ARG FULL_STACK_VER=2024.1021
ARG UV_VER=0.2.27
ARG QE_VER=7.2
ARG QE_DIR=/opt/conda/envs/quantum-espresso-${QE_VER}

FROM ghcr.io/astral-sh/uv:0.2.27 AS uv

# STAGE 1
# Install QE into conda environment in /opt/conda
# This step is independent from the others and can be run in parallel.
FROM ghcr.io/aiidalab/full-stack:${FULL_STACK_VER} AS qe_conda_env
ARG QE_VER
ARG QE_DIR
RUN mamba create -p "${QE_DIR}" --yes qe="${QE_VER}" && \
mamba clean --all -f -y && \
fix-permissions "${CONDA_DIR}"


# STAGE 2
FROM ghcr.io/aiidalab/full-stack:${FULL_STACK_VER}
ARG QE_VER
ARG QE_DIR
# Copy whole repo and pre-install the dependencies and app to the tmp folder.
# In the before notebook scripts the app will be re-installed by moving it to the app folder.
ENV PREINSTALL_APP_FOLDER=${CONDA_DIR}/aiidalab-qe
Expand All @@ -17,32 +36,18 @@ RUN --mount=from=uv,source=/uv,target=/bin/uv \
cd ${PREINSTALL_APP_FOLDER} && \
# Remove all untracked files and directories. For example the setup lock flag file.
git clean -fx && \
# It is important to install from `aiidalab install` to mimic the exact installation operation as
# from the app store.
# The command wil first install the dependencies from list by parsing setup config files,
# (for `aiidalab/aiidalab<23.03.2` the `setup.py` should be in the root folder of the app https://github.com/aiidalab/aiidalab/pull/382).
# and then the app and restart the daemon in the end.
# But since the aiida profile not yet exists, the daemon restart will fail but it is not a problem.
# Because we only need the dependencies to be installed.
# aiidalab install --yes --python ${CONDA_DIR}/bin/python "quantum-espresso@file://${PREINSTALL_APP_FOLDER}" && \
# However, have to use `pip install` explicitly because `aiidalab install` call `pip install --user` which will install the app to `/home/${NB_USER}/.local`.
# It won't cause issue for docker but for k8s deployment the home folder is not bind mounted to the pod and the dependencies won't be found. (see issue in `jupyter/docker-stacks` https://github.com/jupyter/docker-stacks/issues/815)
uv pip install --system --no-cache . && \
fix-permissions "${CONDA_DIR}" && \
fix-permissions "/home/${NB_USER}"

ENV QE_VERSION="7.2"
RUN mamba create -p /opt/conda/envs/quantum-espresso --yes \
qe=${QE_VERSION} \
&& mamba clean --all -f -y && \
fix-permissions "${CONDA_DIR}" && \
fix-permissions "/home/${NB_USER}"
fix-permissions "${CONDA_DIR}"

# Download the QE pseudopotentials to the folder for afterware installation.
ENV PSEUDO_FOLDER=${CONDA_DIR}/pseudo
RUN mkdir -p ${PSEUDO_FOLDER} && \
python -m aiidalab_qe download-pseudos --dest ${PSEUDO_FOLDER}

COPY --from=qe_conda_env "${QE_DIR}" "${QE_DIR}"
# TODO: Remove this once we get rid of 70_prepare-qe-executable.sh
ENV QE_VERSION="$QE_VER"

COPY before-notebook.d/* /usr/local/bin/before-notebook.d/

WORKDIR "/home/${NB_USER}"
11 changes: 3 additions & 8 deletions before-notebook.d/70_prepare-qe-executable.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,9 @@
# Debugging.
set -x

# Copy quantum espresso env to user space.
mkdir -p /home/${NB_USER}/.conda/envs
if [ ! -d /home/${NB_USER}/.conda/envs/quantum-espresso-${QE_VERSION} ]; then
ln -s /opt/conda/envs/quantum-espresso /home/${NB_USER}/.conda/envs/quantum-espresso-${QE_VERSION}

# Install qe so the progress bar not shown in the notebook when first time using app.
echo "Installing qe."
if [[ $(verdi code list -Y localhost -r | wc -l) -eq 0 ]]; then
echo "Installing QE codes..."
python -m aiidalab_qe install-qe
else
echo "Quantum ESPRESSO app is already installed."
echo "Quantum ESPRESSO codes are already installed."
fi
35 changes: 21 additions & 14 deletions src/aiidalab_qe/common/setup_codes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# -*- coding: utf-8 -*-
from pathlib import Path
from shutil import which
from subprocess import CalledProcessError, run
Expand All @@ -21,9 +20,15 @@

QE_VERSION = "7.2"

CONDA_ENV_PREFIX = Path.home().joinpath(
".conda", "envs", f"quantum-espresso-{QE_VERSION}"
)

def get_qe_env():
# QE is already pre-installed in the QE image
path = Path(f"/opt/conda/envs/quantum-espresso-{QE_VERSION}")
if path.exists():
return path
else:
return Path.home().joinpath(".conda", "envs", f"quantum-espresso-{QE_VERSION}")

Comment on lines +24 to +30
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can cause an edge problem that when the container is started with qeapp image and then shifted to using full-stack image. The QE not able to be accessed.
We were trying to provide full-stack, qe-app, base-with-server images in demo server. But as suggested by @giovannipizzi, we agree on only keep qe-app image. This change then wont be a problem.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True. But hopefully that will not happen in practice, and the solution is simply to re-install the app. By the way, in this case the current solution with the symlink would not work anyway since the symlink would be broken, right?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in this case the current solution with the symlink would not work anyway since the symlink would be broken, right?

I think it won't work.


# Add all QE codes with the calcjob entry point in the aiida-quantumespresso.
CODE_NAMES = (
Expand All @@ -46,7 +51,7 @@


def qe_installed():
return CONDA_ENV_PREFIX.exists()
return get_qe_env().exists()


def install_qe():
Expand All @@ -59,7 +64,7 @@ def install_qe():
"--channel",
"conda-forge",
"--prefix",
str(CONDA_ENV_PREFIX),
str(get_qe_env()),
f"qe={QE_VERSION}",
],
capture_output=True,
Expand Down Expand Up @@ -102,9 +107,9 @@ def _generate_string_to_setup_code(code_name, computer_name="localhost"):
except NotExistent:
label = f"{code_name}-{QE_VERSION}"
description = f"{code_name}.x ({QE_VERSION}) setup by AiiDAlab."
filepath_executable = str(CONDA_ENV_PREFIX.joinpath("bin", f"{code_name}.x"))
filepath_executable = get_qe_env().joinpath("bin", f"{code_name}.x")
default_calc_job_plugin = f"quantumespresso.{code_name}"
prepend_text = f'eval "$(conda shell.posix hook)"\\nconda activate {CONDA_ENV_PREFIX}\\nexport OMP_NUM_THREADS=1'
prepend_text = f'eval "$(conda shell.posix hook)"\\nconda activate {get_qe_env()}\\nexport OMP_NUM_THREADS=1'
python_code = """
computer = load_computer('{}')
code = InstalledCode(computer=computer,
Expand All @@ -116,7 +121,7 @@ def _generate_string_to_setup_code(code_name, computer_name="localhost"):
)

code.store()
""".format(
""".format( # noqa: UP032
computer_name,
label,
description,
Expand All @@ -134,7 +139,7 @@ def setup_codes():
try:
run(["python", "-c", python_code], capture_output=True, check=True)
except CalledProcessError as error:
raise RuntimeError(f"Failed to setup codes: {error}")
raise RuntimeError(f"Failed to setup codes: {error}") from None

Copy link
Contributor Author

@danielhollas danielhollas Jul 22, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just to make stack traces nicer. Here's the correspoding flake8-bugbear rule as explained by ruff

❯ ruff rule B904
# raise-without-from-inside-except (B904)

Derived from the **flake8-bugbear** linter.

## What it does
Checks for `raise` statements in exception handlers that lack a `from`
clause.

## Why is this bad?
In Python, `raise` can be used with or without an exception from which the
current exception is derived. This is known as exception chaining. When
printing the stack trace, chained exceptions are displayed in such a way
so as make it easier to trace the exception back to its root cause.

When raising an exception from within an `except` clause, always include a
`from` clause to facilitate exception chaining. If the exception is not
chained, it will be difficult to trace the exception back to its root cause.

Example

try:
    ...
except FileNotFoundError:
    if ...:
        raise RuntimeError("...")
    else:
        raise UserWarning("...")

Use instead:

try:
    ...
except FileNotFoundError as exc:
    if ...:
        raise RuntimeError("...") from None
    else:
        raise UserWarning("...") from exc

References

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, how you find these good thing? Is it from your LSP?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is from ruff, which I have setup as LSP in neovim.

While the pre-commit in this repo uses ruff, it doesn't have any ruff-specific configuration. I think we should copy over the configuration from AWB. I think we have a good experience with that set and it will make things consistent across the two repos.


def install(force=False):
Expand Down Expand Up @@ -175,7 +180,9 @@ def install(force=False):
try:
install_qe()
except CalledProcessError as error:
raise RuntimeError(f"Failed to create conda environment: {error}")
raise RuntimeError(
f"Failed to create conda environment: {error}"
) from None

# After installing QE, we install the corresponding
# AiiDA codes:
Expand All @@ -189,7 +196,7 @@ def install(force=False):
yield "Setting up all codes..."
run(["python", "-c", python_code], capture_output=True, check=True)
except CalledProcessError as error:
raise RuntimeError(f"Failed to setup codes: {error}")
raise RuntimeError(f"Failed to setup codes: {error}") from None

except Timeout:
# Assume that the installation was triggered by a different process.
Expand All @@ -198,7 +205,7 @@ def install(force=False):
if not codes_are_setup():
raise RuntimeError(
"Installation process did not finish in the expected time."
)
) from None


class QESetupWidget(ipw.VBox):
Expand Down Expand Up @@ -311,7 +318,7 @@ def _toggle_error_view(self, change):
@traitlets.observe("busy")
@traitlets.observe("error")
@traitlets.observe("installed")
def _update(self, change):
def _update(self, _change):
with self.hold_trait_notifications():
if self.hide_by_default:
self.layout.visibility = (
Expand Down
Loading