Skip to content

PPPL cluster

Jump to bottom
Bharat Medasani edited this page Dec 1, 2021 · 22 revisions

This Wiki page documents the steps to install VMEC2000 and simsopt on PPPL cluster using virtual environments from conda and python venv packages. The approaches shown here are tested, but not the only approaches to install simsopt and VMEC packages.

Conda

VMEC

  1. Download the package 
    git clone https://github.com/hiddenSymmetries/VMEC2000.git
cd VMEC2000

Conda virtual environment

  1. Load conda module 
    module load anaconda3
  2. Create a conda virtual environment with python version 3.9. Let us call it sims_gcc_py39 indicating that we are using python 3.9 and gcc compiler. 
    conda create -n sims_gcc_py39 python=3.9
  3. Activate the newly created virtual environment 
    conda activate sims_gcc_py39

Dependencies

Intel Compiler

NOTE

Simsopt does not compile with intel compilers on PPPL cluster. Use Intel compiler for VMEC only if you intend to use VMEC without simsopt.

  1. Load all the required compilers and libraries by running 
    module load intel openmpi szip netcdf-c netcdf-fortran
    The above command also load Intel MKL, which can be found running echo $MKLROOT. This should show MKLROOT as /usr/pppl/intel/2019.u3/compilers_and_libraries/linux/mkl/
  2. We use MKL supplied with a different version of intel compiler to use scalapack and blacs supplied with MKL. 
    export MKLROOT=/usr/pppl/intel/2020.u1/compilers_and_libraries_2020/linux/mkl
  3. Copy the relevant cmake config file from cmake/machines folder. 
    cp cmake/machines/pppl_intel.json cmake_config_file.json

GCC compilers

  1. Load all the required compilers and libraries by running 
    module load gcc openmpi blacs scalapack szip netcdf-c netcdf-fortran
    When compared to the corresponding step associated with intel compiler, we are loading default scalapack and blacs libraries from netlib.org compiled with gcc. scalapack and blacs require linear algebra math libraries.
  2. We use Intel MKL supplied with a 2019 version of intel compiler for the math libraries. But we don't have to load intel compilers. We only need to populate the MKLROOT environment variable. 
    export MKLROOT=/usr/pppl/intel/2019.u3/mkl
  3. Copy the relevant cmake config file from cmake/machines folder. 
    cp cmake/machines/pppl_gcc.json cmake_config_file.json

Compile, and test

  1. First, install python dependencies

    pip install --no-cache-dir mpi4py
pip install setuptools wheel scikit-build cmake  ninja numpy f90wrap

  2. Build the python wheel

    python setup.py bdist_wheel

  3. Install the built wheel

    pip install dist/*.whl

  4. Test the installation by loading the installed vmec package

    python -c "import vmec; print('Success')"

    If vmec is successfully installed, the above command should print Success.

SIMSOPT

For installing simsopt, follow all the steps related to loading modules and activating the conda virtual environment if not done already. We are going to use gcc to compile simsopt.

  1. If still in VMEC folder, move out of the VMEC folder and download the simsopt package 
    git clone https://github.com/hiddenSymmetries/simsopt.git
cd simsopt
  2. Instead of using pip to compile and install simsopt in one step, we use two steps for conda virtual environemnts. First install all the required build dependencies specified in pyproject.toml manually. 
    pip install "setuptools>=45" wheel numpy cmake ninja "setuptools_scm[toml]>=6.0"
  3. Then build the simsopt wheel. If simsopt wheels are built before, delete them by deleting the build and dist folders. Before building the wheel, set the environment variable CI to True. This reduces the optimization level because PPPL cluster has heterogenous hardware and we want the compiled code to work on all nodes. 
    export CI=True
python setup.py bdist_wheel
  4. The wheel built with the above command resides in dist folder. Make sure there is only one file in dist folder. However it is not a strict requirement. Install the newly built wheel by running 
    pip install dist/simsopt*.whl
    This command assumes there is only one file with whl extension in the dist folder. If there are multiple whl files, specify the full name of the whl file.

Python venv

Here we use the virtual environment manager venv supplied with python. One drawback of this approach is you can not choose a different version of python other than the installed python. Luckily, different versions of python are available via python modules. For this exercise, we are using python v3.9.4 installed on PPPL cluster and gcc compilers. Python v3.8 and v3.7 are also available on the cluster.

Python Virtual Environment

  1. Load the required python module.

    module load python/3.9.4

    NOTE

    python refers to system python. When the python module is loaded, only python3 is activated.

  2. Create a virtual environment. Lets call it sims_gcc_py39. We create it under ~/venv folder, where ~ refers to your home directory.

    python3 -m venv ~/venv/sims_gcc_py39

  3. Activate the virtual environment

    . ~/venv/sims_gcc_py39/bin/activate

    Now, which python or which python3 should both refer to ~/venv/sims_gcc_py39/bin/ folder.

  4. Upgrade the pip module

    python -m pip install --upgrade pip

    If which pip points to ~/venv/sims_gcc_py39/bin/, you could use pip command directly. Otherwise, use pip using the command python -m pip. All the steps below use python -m pip ... for reliability.

VMEC

  1. Download the package

    git clone https://github.com/hiddenSymmetries/VMEC2000.git
cd VMEC2000

  2. Load all the required compilers and libraries by running

    module load gcc openmpi blacs scalapack szip netcdf-c netcdf-fortran

    We are loading default scalapack and blacs libraries from netlib.org compiled with gcc. scalapack and blacs require linear algebra math libraries.

  3. We use Intel MKL supplied with a 2019 version of intel compiler for the math libraries. But we don't have to load intel compiler module. We only need to populate the MKLROOT environment variable.

    export MKLROOT=/usr/pppl/intel/2019.u3/mkl

  4. Copy the relevant cmake config file from cmake/machines folder.

    cp cmake/machines/pppl_gcc.json cmake_config_file.json

  5. Install mpi4py package manually with fresh compilation.

    python -m pip install --no-cache-dir mpi4py

  6. Here we use only one command to install vmec python extension.

    python -m pip install .

  7. Test the installation by loading the installed vmec package

    python -c "import vmec; print('Success')"

    If vmec is successfully installed, the above command should print Success.

SIMSOPT

  1. If still in VMEC folder, move out of the VMEC folder and download the simsopt package 
    cd ..
git clone https://github.com/hiddenSymmetries/simsopt.git
cd simsopt
  2. Again we use pip to compile and install simsopt in one step. 
    export CI=True
python -m pip install .
  3. Test the installation 
    python -c "import simsopt; print('Success')"
    If simsopt is successfully installed, the above command should print Success.
Clone this wiki locally