EDIS (Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learning)

This repo contains the code of Energy-guided DIffusion Sampling (EDIS) algorithm, proposed by Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learning.

EDIS utilizes a diffusion model to extract prior knowledge from the offline dataset and employs energy functions to distill this knowledge for enhanced data generation in the online phase. The generated samples confirm online fine-tuning distribution without oblivion of transition fidelity.

Getting started

Install MuJoCo.

• Download MuJoCo Key and MuJoCo 2.1 binaries
• Extract the downloaded mujoco210 and mjkey.txt into ~/.mujoco/mujoco210 and ~/.mujoco/mjkey.txt

Add the following environment variables into ~/.bashrc

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/.mujoco/mujoco210/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/nvidia

Install Anaconda environment.

To install the necessary environments, we use

conda create -n edis python=3.8
conda activate edis
pip install -r requirements/requirements_dev.txt

Run the code.

To run the Cal-ql-EDIS or IQL-EDIS, use the command

python -u algorithms/iql_edis.py --env hopper-random-v2 --state_guide --policy_guide --transition_guide --seed 48

or

python -u algorithms/cal_ql_edis.py --env hopper-random-v2 --state_guide --policy_guide --transition_guide --seed 48

Built upon CORL

Our EDIS is built upon CORL, please refer to https://github.com/tinkoff-ai/CORL