Hongyu Zhou¹, Longzhong Lin¹, Jiabao Wang¹, Yichong Lu¹, Dongfeng Bai², Bingbing Liu², Yue Wang¹, Andreas Geiger^3,4, Yiyi Liao^1,†
1 Zhejiang University ² Huawei ³ University of Tübingen ⁴ Tübingen AI Center
^† Corresponding Authors

This is the official project repository of the paper HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Driving.

---

TODO list

• Release sample data and results
• Release unicycle model part
• Release GUI
• Release more scenarios

Installation

Requirements are not in principle hard requirements, but there might be some differences (not tested):

• Linux
• Python 3.11
• CUDA 11.8
• colmap 3.10-dev
• Pytorch 2.5.1 / Pytorch 2.2.0

Create conda environment needed to run HUGSIM with:

conda create --name hugsim python=3.11

Please install these packages following official instructions

• Pytorch
• simple_waymo_open_dataset_reader
• tinycudann
• unidepth
• flow_vis_torch
• pytorch3d
• kitti360Scripts
• nuscenes-devkit (have to install from the source code for python 3.11)
• HUGSIM_splat

Please install packages required by InverseForm

Install remaining dependencies by:

pip install -r requirements.txt

To run the closed-loop simulation, the hug_sim Gymnasium environment should be installed:

cd ./sim
pip install -e .

Data Preparation

Please refer to Data Preparation Document

You can download sample data from here, which includes a sample sequence, results of the sequence reconstruction, and results of 3dRealCar vehicles reconstruction.

Reconstruction

seq=${seq_name}
input_path=${datadir}/${seq}
output_path=${modeldir}/${seq}
mkdir -p ${output_path}
CUDA_VISIBLE_DEVICES=4 \
python -u train_ground.py --data_cfg ./configs/${dataset_name: [kitti360, waymo, nusc, pandaset]}.yaml \
        --source_path ${input_path} --model_path ${output_path}
CUDA_VISIBLE_DEVICES=4 \
python -u train.py --data_cfg ./configs/${dataset_name}.yaml \
        --source_path ${input_path} --model_path ${output_path}

Simulation

Before simulation, UniAD_SIM, VAD_SIM and NAVSIM client should be installed.

In closed_loop.py, we automatically launch autonomous driving algorithms. In practice, you may encounter errors due to an incorrect environment, path, and etc. For debugging purposes, you can modify the last part of code as:

# process = launch(ad_path, args.ad_cuda, output)
# try:
#     create_gym_env(cfg, output)
#     check_alive(process)
# except Exception as e:
#     print(e)
#     process.kill()

# For debug
create_gym_env(cfg, output)

Paths in configs/sim/*_base.yaml should be updated as paths on your machine.

CUDA_VISIBLE_DEVICES=${sim_cuda} \
python closed_loop.py --scenario_path ${scenario_cfg_path} \
            --base_path ./configs/sim/${dataset_name}_base.yaml \
            --camera_path ./configs/sim/${dataset_name}_camera.yaml \
            --kinematic_path ./configs/sim/kinematic.yaml \
            --ad ${method_name: [uniad, vad, ltf]} \
            --ad_cuda ${ad_cuda}

Run the following commands to execute the provided examples.

sim_cuda=0
ad_cuda=1
scenario_dir=./configs/benchmark/nuscenes
for cfg in ${scenario_dir}/*.yaml; do
    echo ${cfg}
    CUDA_VISIBLE_DEVICES=${sim_cuda} \
    python closed_loop.py --scenario_path ${cfg} \
                        --base_path ./configs/sim/nuscenes_base.yaml \
                        --camera_path ./configs/sim/nuscenes_camera.yaml \
                        --kinematic_path ./configs/sim/kinematic.yaml \
                        --ad uniad \
                        --ad_cuda ${ad_cuda}
done

Citation

If you find our paper and codes useful, please kindly cite us via:

@article{zhou2024hugsim,
  title={HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Driving},
  author={Zhou, Hongyu and Lin, Longzhong and Wang, Jiabao and Lu, Yichong and Bai, Dongfeng and Liu, Bingbing and Wang, Yue and Geiger, Andreas and Liao, Yiyi},
  journal={arXiv preprint arXiv:2412.01718},
  year={2024}
}