Locate n' Rotate: Two-stage Openable Part Detection with Foundation Model Priors

accepted by ACCV 2024(oral)

conda create -n opd python=3.9
conda activate opd
pip install -r requirements.txt

Dataset

Download OPDMulti dataset (7.2G) and extract it inside ./dataset/ folder. Make sure the data is in this format. You can follow these steps if you want to convert your data to OPDMulti dataset. To try our model on OPDSynth and OPDReal datasets, download the data from OPD repository.

Training

To train from the scratch, you can use the below commands. The output will include evaluation results on the val set.

cd opdformer
python train.py \
--config-file <MODEL_CONFIG> \
--output-dir <OUTPUT_DIR> \
--data-path <PATH_TO_DATASET> \
--input-format <RGB/depth/RGBD> \
--model_attr_path <PATH_TO_ATTR> 

• <MODEL_CONFIG>: the config file path for different model variants can be found in the table OPDMulti "Model Name" column.

• Dataset:

  • --data-path OPDMulti/MotionDataset_h5
  • --model_attr_path: OPDMulti/obj_info.json

• You can add the following command to use the model weights, pretrained on OPDReal dataset. We finetune this model on OPDMulti dataset:

  --opts MODEL.WEIGHTS <PPRETRAINED_MODEL>

Evaluation

To evaluate, use the following command:

python evaluate_on_log.py \
--config-file <MODEL_CONFIG> \
--output-dir <OUTPUT_DIR> \
--data-path <PATH_TO_DATASET> \
--input-format <RGB/depth/RGBD> \
--model_attr_path <PATH_TO_ATTR> \
--opts MODEL.WEIGHTS <PPRETRAINED_MODEL>

• Evaluate on test set: --opts MODEL.WEIGHTS <PPRETRAINED_MODEL> DATASETS.TEST "('MotionNet_test',)".
• To evaluate directly on pre-saved inference file, pass the file path as an argument --inference-file <PATH_TO_INFERENCE_FILE>.

Pretrained-Models

https://pan.baidu.com/s/10mY2fRmE3ms3dFzBGrzpUw?pwd=478x Extract code：478x

Visualization

The visualization code is based on OPD repository. We only support visualization based on raw dataset format (download link (5.0G)).

And the visualization uses the inference file, which can be obtained after the evaluation.

• Visualize the GT with 1000 random images in val set

  cd opdformer
  python render_gt.py \
  --output-dir vis_output \
  --data-path <PATH_TO_DATASET> \
  --valid-image <IMAGE_LIST_FILE> \
  --is-real

• Visualize the PREDICTION with 1000 random images in val set

  cd opdformer
  python render_pred.py \
  --output-dir vis_output \
  --data-path <PATH_TO_DATASET> \
  --model_attr_path <PATH_TO_ATTR> \
  --valid-image <IMAGE_LIST_FILE> \
  --inference-file <PATH_TO_INFERENCE_FILE> \
  --score-threshold 0.8 \
  --update-all \
  --is-real

  • --data-path dataset/MotionDataset
  • --valid_image dataset/MotionDataset/valid_1000.json