# COSMO [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/bringing-back-the-context-camera-trap-species/image-classification-on-iwildcam2020-wilds)](https://paperswithcode.com/sota/image-classification-on-iwildcam2020-wilds?p=bringing-back-the-context-camera-trap-species) ## [CIKM'24] Reviving the Context: Camera Trap Species Classification as Link Prediction on Multimodal Knowledge Graphs **Paper**: https://arxiv.org/pdf/2401.00608.pdf **Project webpage**: https://osu-nlp-group.github.io/COSMO/ **Authors**: Vardaan Pahuja, Weidi Luo, Yu Gu, Cheng-Hao Tu, Hong-You Chen, Tanya Berger-Wolf, Charles Stewart, Song Gao, Wei-Lun Chao, and Yu Su ## Installation ``` pip install -r requirements.txt ``` ## Data Preprocessing ### iWildCam2020-WILDS ``` bash preprocess_iwildcam.sh ``` Note: The dir. `data/iwildcam_v2.0/train/` contains images for all splits. ### Snapshot Mountain Zebra 1. Download snapshot_mountain_zebra.zip from [this link](https://buckeyemailosu-my.sharepoint.com/:u:/g/personal/pahuja_9_buckeyemail_osu_edu/EWI05mXQsopNskBo78a_l_ABSZJHl0uCsdNMu72aXmNNiA?e=LOtm5Q) and uncompress it into a directory `data/snapshot_mountain_zebra/`. 2. Download images using the command `gsutil -m cp -r "gs://public-datasets-lila/snapshot-safari/MTZ/MTZ_public" data/snapshot_mountain_zebra/` 2. Run `bash preprocess_mountain_zebra.sh` ## Training Note: The below commands will use the DistMult model by default. Use the following hyperparameter configuration: - For iWildCam2020-WILDS, set `DATA_DIR` to `data/iwildcam_v2.0/`, `IMG_DIR` to `data/iwildcam_v2.0/train/`, and `DATASET` to `iwildcam` - For Snapshot Mountain Zebra, set `DATA_DIR` to `data/snapshot_mountain_zebra/` and `IMG_DIR` to `data/snapshot_mountain_zebra/`, and `DATASET` to `mountain_zebra`. - For ConvE, use `--kg-embed-model conve --embedding-dim 200` in args. ### Image-only model (ERM baseline) ``` python -u run_image_only_model.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR > CKPT_DIR/log.txt ``` ### COSMO, no-context baseline ``` python -u main.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR > CKPT_DIR/log.txt ``` ### COSMO, taxonomy ``` python -u main.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR --add-id-id > CKPT_DIR/log.txt ``` ### COSMO, location ``` python -u main.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR --add-image-location > CKPT_DIR/log.txt ``` ### COSMO, time ``` python -u main.py --data-dir DATA_DIR --img-dir IMG_DIR/ --save-dir CKPT_DIR --add-image-time > CKPT_DIR/log.txt ``` ### COSMO, taxonomy + location + time ``` python -u main.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR --add-id-id --add-image-time --add-image-location > CKPT_DIR/log.txt ``` ### MLP-concat baseline ``` python -u run_kge_model_baseline.py --data-dir DATA_DIR --img-dir IMG_DIR --save-dir CKPT_DIR --embedding-dim 512 --use-subtree --only-hour --time_input_dim 1 --early-stopping-patience 10 > CKPT_DIR/log.txt ``` ## Evaluation ### Evaluate a model (specify split) ``` python eval.py --ckpt-path <PATH TO COSMO CKPT> --split test --data-dir DATA_DIR --img-dir IMG_DIR ``` ## Error Analysis ### Taxonomy analysis ``` cd gen_utils/ python analyze_taxonomy_model.py --data-dir DATA_DIR --img-dir IMG_DIR --ckpt-1-path <PATH TO COSMO+TAXONOMY CKPT> --ckpt-2-path <PATH TO COSMO BASE CKPT> ``` ### Plot location correlation analysis ``` cd gen_utils/ python analyze_img_loc.py --data-dir DATA_DIR ``` ### Plot time correlation analysis ``` cd gen_utils/ python analyze_img_time.py --data-dir DATA_DIR ``` ### Under-represented Species Analysis #### Dump predictions for baseline image-only model ``` cd gen_utils/ python dump_imageonly_pred_specie_wise.py --ckpt-path <PATH TO IMAGE-ONLY MODEL> --split test --out-dir <OUT DIR> ``` #### Dump predictions for COSMO model ``` cd gen_utils/ python dump_kge_pred_specie_wise.py --ckpt-path <PATH TO COSMO MODEL> --split test --out-dir <OUT DIR> ``` #### Compare performance for under-represented species ``` cd gen_utils/ python eval_kge_specie_wise.py --y-pred-path-1 <PATH TO PREDICTIONS JSON FILE OF BASELINE MODEL> --y-pred-path-2 <PATH TO COSMO PREDICTIONS JSON FILE> ``` ## Citation ``` @inproceedings{10.1145/3627673.3679545, author = {Pahuja, Vardaan and Luo, Weidi and Gu, Yu and Tu, Cheng-Hao and Chen, Hong-You and Berger-Wolf, Tanya and Stewart, Charles and Gao, Song and Chao, Wei-Lun and Su, Yu}, title = {Reviving the Context: Camera Trap Species Classification as Link Prediction on Multimodal Knowledge Graphs}, year = {2024}, isbn = {9798400704369}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi-org.proxy.lib.ohio-state.edu/10.1145/3627673.3679545}, doi = {10.1145/3627673.3679545}, booktitle = {Proceedings of the 33rd ACM International Conference on Information and Knowledge Management}, pages = {1825–1835}, numpages = {11}, keywords = {KG link prediction, camera traps, multimodal knowledge graph, species classification}, location = {Boise, ID, USA}, series = {CIKM '24} } ```