🔎 Compression with Global Guidance: Towards Training-free High-Resolution MLLMs Acceleration 🚀

Xuyang Liu¹, Ziming Wang², Yuhang Han³, Yingyao Wang², Jiale Yuan², Jun Song^2✉, Bo Zheng²,
Linfeng Zhang⁴, Siteng Huang⁵, Honggang Chen^1✉

¹Sichuan University, ²Taobao & Tmall Group of Alibaba,
³Northeast Forestry University, ⁴Shanghai Jiaotong University, ⁵Zhejiang University

🔥 News

• 2025.01.10 🤗🤗 We release our latest work GlobalCom², a "global-to-local" approach for training-free acceleration of high-resolution MLLMs. Code is available!
• 2024.11.17 🤗🤗 We release our work FiCoCo which proposes a unified paradigm to demystify the popular works and guide the future designs of training-free token reduction for MLLMs. Code is available!

✨ Overview

TLDR: We present GlobalCom², a novel token compression method for high-resolution MLLMs that uses thumbnail tokens to guide crop compression. Evaluations on 10 benchmarks show that GlobalCom² achieves superior efficiency-performance trade-off with LLaVA-NeXT models.

💥 Core Codes

The two key functions in llava/model/llava_arch.py implement our global-guided local compression: (a) generate_scale_for_crop_features for allocating optimal retention ratios based on each crop's global importance, and (b) interpolate_and_split_cls_attn_scores for performing token compression with importance from the global perspective.

🛠 Preparation

1. Clone this repository.

git clone https://github.com/xuyang-liu16/GlobalCom2.git
cd GlobalCom2

2. Environment Setup and Preparation

 conda create -n GlobalCom2 python=3.10 -y
 conda activate GlobalCom2
 pip install -e .

3. Download Multimodal Benchmark

Please follow the detailed instruction in LLaVA-Evaluation.

4. Download LLaVA-NeXT-7B and LLaVA-NeXT-13B and put them under ./liuhaotian/llava-next-7b and ./liuhaotian/llava-next-13b.

For users with limited access to Hugging Face (e.g., from mainland China), you can refer to this you can refer this alternative guide and use the following command, with LLaVA-NeXT-7B as an example:

pip install -U huggingface_hub hf_transfer -i https://mirrors.aliyun.com/pypi/simple/
export HF_ENDPOINT=https://hf-mirror.com
huggingface-cli download --resume-download liuhaotian/llava-v1.6-vicuna-7b --local-dir ./liuhaotian/llava-next-7b

🚀 Evaluation

👉 The only hyper-parameter is retention_ratio in line 101 of llava/model/llava_arch.py. You can achieve different acceleration effects by setting different retention_ratio values (default retention_ratio = 0.25).

Example for evaluating TextVQA results:

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/textvqa.sh

Example for evaluating MME results:

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mme.sh

To calculate the theoretical computational efficiency shown above, we recommend the methodology presented in the work of LLM-Viewer. We deeply appreciate their outstanding contribution to this field.

🩻 Visualization

To visualize the compression performance shown above, we recommend utilizing the visualization tools provided in tools, which include mask visualization and attention score visualization utilities. We hope these tools will assist in understanding the compression mechanism.

📌 Citation

Please consider citing our paper in your publications, if our findings help your research.

@article{Liu2025:GlobalCom,
    title={Compression with Global Guidance: Towards Training-free High-Resolution MLLMs Acceleration}, 
    author={Xuyang Liu and Ziming Wang and Yuhang Han and Yingyao Wang and Jiale Yuan and Jun Song and Bo Zheng and Linfeng Zhang and Siteng Huang and Honggang Chen},
    year={2025},
    eprint={2501.05179},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

💻 Related Works

• Awesome Token Reduction for Model Compression: An open-source repository that curates a collection of recent awesome papers on token reduction for model compression.
• FiCoCo: A systematic study that proposes a unified "filter-correlate-compress" paradigm for training-free token reduction in MLLMs, achieving up to 82.4% FLOPs reduction while maintaining model performance.

👍 Acknowledgment

We extend our gratitude to the open-source efforts of LLaVA and LLM-Viewer.

📩 Contact

For any question about our paper or code, please email liuxuyang@stu.scu.edu.cn.