MergeVQ: A Unified Framework for Visual Generation and Representation with Token Merging and Quantization (CVPR 2025)

Siyuan Li^1,3*, Luyuan Zhang^2*, Zedong Wang⁴, Juanxi Tian³, Cheng Tan^1,3, Zicheng Liu^1,3, Chang Yu³, Qingsong Xie^5†, Haoqian Wang², Zhen Lei^6,7,8†

¹ Zhejiang University   ² Tsinghua University   ³ Westlake University   ⁴ HKUST   ⁵ OPPO AI Center   ⁶ CAIR, HKISI-CAS   ⁷ MAIS CASIA   ⁸ University of Chinese Academy of Sciences

^* Equal Contributions. ^† Corresponding Authors.

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2025

Masked Image Modeling (MIM) with Vector Quantization (VQ) has achieved great success in both self-supervised pre-training and image generation. However, most existing methods struggle to address the trade-off in shared latent space for generation quality vs. representation learning and efficiency. To push the limits of this paradigm, we propose MergeVQ, which incorporates token merging techniques into VQ-based autoregressive generative models to bridge the gap between visual generation and representation learning in a unified architecture. During pre-training, MergeVQ decouples top-k semantics from latent space with a token merge module after self-attention blocks in the encoder for subsequent Look-up Free Quantization (LFQ) and global alignment and recovers their fine-grained details through cross-attention in the decoder for reconstruction. As for the second-stage generation, we introduce MergeAR, which performs KV Cache compression for efficient raster-order prediction. Experiments on ImageNet verify that MergeVQ as an AR generative model achieves competitive performance in both representation learning and image generation tasks while maintaining favorable token efficiency and inference speed.

🤗 HuggingFace Daily Papers Top1: https://huggingface.co/papers

Catalog

We plan to release implementations of MergeVQ in a few months (before CVPR2025 taking place). Please watch us for the latest release and welcome to open issues for discussion!

Citation

@inproceedings{cvpr2025mergevq,
    title={MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantization},
    author={Li, Siyuan and Zhang, Luyuan and Wang, Zedong and Tian, Juanxi and Tan, Cheng and Liu, Zicheng and Yu, Chang and Xie, Qingsong and Lu, Haonan and Wang, Haoqian and Lei, Zhen},
    booktitle={Conference on Computer Vision and Pattern Recognition (CVPR)},
    year={2025}
}

@misc{li2025mergevqunifiedframeworkvisual,
      title={MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantization}, 
      author={Siyuan Li and Luyuan Zhang and Zedong Wang and Juanxi Tian and Cheng Tan and Zicheng Liu and Chang Yu and Qingsong Xie and Haonan Lu and Haoqian Wang and Zhen Lei},
      year={2025},
      eprint={2504.00999},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2504.00999}, 
}

(back to top)