Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark
Wukong: 100M Chinese cross-modal dataset with contrastive and token interaction techniques, boosting zero-shot classification accuracy to 73.03%.
Key Findings
Methodology
Wukong employs a dual-stream transformer architecture with ViT and SwinT as visual encoders, and Chinese WordPiece tokenization for text. The training integrates contrastive loss (CLIP style) and token-wise similarity (FILIP style), enhanced by a token reduction layer for efficiency. Data filtering includes image size thresholds, Chinese text length, frequency, and sensitive word removal, with human verification for quality. The training uses a locked-image encoder with a trainable Chinese text encoder (LiT approach). The model optimizes both global and token-level similarities, leveraging multi-architecture and multi-strategy pretraining for robust cross-modal alignment.
Key Results
- WukongViT-L achieves an average top-1 accuracy of 73.03% across 10 zero-shot image classification datasets, outperforming previous Chinese models. It attains 71.6% mean recall on AIC-ICC retrieval, surpassing WenLan 2.0 by 12.9%. The models demonstrate strong transferability across multiple downstream tasks, validating the dataset and architecture effectiveness.
- Different visual encoders (ViT-B/ViT-L/SwinT) show consistent performance, with ViT-L providing the best results. The combination of contrastive and token interaction strategies yields superior fine-grained alignment, especially with token reduction, balancing accuracy and efficiency.
- Extensive experiments confirm that Wukong models outperform industry benchmarks in zero-shot classification and retrieval, indicating the success of large-scale Chinese multimodal pretraining.
Significance
This work addresses the critical lack of large-scale Chinese cross-modal datasets, providing a comprehensive benchmark and pretraining models that significantly advance Chinese multimodal AI. The high performance in zero-shot tasks demonstrates the potential for broad industry adoption, enabling intelligent applications such as content filtering, image retrieval, and multilingual understanding. The open release of datasets and models fosters fair competition and accelerates research in Chinese AI, bridging the gap with English-centric datasets.
Technical Contribution
The paper introduces a novel multi-architecture, multi-strategy framework combining contrastive learning with token-level fine-grained alignment, enhanced by a token reduction mechanism for efficiency. It innovatively applies the LiT strategy to Chinese data, leveraging a locked image encoder with a trainable Chinese text encoder. The dataset construction pipeline ensures high-quality, diverse Chinese image-text pairs, setting a new standard for Chinese multimodal resources. The models demonstrate state-of-the-art performance, validating the effectiveness of these technical innovations.
Novelty
This is the first large-scale Chinese cross-modal dataset with 100 million pairs, integrating contrastive and token interaction methods, and introducing token reduction for efficiency. Unlike prior Chinese datasets, Wukong emphasizes data quality and diversity, enabling models to achieve unprecedented zero-shot accuracy. The combination of multi-architecture pretraining and innovative data filtering sets a new benchmark for Chinese multimodal AI.
Limitations
- Data collection relies on web scraping, which may introduce noise and bias despite filtering and verification, potentially affecting model robustness.
- Training large models requires extensive computational resources, limiting accessibility for smaller research groups.
- Chinese text encoding via WordPiece may not fully capture semantic nuances, suggesting future exploration of advanced tokenization methods.
Future Work
Future efforts include expanding multi-modal data sources, refining filtering techniques, reducing training costs, and exploring multi-task learning to improve contextual understanding. Additionally, integrating generative capabilities and multilingual support will broaden application scenarios, pushing Chinese AI towards more intelligent and versatile systems.
AI Executive Summary
The Wukong project introduces the largest Chinese cross-modal dataset, comprising 100 million image-text pairs collected from the web. This dataset addresses the critical gap in Chinese multimodal resources, enabling the training of advanced models that excel in zero-shot classification and retrieval tasks. The data collection process involved keyword-based web crawling, filtering for quality, and human verification, ensuring diversity and reliability.
Building on this dataset, the researchers developed a suite of models based on dual-stream transformer architectures, including ViT and SwinT visual encoders, combined with Chinese WordPiece tokenization. The training employed contrastive learning strategies similar to CLIP, enhanced with token-wise similarity (FILIP style) and a token reduction layer to improve efficiency. The models were trained using the LiT approach, locking the image encoder while training the Chinese text encoder, which facilitated cross-lingual generalization.
Experimental results demonstrate that WukongViT-L achieves an average accuracy of 73.03% on ten zero-shot image classification datasets, outperforming previous Chinese models. It also attains a 71.6% mean recall in image-text retrieval, surpassing industry benchmarks. These outcomes validate the effectiveness of the large-scale dataset and the proposed training strategies. The models exhibit strong transferability across various downstream tasks, including datasets like Flickr8K-CN and COCO-CN.
This work significantly advances Chinese multimodal AI, providing a high-quality, publicly available benchmark that fosters fair competition and innovation. The models' robust performance indicates promising applications in content retrieval, intelligent content understanding, and multilingual AI systems. Future directions involve expanding data sources, reducing computational costs, and enhancing semantic understanding, aiming to bring Chinese AI closer to the global forefront.
Deep Dive
Abstract
Vision-Language Pre-training (VLP) models have shown remarkable performance on various downstream tasks. Their success heavily relies on the scale of pre-trained cross-modal datasets. However, the lack of large-scale datasets and benchmarks in Chinese hinders the development of Chinese VLP models and broader multilingual applications. In this work, we release a large-scale Chinese cross-modal dataset named Wukong, which contains 100 million Chinese image-text pairs collected from the web. Wukong aims to benchmark different multi-modal pre-training methods to facilitate the VLP research and community development. Furthermore, we release a group of models pre-trained with various image encoders (ViT-B/ViT-L/SwinT) and also apply advanced pre-training techniques into VLP such as locked-image text tuning, token-wise similarity in contrastive learning, and reduced-token interaction. Extensive experiments and a benchmarking of different downstream tasks including a new largest human-verified image-text test dataset are also provided. Experiments show that Wukong can serve as a promising Chinese pre-training dataset and benchmark for different cross-modal learning methods. For the zero-shot image classification task on 10 datasets, $Wukong_{ViT-L}$ achieves an average accuracy of 73.03%. For the image-text retrieval task, it achieves a mean recall of 71.6% on AIC-ICC which is 12.9% higher than WenLan 2.0. Also, our Wukong models are benchmarked on downstream tasks with other variants on multiple datasets, e.g., Flickr8K-CN, Flickr-30K-CN, COCO-CN, et al. More information can be referred to: https://wukong-dataset.github.io/wukong-dataset/.