GptGet

Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent

TL;DR

Hunyuan-Large is Tencent's open-source MoE model with 52 billion activated parameters, handling up to 256K tokens.

cs.CL 🔴 Advanced 2024-11-05 2 views
Xingwu Sun Yanfeng Chen Yiqing Huang Ruobing Xie Jiaqi Zhu Kai Zhang Shuaipeng Li Zhen Yang Jonny Han Xiaobo Shu Jiahao Bu Zhongzhi Chen Xuemeng Huang Fengzong Lian Saiyong Yang Jianfeng Yan Yuyuan Zeng Xiaoqin Ren Chao Yu Lulu Wu Yue Mao Jun Xia Tao Yang Suncong Zheng Kan Wu Dian Jiao Jinbao Xue Xipeng Zhang Decheng Wu Kai Liu Dengpeng Wu Guanghui Xu Shaohua Chen Shuang Chen Xiao Feng Yigeng Hong Junqiang Zheng Chengcheng Xu Zongwei Li Xiong Kuang Jianglu Hu Yiqi Chen Yuchi Deng Guiyang Li Ao Liu Chenchen Zhang Shihui Hu Zilong Zhao Zifan Wu Yao Ding Weichao Wang Han Liu Roberts Wang Hao Fei Peijie Yu Ze Zhao Xun Cao Hai Wang Fusheng Xiang Mengyuan Huang Zhiyuan Xiong Bin Hu Xuebin Hou Lei Jiang Jianqiang Ma Jiajia Wu Yaping Deng Yi Shen Qian Wang Weijie Liu Jie Liu Meng Chen Liang Dong Weiwen Jia Hu Chen Feifei Liu Rui Yuan Huilin Xu Zhenxiang Yan Tengfei Cao Zhichao Hu Xinhua Feng Dong Du Tinghao Yu Yangyu Tao Feng Zhang Jianchen Zhu Chengzhong Xu Xirui Li Chong Zha Wen Ouyang Yinben Xia Xiang Li Zekun He Rongpeng Chen Jiawei Song Ruibin Chen Fan Jiang Chongqing Zhao Bo Wang Hao Gong Rong Gan Winston Hu Zhanhui Kang Yong Yang Yuhong Liu Di Wang Jie Jiang
AI Reader Arxiv Page Download PDF
MoE Model Transformer Open Source Large Parameters Long Text Processing

Key Findings

Methodology

Hunyuan-Large employs mixed expert routing, KV cache compression, and expert-specific learning rate strategies. Pre-trained on synthetic data, it handles 256K tokens.

Key Results

  • Outperforms LLama3.1-70B in language tasks, comparable to LLama3.1-405B.
  • Excels in math problem-solving and coding tasks, showcasing strong long-text processing capabilities.
  • Ablation studies confirm the efficacy of mixed expert routing strategy.

Significance

Hunyuan-Large offers new insights for scaling MoE models, enhancing multi-task applications, especially in long-text processing.

Technical Contribution

Introduces mixed expert routing and KV cache compression, significantly improving training and inference efficiency, supporting larger parameters and long-text processing.

Novelty

First open-source model to achieve 389 billion total parameters and 52 billion activated parameters in MoE architecture, breaking existing model scale limits.

Limitations

  • Performance in specific tasks needs optimization, especially in low-resource language settings.
  • High computational cost for long-text processing requires further optimization.

Future Work

Future work will focus on optimizing efficiency in long-text processing and exploring adaptability to more tasks.

AI Executive Summary

Hunyuan-Large is Tencent's open-source MoE model with 389 billion total parameters and 52 billion activated parameters, capable of handling texts up to 256K tokens. The model excels in language understanding, generation, logical reasoning, math problem-solving, and coding tasks, outperforming LLama3.1-70B and approaching LLama3.1-405B. Its core technologies include mixed expert routing, KV cache compression, and expert-specific learning rate strategies. Pre-trained on large-scale synthetic data, Hunyuan-Large demonstrates strong long-text processing capabilities, advancing multi-task applications of large-scale language models. Although there is room for optimization in specific tasks, particularly in low-resource language settings, the model provides a solid technical foundation and direction for future research. Future work will focus on optimizing efficiency in long-text processing and exploring adaptability to more tasks.

Deep Analysis

Background

In recent years, large language models have significantly advanced artificial intelligence, particularly in NLP, CV, and speech recognition. With the emergence of models like ChatGPT, researchers are exploring new techniques and paradigms to push the boundaries of model size and performance. Mixture of Experts (MoE) models offer the possibility of achieving more efficient training and inference by dynamically activating experts.

Core Problem

Existing open-source models are mostly based on dense architectures with relatively small parameter scales, unable to effectively handle ultra-long texts. Achieving large-scale parameters and long-text processing capabilities in open-source models is a crucial and challenging problem.

Innovation

Hunyuan-Large achieves a MoE architecture with 389 billion total parameters and 52 billion activated parameters through mixed expert routing, KV cache compression, and expert-specific learning rate strategies, breaking existing model scale limits.

Methodology

  • �� Mixed Expert Routing: Combines shared and specialized experts, dynamically allocating tokens.
  • �� KV Cache Compression: Uses GQA and CLA techniques to reduce memory pressure.
  • �� Expert-Specific Learning Rate: Adjusts learning rates based on the number of tokens trained by different experts.

Experiments

Experimental design includes language understanding, generation, logical reasoning, math problem-solving, and coding tasks, using LLama3.1-70B and LLama3.1-405B as baselines for comparison.

Results

Hunyuan-Large excels in multiple tasks, particularly in long-text processing capabilities, outperforming LLama3.1-70B and approaching LLama3.1-405B.

Applications

The model can be applied to multi-task processing in fields like NLP, CV, and speech recognition, especially suitable for applications requiring long-text processing.

Limitations & Outlook

While the model performs well in multiple tasks, optimization is needed for low-resource language settings and computational costs in long-text processing.

Plain Language Accessible to non-experts

Imagine a large library where Hunyuan-Large acts like a super librarian, quickly finding and processing vast amounts of information. It uses a special classification system to distribute information to different experts, each skilled at handling specific types of information. This way, the library can efficiently manage various complex tasks.

ELI14 Explained like you're 14

Imagine playing a super complex game where Hunyuan-Large is your game assistant, helping you find answers quickly. It has many mini-assistants, each good at different tasks like math, coding, language understanding, etc. With these assistants working together, you can easily solve all the game's challenges!

Glossary

MoE Model

An architecture using multiple expert models to improve performance.

Used for dynamically activating experts to handle different tasks.

KV Cache Compression

Reduces memory usage of KV cache to improve inference efficiency.

Used to optimize model's memory usage.

Mixed Expert Routing Strategy

Combines shared and specialized expert routing strategies.

Used for dynamic token allocation to enhance model performance.

SwiGLU

An activation function to enhance model's nonlinear expression capability.

Used in the model's activation layers.

RoPE (Rotary Position Embedding)

An embedding technique for position learning.

Used for positional information encoding in long-text processing.

Open Questions Unanswered questions from this research

  • 1 How to further optimize performance in low-resource language settings?
  • 2 How to reduce computational costs for long-text processing?

Applications

Immediate Applications

Natural Language Processing

Applications supporting long-text processing, such as document analysis and information retrieval.

Long-term Vision

Intelligent Assistant

Develop smarter personal assistants by optimizing long-text processing capabilities.

Abstract

In this paper, we introduce Hunyuan-Large, which is currently the largest open-source Transformer-based mixture of experts model, with a total of 389 billion parameters and 52 billion activation parameters, capable of handling up to 256K tokens. We conduct a thorough evaluation of Hunyuan-Large's superior performance across various benchmarks including language understanding and generation, logical reasoning, mathematical problem-solving, coding, long-context, and aggregated tasks, where it outperforms LLama3.1-70B and exhibits comparable performance when compared to the significantly larger LLama3.1-405B model. Key practice of Hunyuan-Large include large-scale synthetic data that is orders larger than in previous literature, a mixed expert routing strategy, a key-value cache compression technique, and an expert-specific learning rate strategy. Additionally, we also investigate the scaling laws and learning rate schedule of mixture of experts models, providing valuable insights and guidances for future model development and optimization. The code and checkpoints of Hunyuan-Large are released to facilitate future innovations and applications. Codes: https://github.com/Tencent/Hunyuan-Large Models: https://huggingface.co/tencent/Tencent-Hunyuan-Large

cs.CL cs.AI