Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
Proposes Loss-Free Balancing, a method that dynamically adjusts expert biases to improve load balance without interference gradients, outperforming auxiliary loss methods.
Lean Wang, Huazuo Gao, Chenggang Zhao et al.