Introduces DataComp-LM, a benchmark with 240T tokens, using model-based filtering to improve training data quality, achieving 64% MMLU 5-shot accuracy with a 7B model.

Jeffrey Li, Alex Fang, Georgios Smyrnis et al.

Automated BenchBuilder pipeline creates 500 challenging prompts, improving model separation by 3x, with 98.6% correlation to human preferences.

Tianle Li, Wei-Lin Chiang, Evan Frick et al.

Refusal in language models is mediated by a single linear direction; interventions manipulating this direction can control refusal behavior.

Andy Arditi, Oscar Obeso, Aaquib Syed et al.

QTIP employs trellis-coded quantization (TCQ) with hardware-efficient codes to enable ultra-high-dimensional weight quantization, boosting model compression and inference speed.

Albert Tseng, Qingyao Sun, David Hou et al.

This survey links RandNLA’s three sketching paradigms to RMT-based Gaussianization; Gaussian sketches achieve condition number at most 6 for s≥2d.

Michał Dereziński, Michael W. Mahoney

DigiRL combines offline and online RL to fine-tune a 1.3B VLM, boosting success rate from 17.7% to 67.2% in Android device control tasks.

Hao Bai, Yifei Zhou, Mert Cemri et al.

MV-Mol employs multi-modal fusion with Q-Former, using two-stage pretraining to enhance molecular property prediction accuracy by 1.24%.

Yizhen Luo, Kai Yang, Massimo Hong et al.

MLKV introduces cross-layer KV sharing, reducing memory cache by 6× with minimal performance loss in Transformer decoding.

Zayd Muhammad Kawakibi Zuhri, Muhammad Farid Adilazuarda, Ayu Purwarianti et al.

Proposes Center-sensitive Kernel Optimization (CsKO) for low-resource on-device incremental learning, boosting accuracy by 38.08% with 75% less computation.

Dingwen Zhang, Yan Li, De Cheng et al.

This study compares 8B Mamba, Mamba-2, and Transformer models trained on up to 3.5T tokens, showing hybrid models outperform in speed and long-sequence tasks.

Roger Waleffe, Wonmin Byeon, Duncan Riach et al.

Proposes Locally Interdependent Multi-Agent MDP with three closed-form near-optimal policies, performance improves exponentially with visibility radius.

Alex DeWeese, Guannan Qu

R2N boosts PbRL robustness via dynamic sparsity, improving 15 baseline-environment settings on noisy DMControl tasks.

Calarina Muslimani, Bram Grooten, Deepak Ranganatha Sastry Mamillapalli et al.

DeltaNet parallelizes linear transformers over sequence length, enhancing training efficiency and language modeling performance.

Songlin Yang, Bailin Wang, Yu Zhang et al.

Proposes Variational Flow Matching (VFM) and CatFlow for categorical graph generation, outperforming current SOTA.

Floor Eijkelboom, Grigory Bartosh, Christian Andersson Naesseth et al.

FlowMM employs Riemannian Flow Matching to generate stable crystal structures efficiently, reducing inference steps by 3x compared to diffusion models.

Benjamin Kurt Miller, Ricky T. Q. Chen, Anuroop Sriram et al.

This study evaluates PaLM 2's forecasting ability using a real-world event dataset, revealing a bias toward underestimating event probabilities.

Sarah Pratt, Seth Blumberg, Pietro Kreitlon Carolino et al.

Proposes a simplified masked diffusion framework using weighted cross-entropy, surpassing GPT-2 scale in text and pixel modeling.

Jiaxin Shi, Kehang Han, Zhe Wang et al.

Using k-sparse autoencoders with TopK activation, trained on GPT-4 activations, scaled to 16 million latent variables, revealing clear power-law relationships between size and feature quality.

Leo Gao, Tom Dupré la Tour, Henk Tillman et al.

Introduces Latent Neural Operator (LNO) using Physics-Cross-Attention in latent space, reducing GPU memory by 50%, training 1.8× faster, achieving state-of-the-art accuracy.

Tian Wang, Chuang Wang

RADD model reparameterizes absorbing discrete diffusion to enhance language modeling performance.

Jingyang Ou, Shen Nie, Kaiwen Xue et al.