RefinedWeb dataset uses rigorous filtering and deduplication on web data alone, training models that outperform The Pile with 5 trillion tokens, matching GPT-3 performance.

Guilherme Penedo, Quentin Malartic, Daniel Hesslow et al.

This study evaluates GPT-4's geographic knowledge across tasks like location, distance, and topography, highlighting strengths and limitations.

Jonathan Roberts, Timo Lüddecke, Sowmen Das et al.

This study evaluates LFQA with human experts and automatic metrics, revealing no single metric reliably predicts human preferences.

Fangyuan Xu, Yixiao Song, Mohit Iyyer et al.

Marked Personas employs natural language prompts to unsupervisedly quantify intersectional stereotypes in LLM outputs, revealing higher bias levels in GPT-3.5 and GPT-4 compared to human descriptions.

Myra Cheng, Esin Durmus, Dan Jurafsky

Proposes TTT-NN, a test-time fine-tuning method using nearest neighbor retrieval, improving over 20 tasks with only one gradient step on 20 neighbors.

Moritz Hardt, Yu Sun

Proposed a calibration framework to address positional bias in LLM evaluations, improving human alignment by 14.3%.

Peiyi Wang, Lei Li, Liang Chen et al.

LLM-QAT achieves 4-bit quantization via data-free distillation, enhancing large language model performance.

Zechun Liu, Barlas Oguz, Changsheng Zhao et al.

Proposes Iter-RetGen, an iterative retrieval-generation framework, boosting multi-task QA accuracy by over 8% with fewer API calls.

Zhihong Shao, Yeyun Gong, Yelong Shen et al.

Introduces JEEBench, a challenging dataset from Indian IIT JEE exams, revealing GPT-4's performance below 40% on complex problems.

Daman Arora, Himanshu Gaurav Singh, Mausam

Proposes RoHT framework with hierarchical question decomposition tree, achieving 29.7% EM improvement on KQA Pro, surpassing SOTA.

Jiajie Zhang, Shulin Cao, Tingjia Zhang et al.

This study systematically compares distillation objectives (prediction, hidden states, attention) and teacher layer initialization, revealing attention transfer's robustness and low-layer initialization's benefits, especially on GLUE tasks.

Xinpeng Wang, Leonie Weissweiler, Hinrich Schütze et al.

Large language models excel in sentiment analysis but struggle with complex tasks.

Wenxuan Zhang, Yue Deng, Bing Liu et al.

Formalized jailbreak taxonomy; tested 3700 prompts across GPT, BLOOM, OPT, FLAN-T5-XXL; success rates >78%; detection via GPT-4 accuracy 92%.

Abhinav Rao, Sachin Vashistha, Atharva Naik et al.

Introduced MetricEval framework using measurement theory to evaluate NLG metric reliability and validity.

Ziang Xiao, Susu Zhang, Vivian Lai et al.

Proposes AutoCompressors, an unsupervised method compressing long texts into summary vectors, improving perplexity and in-context learning efficiency.

Alexis Chevalier, Alexander Wettig, Anirudh Ajith et al.

Contrastive decoding (CAD) improves factual consistency across models by adjusting output probabilities based on context, especially effective in knowledge conflict scenarios.

Weijia Shi, Xiaochuang Han, Mike Lewis et al.

Proposed ALCE benchmark for automatic evaluation of LLMs generating citation-supported texts, covering fluency, correctness, and citation quality.

Tianyu Gao, Howard Yen, Jiatong Yu et al.

Proposes LLM-based dynamic model selection combining CoT and PAL, achieving 96.8% on GSM8K, surpassing state-of-the-art.

James Xu Zhao, Yuxi Xie, Kenji Kawaguchi et al.

CREATOR enables LLMs to generate tools via documentation and code, disentangling abstract creation from concrete reasoning, improving performance on math and tabular tasks.

Cheng Qian, Chi Han, Yi R. Fung et al.

SciMON integrates knowledge retrieval and iterative optimization, significantly enhancing LLMs' ability to generate novel scientific ideas grounded in literature.

Qingyun Wang, Doug Downey, Heng Ji et al.