Benchmarking LLMs on File System Design and Implementation
Introduces φ-Bench, a benchmark with 505 AI-assisted file system tasks, evaluating multiple LLMs' performance and failure modes.
Key Findings
Methodology
The study constructs φ-Bench with 505 tasks covering understanding, implementation, modeling, debugging, optimization, and feature development. An AI-assisted pipeline, integrating OS textbooks and Linux source code, generates high-quality tasks with minimal human effort. Tasks are peer-reviewed, ensuring quality. Empirical evaluation involves open-source models (DeepSeek-V4-Flash, GLM-5.1, MiniMax-2.7) and proprietary models (Claude-Opus-4.7, GPT-5.2, Gemini-3.1-Pro). Performance metrics include pass rate, stability, failure causes, and cost analysis, with techniques like prompt engineering and self-review improving outcomes.
Key Results
- Models excel in basic understanding, implementation, and modeling tasks, achieving up to 95.8% pass rate. Complex tasks like debugging, optimization, and feature development see significant drops, with success rates of 61.6%, 37.6%, and 41.9%. Failures mainly stem from semantic misuse (19.1%), logical errors (22.3%), incomplete reasoning (14.5%), and suboptimal code (17.5%). Cost analysis shows low-cost models like DeepSeek-V4-Flash are 16×–107× cheaper than GPT-5.2, with performance correlated to token usage.
- Multiple sampling and prompt techniques improve pass rates, but 75%-79% failures persist, indicating inherent model limitations. Failures are often due to incomplete reasoning and suboptimal implementations. Techniques such as self-review and iterative refinement significantly boost success, with improvements up to 83%-89%. The study highlights the need for models to better handle edge cases and performance-aware code generation.
- This work provides a comprehensive evaluation framework, revealing strengths and weaknesses of current LLMs in file system tasks. It guides future model training, task design, and system integration efforts. The open-source pipeline for task creation offers a scalable approach for other domains, fostering broader research and development in AI-assisted system engineering.
Significance
This research addresses a critical gap in evaluating LLMs for complex system development, specifically file systems. By providing a detailed, fine-grained benchmark, it enables precise assessment of models' capabilities across different development stages. The insights into failure modes and optimization strategies inform both academia and industry, guiding the design of more robust, reasoning-capable models. The open-source nature of φ-Bench accelerates community-driven innovation, reducing development costs and improving system reliability. It paves the way for future AI-automated system design, integrating deep reasoning and performance awareness, ultimately transforming software engineering workflows.
Technical Contribution
The paper introduces φ-Bench, a novel, comprehensive benchmark tailored for file system development tasks, integrating six task categories and detailed capability assessments. It innovates with an AI-assisted task generation pipeline that combines expert input, textbooks, and source code, ensuring high-quality, diverse tasks with minimal manual effort. The framework enables fine-grained capability analysis, revealing specific failure modes related to semantics, logic, and performance. It also explores prompt engineering and resampling techniques to enhance model robustness. These contributions significantly advance the evaluation methodology for domain-specific LLM applications, especially in complex system engineering.
Novelty
This is the first systematic benchmark specifically designed for evaluating LLMs on file system design and implementation tasks. Its innovative AI-assisted task generation pipeline reduces manual labor while maintaining high quality, a significant improvement over existing benchmarks that lack domain specificity and granularity. The detailed taxonomy of tasks and capabilities, combined with empirical insights into failure modes and mitigation strategies, sets this work apart from prior general software or scientific benchmarks, establishing a new standard for domain-specific model evaluation.
AI Executive Summary
The rapid advancement of large language models (LLMs) has opened new horizons in software engineering, yet their application to complex, domain-specific tasks like file system development remains underexplored. Existing benchmarks primarily evaluate general language understanding or coding ability, leaving a gap in assessing models' competence in system-level design and implementation. To address this, the authors introduce φ-Bench, a comprehensive, AI-assisted benchmarking framework tailored for file system tasks.
φ-Bench comprises 505 meticulously crafted tasks spanning understanding, implementation, performance modeling, debugging, optimization, and new feature development. These tasks are grounded in Linux kernel source code and OS textbooks, ensuring relevance and realism. An innovative AI pipeline automates task generation, with human experts overseeing quality, drastically reducing manual effort while maintaining high standards. This approach enables broad coverage of domain concepts and development stages.
Empirical evaluation of six leading LLMs reveals significant performance disparities. Models perform well on basic tasks, with success rates exceeding 90%, but struggle with complex activities like debugging and optimization, where success drops below 50%. Failures often stem from semantic errors, logical reasoning gaps, and incomplete consideration of edge cases. Techniques such as prompt engineering and multiple sampling improve outcomes, yet persistent challenges remain.
The findings highlight critical limitations in current models, emphasizing the need for enhanced reasoning and performance-awareness capabilities. The study offers actionable insights for future model training, including targeted fine-tuning and task-specific prompting strategies. The open-source φ-Bench framework provides a valuable tool for researchers and practitioners, fostering innovation in AI-assisted system development. Overall, this work marks a significant step toward integrating LLMs into the core of system engineering workflows, promising more automated, reliable, and efficient development processes in the future.
Deep Dive
Abstract
Large Language Models (LLMs) are fundamentally transforming computer system research and development. As we employ LLMs in file system (fs) development, it is essential to understand their capabilities, limitations, and operational efficiency for domain-specific tasks. We present φ-Bench, an LLM benchmarking framework for fs-specific tasks. To facilitate benchmarking, we develop six types of tasks in φ-Bench: basic understanding, basic implementation, performance modeling, debugging, optimization, and new feature development. Each type emphasizes different LLM capabilities: instruction following, knowledge recall, reasoning, or coding. To create high-quality tasks while achieving broad coverage with minimal human effort, we develop a new AI-assisted task generation pipeline in addition to expert-written and textbook-adapted tasks. With 505 tasks in φ-Bench, we conduct an empirical study with both open source (DeepSeek-V4-Flash, GLM-5.1, and MiniMax-M2.7) and proprietary (Claude-Opus-4.7, GPT-5.2, and Gemini-3.1-Pro) LLMs. Our study discloses the model efficiency for different tasks, causes of failed fs tasks, and techniques for mitigating LLM failures. We will open source φ-Bench to facilitate public research on using LLMs for fs development.