From Execution to Capability: Scientific Experience Consolidation via Procedural Knowledge Synthesis
SciConsolidate synthesizes procedural knowledge to enhance scientific computing, improving Qwen3.6-27B by 6.26 points.
Key Findings
Methodology
The paper introduces the SciConsolidate framework, which contrasts verified successes and failures to induce cross-task procedures, selects them through a development-validation gate, and uses failure-informed, answer-free query synthesis to expand consolidation data. A stronger model concretizes abstractions into executable code supervision.
Key Results
- On SciCode, runtime procedure injection improves Qwen3.6-27B by 3.85/6.26 points, but yields almost no aggregate main-problem gain for Qwen3.5-9B, providing operational evidence of the abstraction-execution gap.
- After procedure-guided concretization, the 9B student improves under procedure-free deployment by 3.89/6.25 points over the no-procedure SFT control and by 5.62/11.25 over the original 9B model.
- These results establish an experience-to-capability pathway for scientific computing and provide a practical starting point for scaling self-improving scientific assistance.
Significance
This research provides a pathway for converting experience into capability in scientific computing, addressing the issue of large language models failing to achieve persistent improvement in scientific computing tasks. By synthesizing and concretizing procedural knowledge, models can improve performance without external procedures, offering a practical starting point for expanding self-improving scientific assistance.
Technical Contribution
Technical contributions include introducing a new experience consolidation framework, SciConsolidate, addressing the abstraction-execution gap, and enhancing model capability through procedure-guided supervised learning. Compared to existing methods, this framework offers new theoretical guarantees and engineering possibilities.
Novelty
This study is the first to propose converting scientific computing experience into reusable procedural knowledge and enhancing model capability through procedure-guided supervised learning. The innovation lies in addressing the abstraction-execution gap compared to related work.
Limitations
- For weaker target models, abstract procedures may not be directly executable, indicating a capacity-dependent abstraction-execution gap.
- The utility of procedural knowledge varies across task types and failure families.
- The multi-round self-improvement pathway is not evaluated in this study.
Future Work
Future work could include evaluating the multi-round self-improvement pathway, further exploring the utility of procedural knowledge across different task types, and developing stronger models to bridge the abstraction-execution gap.
AI Executive Summary
Large language models have made significant progress in scientific computing tasks, yet executable feedback from one problem rarely becomes durable capability on subsequent problems. This paper studies scientific-computing experience consolidation: converting verified runtime experience into transferable procedural knowledge and persistent model improvement. We introduce the SciConsolidate framework, which contrasts verified successes and failures to induce cross-task procedures, selects them through a development-validation gate, and uses failure-informed, answer-free query synthesis to expand consolidation data. A stronger model concretizes abstractions into executable code supervision. Experimental results show that on SciCode, runtime procedure injection improves Qwen3.6-27B by 3.85/6.26 points, but yields almost no aggregate main-problem gain for Qwen3.5-9B, providing operational evidence of the abstraction-execution gap. After procedure-guided concretization, the 9B student improves under procedure-free deployment by 3.89/6.25 points over the no-procedure SFT control and by 5.62/11.25 over the original 9B model. These results establish an experience-to-capability pathway for scientific computing and provide a practical starting point for scaling self-improving scientific assistance.
Deep Analysis
Background
In recent years, large language models have made significant progress in scientific computing, especially in translating scientific specifications into executable numerical, analytical, and simulation procedures. However, current evaluations emphasize whether a model solves each problem once, not whether such execution evidence reduces related failures on subsequent problems. Therefore, scientific computing requires a mechanism that abstracts verified execution evidence beyond individual problems, validates its transfer, and returns it to the model as durable capability.
Core Problem
Scientific computing experience consolidation faces two challenges: trajectory-derived artifacts may encode source-specific repairs rather than cross-task computational mechanisms; a weaker target model may be unable to operationalize an otherwise valid abstract procedure, creating an abstraction-execution gap.
Innovation
We introduce the SciConsolidate framework, which contrasts verified successes and failures to induce cross-task procedures, selects them through a development-validation gate, and uses failure-informed, answer-free query synthesis to expand consolidation data. A stronger model concretizes abstractions into executable code supervision.
Methodology
- �� Contrast verified successes and failures to induce cross-task procedures
- �� Select procedures through a development-validation gate
- �� Use failure-informed, answer-free query synthesis to expand consolidation data
- �� A stronger model concretizes abstractions into executable code supervision
Experiments
We evaluate this pathway on SciCode. Runtime procedure injection improves Qwen3.6-27B by 3.85 sub-step points and 6.26 main-problem points, but produces almost no aggregate main-problem gain for Qwen3.5-9B. After procedure-guided concretization and SFT, the 9B student evaluated without runtime procedures improves over the no-procedure SFT control by 3.89/6.25 points and over the original 9B model by 5.62/11.25 points.
Results
Experimental results show that on SciCode, runtime procedure injection improves Qwen3.6-27B by 3.85/6.26 points, but yields almost no aggregate main-problem gain for Qwen3.5-9B, providing operational evidence of the abstraction-execution gap. After procedure-guided concretization, the 9B student improves under procedure-free deployment by 3.89/6.25 points over the no-procedure SFT control and by 5.62/11.25 over the original 9B model.
Applications
This research provides a pathway for converting experience into capability in scientific computing, addressing the issue of large language models failing to achieve persistent improvement in scientific computing tasks. By synthesizing and concretizing procedural knowledge, models can improve performance without external procedures, offering a practical starting point for expanding self-improving scientific assistance.
Limitations & Outlook
For weaker target models, abstract procedures may not be directly executable, indicating a capacity-dependent abstraction-execution gap. The utility of procedural knowledge varies across task types and failure families. The multi-round self-improvement pathway is not evaluated in this study.
Plain Language Accessible to non-experts
Imagine a kitchen where a chef encounters problems like mismatched ingredients or incorrect cooking steps. By observing successful dishes and failed attempts, the chef can summarize some general cooking techniques, such as how to properly cut ingredients or adjust the heat. These techniques are like procedural knowledge, helping the chef avoid similar mistakes in future cooking. This study investigates how to transform these experiences into model capabilities, allowing models to perform better in scientific computing tasks.
ELI14 Explained like you're 14
Imagine you're playing a game, and after each level, you learn new skills like defeating enemies or solving puzzles. These skills help you pass future levels faster. This study investigates how large language models can accumulate experience like you do, becoming stronger in solving scientific computing problems. By observing successful and failed cases, the model can summarize some general solutions, helping it perform better in future tasks.
Glossary
SciConsolidate
A framework for converting scientific computing experience into reusable procedural knowledge.
The core method proposed in this paper to address the abstraction-execution gap.
Abstraction-Execution Gap
A capacity-dependent issue where models struggle to operationalize abstract procedures.
One of the main challenges studied in this paper, affecting model execution capability.
Procedural Knowledge
Reusable solutions based on execution evidence.
Knowledge induced by contrasting successes and failures to enhance model capability.
SFT
A training method that enhances model performance through supervised learning.
Used to train student models to improve their performance without runtime procedures.
Qwen3.6-27B
A stronger model used to concretize abstract procedures.
Used in experiments for procedure-guided supervised learning.
Open Questions Unanswered questions from this research
- 1 How to effectively apply procedural knowledge across different task types?
- 2 How to bridge the abstraction-execution gap for weaker models?
- 3 How to evaluate the multi-round self-improvement pathway?
Applications
Immediate Applications
Scientific Computing Task Optimization
Enhance model performance in scientific computing tasks by reducing error rates through procedural knowledge.
Model Capability Enhancement
Improve model performance in scientific computing tasks without external procedures.
Long-term Vision
Self-Improving Scientific Assistant
Through a multi-round self-improvement pathway, models can continuously enhance their capabilities, becoming stronger scientific assistants.
Abstract
Large language models increasingly solve scientific-computing tasks, but executable feedback from one problem rarely becomes durable capability on subsequent problems. We study scientific-computing experience consolidation: converting verified runtime experience into transferable procedural knowledge and persistent model improvement. This setting presents two challenges: trajectory-derived artifacts may encode source-specific repairs rather than cross-task computational mechanisms; and a weaker target model may be unable to operationalize an otherwise valid abstract procedure - an abstraction-execution gap. We introduce SciConsolidate, which contrasts verified successes and failures to induce cross-task procedures, selects them through a development-validation gate, and uses failure-informed, answer-free query synthesis to expand the consolidation data without requiring pre-existing reference answers. Because the target model may not directly execute these abstractions, a stronger model concretizes them into executable code supervision for standard, procedure-free SFT; a matched no-procedure teacher branch isolates the value of procedural guidance. On SciCode, runtime procedure injection improves Qwen3.6-27B by +3.85/+6.26 sub-step/main-problem points, but yields almost no aggregate main-problem gain for Qwen3.5-9B, providing operational evidence of the abstraction-execution gap. After procedure-guided concretization, the 9B student improves under procedure-free deployment by +3.89/+6.25 points over the no-procedure SFT control and by +5.62/+11.25 over the original 9B model. These results establish an experience-to-capability pathway for scientific computing and provide a practical starting point for scaling self-improving scientific assistance.