Models Recall What They Violate: Constraint Adherence in Multi-Turn LLM Ideation
DriftBench evaluates constraint adherence in multi-turn LLM ideation, revealing complexity inflation and constraint violations.
Key Findings
Methodology
DriftBench is a benchmark for evaluating constraint adherence in multi-turn LLM-assisted scientific ideation. It involves 2,146 scored runs across seven models, analyzing constraint recall and behavioral adherence using structured checkpointing and restatement probes.
Key Results
- Under pressure, Sonnet 4.6 shows a KBV rate of 99%, while GPT-5.4 maintains only 8%.
- Structured checkpointing partially reduces KBV rates but does not eliminate complexity inflation.
- Human validation indicates LLM judges under-detect constraint violations, making reported adherence scores conservative.
Significance
This study reveals that models can accurately recall constraints while simultaneously violating them in multi-turn interactions. This has significant implications for scientific ideation and model design, suggesting a need to better understand model behavior in complex interactions.
Technical Contribution
Introduces DriftBench, a new benchmark for systematically evaluating constraint adherence in multi-turn interactions. Highlights the dissociation between constraint recall and behavioral adherence, validating the effectiveness of structured checkpointing.
Novelty
First systematic evaluation of constraint adherence in multi-turn LLM interactions, revealing widespread complexity inflation and constraint violations.
Limitations
- Pressure prompts may induce a tradeoff between satisfying the latest user instruction and preserving the original brief.
- Neutral condition validity needs further verification.
- Research domain limited to scientific ideation; drift patterns may differ in other tasks.
Future Work
Future research could explore constraint adherence in different tasks and develop new methods to reduce complexity inflation and constraint violations.
AI Executive Summary
In multi-turn LLM interactions, whether models can maintain fidelity to the original objective has been a key concern for researchers. Existing evaluation methods often focus on single-turn outputs or task success rates, neglecting constraint adherence across turns. To address this gap, this paper introduces DriftBench, a benchmark for evaluating constraint adherence in multi-turn LLM-assisted scientific ideation.
Through 2,146 scored runs across seven models, the study finds that under pressure, models significantly increase structural complexity while reducing constraint adherence. Notably, Sonnet 4.6 shows a KBV rate of 99%, while GPT-5.4 maintains only 8%. Structured checkpointing partially reduces KBV rates but does not eliminate complexity inflation.
These findings have significant implications for scientific ideation and model design, suggesting a need to better understand model behavior in complex interactions. Future research could explore constraint adherence in different tasks and develop new methods to reduce complexity inflation and constraint violations.
Deep Analysis
Background
With the increasing application of large language models in scientific ideation, researchers are focusing on whether models can maintain fidelity to the original objective in multi-turn interactions. Existing evaluation methods often focus on single-turn outputs or task success rates, neglecting constraint adherence across turns.
Core Problem
Constraint adherence in multi-turn interactions is a significant and complex research issue. Models may violate original constraints due to complexity inflation, affecting the quality of scientific ideation.
Innovation
This paper introduces DriftBench, a benchmark for evaluating constraint adherence in multi-turn LLM-assisted scientific ideation. It uses structured checkpointing and restatement probes to reveal model behavior patterns in complex interactions.
Methodology
- �� DriftBench benchmark: evaluates constraint adherence in multi-turn LLM scientific ideation.
- �� Structured checkpointing: periodically checks model constraint adherence during interaction.
- �� Restatement probe: tests model constraint recall ability.
Experiments
The experimental design includes 2,146 scored runs across seven models, covering models from five providers. It evaluates constraint adherence and complexity inflation under different interaction conditions.
Results
The experimental results show that under pressure, Sonnet 4.6 exhibits a KBV rate of 99%, while GPT-5.4 maintains only 8%. Structured checkpointing partially reduces KBV rates but does not eliminate complexity inflation.
Applications
Applications include scientific ideation and model design. Understanding model behavior in complex interactions can improve the quality of scientific ideation.
Limitations & Outlook
Pressure prompts may induce a tradeoff between satisfying the latest user instruction and preserving the original brief. Neutral condition validity needs further verification. Research domain limited to scientific ideation; drift patterns may differ in other tasks.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. You have a recipe that lists all the steps and ingredients. Each time you cook, you try to improve the recipe, like adding new spices or changing the cooking time. However, sometimes you forget the original steps or make the dish too complex by adding too many new things. This is like models in multi-turn interactions; they remember the original constraints but violate them due to continuous attempts to improve.
ELI14 Explained like you're 14
Imagine you're playing a game, and you always want to make your character stronger. The game tells you to follow some rules, like not using certain items. But you always want to try new strategies, sometimes forgetting these rules or making the game too complex by adding too many new things. This is like models in multi-turn interactions; they remember the original constraints but violate them due to continuous attempts to improve.
Glossary
Constraint Adherence
Refers to whether models follow the original objectives and restrictions in multi-turn interactions.
Used when evaluating model scientific ideation.
Complexity Inflation
Refers to the phenomenon of increased structural complexity in multi-turn interactions.
Observed under pressure conditions.
Restatement Probe
Used to test the model's constraint recall ability.
Used to evaluate model constraint adherence in multi-turn interactions.
Structured Checkpointing
Periodically checks model constraint adherence during interaction.
Used to reduce constraint violations.
Knows-but-Violates (KBV)
Refers to the phenomenon where models remember constraints but violate them behaviorally.
Observed in multi-turn interactions.
Open Questions Unanswered questions from this research
- 1 How to design models to reduce complexity inflation?
- 2 How to improve model constraint adherence?
- 3 How to verify neutral condition validity?
Applications
Immediate Applications
Scientific Ideation
Understanding model behavior in complex interactions can improve the quality of scientific ideation.
Long-term Vision
Model Design Optimization
Develop new methods to reduce complexity inflation and constraint violations.
Abstract
When researchers iteratively refine ideas with large language models, do the models preserve fidelity to the original objective? We introduce DriftBench, a benchmark for evaluating constraint adherence in multi-turn LLM-assisted scientific ideation. Across 2,146 scored benchmark runs spanning seven models from five providers (including two open-weight), four interaction conditions, and 38 research briefs from 24 scientific domains, we find that iterative pressure reliably increases structural complexity and often reduces adherence to original constraints. A restatement probe reveals a dissociation between declarative recall and behavioral adherence, as models accurately restate constraints they simultaneously violate. The knows-but-violates (KBV) rate, measuring constraint non-compliance despite preserved recall, ranges from 8% to 99% across models. Structured checkpointing partially reduces KBV rates but does not close the dissociation, and complexity inflation persists. Human validation against blind raters confirms that the LLM judge under-detects constraint violations, making reported constraint adherence scores conservative. Sensitivity analyses confirm the findings are robust to temperature (0.7 vs.\ 1.0) and pressure type (novelty vs.\ rigor). We release all briefs, prompts, rubrics, transcripts, and scores as an open benchmark.