CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context
CAPITU uses literary texts to design 59 automatically verifiable instructions, evaluating LLMs' instruction-following in Portuguese with cultural and morphological constraints.
Key Findings
Methodology
CAPITU employs literary-contextualized prompts with 59 instruction types, combining regex and lexicon matching for automatic verification. It covers Portuguese-specific morphology (e.g., -ando/-endo suffixes, diminutives) and structural constraints (e.g., acrostics). Tasks are generated via templates, with difficulty increasing through constraint combinations and multi-turn dialogues. Evaluation on 18 models shows high performance for state-of-the-art models (e.g., GPT-5.2 achieves 98.5% strict accuracy). Multi-turn assessments reveal challenges in constraint persistence, with accuracy ranging from 60% to 96%.
Key Results
- State-of-the-art GPT-5.2 achieves 98.5% strict accuracy on single-turn tasks, demonstrating strong reasoning and instruction adherence.
- Portuguese-optimized models like Sabiazinho-4 perform well at 87.0% accuracy at a low cost of $0.13 per query, while multi-purpose models like Claude-Haiku-4.5 reach 73.5% at $1.12.
- In multi-turn settings, accuracy varies significantly (60-96%), highlighting difficulties in maintaining constraints over dialogue turns.
Significance
This benchmark addresses a critical gap in evaluating Portuguese language models, integrating cultural and morphological constraints with automatic validation. It advances fair and robust evaluation, enabling better control of models in real-world applications involving cultural content and multi-turn interactions. The literary grounding enriches the contextual relevance, fostering models that understand language deeply within cultural norms.
Technical Contribution
Introduces a taxonomy of 59 instruction types tailored for Portuguese, with automatic verification via regex and lexicons. Implements multi-turn dialogue scenarios to test constraint persistence. Builds a literary corpus for contextualized prompts, and releases open-source tools and baselines, establishing a standardized evaluation framework for Portuguese NLP.
Novelty
First comprehensive instruction-following benchmark for Portuguese integrating literary context, morphological constraints, and automatic verification. It surpasses translation-based approaches by designing native, culture-specific instructions, providing a more authentic assessment of language models' capabilities.
Limitations
- Verification relies mainly on string matching, which may miss semantic nuances, especially in complex or nested constraints. Models struggle with nested morphological patterns and long texts.
- Multi-turn performance varies; models often lose constraint adherence over dialogue turns, indicating need for better memory and consistency mechanisms.
- Cost-effective but limited in capturing semantic and pragmatic complexities; future work should incorporate semantic understanding for more comprehensive evaluation.
Future Work
Future directions include expanding instruction types to cover more genres and pragmatic constraints, integrating semantic verification methods, and improving multi-turn memory mechanisms. Enhancing the robustness of constraint persistence over extended dialogues and exploring cross-cultural adaptation are also key goals.
AI Executive Summary
CAPITU represents a pioneering effort to evaluate instruction-following in Brazilian Portuguese, leveraging literary texts to embed cultural and morphological richness into automated benchmarks. Unlike prior multilingual benchmarks, CAPITU’s design emphasizes native language features such as suffixes (-inho, -mente) and structural constraints like acrostics, which are critical for authentic language understanding. The benchmark includes 59 instruction types organized into seven categories, covering word counts, morphological patterns, structural formats, and discourse markers, all verified automatically through regex and lexicon matching.
The prompts are contextualized within eight canonical Brazilian literary works spanning different eras and styles, providing a culturally grounded environment for model evaluation. This literary framing ensures that models are tested not only on linguistic accuracy but also on cultural comprehension, which is vital for real-world applications like content generation and conversational agents.
Evaluation on 18 models, including state-of-the-art and Portuguese-optimized variants, demonstrates that models like GPT-5.2 achieve near-perfect accuracy (98.5%) in single-turn tasks, while specialized models like Sabiazinho-4 perform efficiently at a fraction of the cost. Multi-turn experiments reveal significant variability in constraint maintenance, with accuracy ranging from 60% to 96%, exposing challenges in model memory and consistency.
This work advances the field by providing a rigorous, culturally-aware, and fully automated evaluation framework tailored for Portuguese. It highlights the importance of morphological and structural constraints in language control and sets the stage for future research in multilingual, multi-turn, culturally embedded NLP systems. The open-source release of the benchmark and baseline results aims to foster community-wide progress in this underserved language domain.
Deep Analysis
Background
The rapid development of large language models (LLMs) has transformed NLP, emphasizing their ability to follow instructions accurately. Early benchmarks like IFEval introduced verifiable constraints, mainly for English, focusing on format, keyword inclusion, and structural rules. As multilingual NLP advanced, datasets like Multi-IF extended evaluation to multiple languages, but often relied on translation, which inadequately captures language-specific features. Portuguese, with its rich morphology and discourse markers, remains underrepresented. Existing efforts like PoETa v2 and OAB focus on knowledge and reasoning but lack systematic, automatable instruction adherence evaluation. Consequently, there is a pressing need for native, culturally grounded benchmarks that incorporate language-specific constraints, which CAPITU aims to fulfill by integrating literary context and morphological features into an automated framework.
Core Problem
Current evaluation methods for Portuguese models are limited by translation-based prompts and lack of explicit, verifiable constraints tailored to Portuguese morphology and syntax. This results in models that perform well on knowledge tasks but struggle with strict structural adherence, especially in multi-turn dialogues. The absence of a comprehensive, automated, culturally relevant benchmark hampers progress in developing robust Portuguese NLP systems. Addressing this gap requires designing native instruction types that reflect linguistic complexities and cultural nuances, enabling precise measurement of models’ control and understanding capabilities in realistic scenarios.
Innovation
CAPITU’s core innovations include: 1) a taxonomy of 59 instruction types specific to Portuguese, covering morphological suffixes, discourse markers, and structural constraints; 2) literary-contextualized prompts based on eight Brazilian classics, embedding cultural relevance; 3) an automated verification system utilizing regex and lexicon matching, ensuring objective and scalable evaluation; 4) multi-turn dialogue scenarios that simulate real conversational settings, testing model consistency over multiple interactions. These innovations collectively enable a more authentic, culture-aware, and precise assessment of instruction-following in Portuguese, surpassing prior translation-dependent benchmarks and broadening the scope of multilingual NLP evaluation.
Methodology
- �� Instruction Design: Define 59 instruction types across seven categories, including morphological patterns (-inho, -mente), structural constraints (acrostics, line prefixes), and discourse markers. Use regex and lexicons for verification.
- �� Literary Context: Select eight canonical Brazilian literary works, covering diverse styles and eras, to generate prompts that reflect cultural richness.
- �� Prompt Generation: Use template-based systems with parameterized randomization to produce varied tasks, such as critical analysis, book reviews, and comparisons.
- �� Difficulty Scaling: Combine constraints in single, double, and multiple configurations, and implement multi-turn dialogues with progressive constraint accumulation.
- �� Verification: Implement deterministic functions for counting, pattern matching, lexicon validation, and structural checks, ensuring reproducibility.
- �� Evaluation: Measure strict and loose accuracy across 18 models, analyze performance patterns, and identify failure modes related to morphology, constraint persistence, and multi-turn robustness.
Experiments
The evaluation involved 18 models, including GPT-5.2, Sabiazinho-4, and Claude-Haiku-4.5, tested on 200 single-turn prompts and 100 multi-turn dialogues. The prompts were generated from literary templates with varying difficulty levels and constraints. Metrics included strict accuracy (all constraints met exactly), loose accuracy (allowing minor variations), and multi-turn constraint retention. Results showed GPT-5.2 reaching 98.5% in single-turn, while multi-turn accuracy varied significantly, exposing challenges in maintaining constraints over dialogue. Ablation studies confirmed the importance of morphological and structural instructions. The experiments validated the effectiveness of the benchmark in differentiating model capabilities and highlighted areas for improvement.
Results
GPT-5.2 achieved 98.5% strict accuracy in single-turn tasks, demonstrating superior reasoning and control. Specialized models like Sabiazinho-4 maintained 87.0% accuracy at low cost, outperforming larger models in efficiency. Multi-turn results varied, with some models dropping to 60%, indicating difficulties in constraint persistence. The analysis revealed that morphological and structural constraints are the most challenging, especially over multiple turns, emphasizing the need for models with better memory and consistency mechanisms.
Applications
CAPITU can be used to benchmark and improve Portuguese language models for applications such as AI writing assistants, educational tools, and conversational agents. Its focus on cultural and morphological constraints ensures outputs are contextually appropriate and linguistically accurate, vital for local content generation, legal document drafting, and cultural content curation. The benchmark also supports research in multi-turn dialogue systems, fostering models capable of sustained, coherent interactions in Portuguese.
Limitations & Outlook
While the benchmark offers comprehensive coverage of morphological and structural constraints, it relies primarily on deterministic string matching, which may overlook semantic nuances. Complex nested constraints and long texts pose challenges for current models, leading to decreased performance in multi-turn settings. Additionally, the evaluation does not incorporate deep semantic understanding, which could further improve robustness. Future work should integrate semantic validation and explore more sophisticated memory mechanisms to enhance multi-turn adherence.
Plain Language Accessible to non-experts
Imagine you’re in a kitchen where a chef has to follow very strict recipes. These recipes specify exactly how many ingredients to use, how to prepare each dish, and even the order of steps. The chef must follow these rules perfectly to make a good meal. CAPITU is like a set of these recipes, but for AI models that write texts in Portuguese. It checks if the AI follows all the rules correctly, like using certain word endings or starting sentences with specific letters. The recipes are based on famous Brazilian stories, so the AI learns to understand not just language rules but also cultural stories. This way, the AI becomes better at writing in a way that makes sense in Brazilian culture, just like a chef who follows a recipe to make a traditional dish. The system automatically checks the AI’s work, so there’s no need for a person to judge if it’s correct. This helps improve AI’s ability to follow instructions and understand culture at the same time.
ELI14 Explained like you're 14
Imagine you’re playing a game where you have to follow very specific rules to win. For example, you might need to write a story using only certain words, start each sentence with a particular letter, or include specific phrases. Now, think of a robot friend that’s really good at following these rules, but it needs to be tested to see how well it can stick to them. CAPITU is like a test for this robot, using famous Brazilian stories as the background. The rules are based on these stories, so the robot learns not just to follow instructions but also to understand Brazilian culture. The test checks if the robot can follow all the rules perfectly, even when the instructions get more complicated or when it has to keep the rules over several turns of a conversation. This way, we can see how smart and reliable the robot is when it comes to understanding language and culture, making it better at helping us with writing, chatting, or learning about Brazil.
Abstract
We introduce CAPITU, a benchmark for evaluating instruction-following capabilities of Large Language Models (LLMs) in Brazilian Portuguese. Unlike existing benchmarks that focus on English or use generic prompts, CAPITU contextualizes all tasks within eight canonical works of Brazilian literature, combining verifiable instruction constraints with culturally-grounded content. The benchmark comprises 59 instruction types organized into seven categories, all designed to be automatically verifiable without requiring LLM judges or human evaluation. Instruction types include Portuguese-specific linguistic constraints (word termination patterns like -ando/-endo/-indo, -inho/-inha, -mente) and structural requirements. We evaluate 18 state-of-the-art models across single-turn and multi-turn settings. Our results show that frontier reasoning models achieve strong performance (GPT-5.2 with reasoning: 98.5% strict accuracy), while Portuguese-specialized models offer competitive cost-efficiency (Sabiazinho-4: 87.0% at \$0.13 vs Claude-Haiku-4.5: 73.5% at \$1.12). Multi-turn evaluation reveals significant variation in constraint persistence, with conversation-level accuracy ranging from 60% to 96% across models. We identify specific challenges in morphological constraints, exact counting, and constraint persistence degradation across turns. We release the complete benchmark, evaluation code, and baseline results to facilitate research on instruction-following in Portuguese.