When Many-Shot Prompting Fails: An Empirical Study of LLM Code Translation
Study finds many-shot prompting degrades functional correctness in code translation; optimal examples are 5-25.
Key Findings
Methodology
The study conducted large-scale experiments to evaluate the impact of prompting from zero to 625 examples using the Gemini model family. It analyzed static and dynamic performance metrics, focusing on functional correctness.
Key Results
- Pass@1 metric shows functional correctness peaks at 5-25 examples, with Gemini 1.5 and 2.0 Flash reaching 61.8% at 25 examples.
- Static metrics like BLEU and CodeBLEU show inconsistent behavior in many-shot settings, with Gemini 2.0 Flash peaking at 125 and 25 examples.
- Many-shot prompting increases functional and runtime errors, reducing translation success rates.
Significance
The study challenges the universality of many-shot prompting in in-context learning, highlighting the importance of a few high-quality examples in code translation. This has significant implications for effectively leveraging LLMs in software engineering.
Technical Contribution
The study reveals the limitations of many-shot prompting, proposing that in code translation, the quality of examples outweighs quantity, offering new directions for optimizing prompting strategies.
Novelty
This is the first systematic revelation of the 'many-shot paradox' in code translation, contrasting with other natural language tasks.
Limitations
- The study is limited to the Gemini model family; performance of other models remains unclear.
- Randomly chosen examples may introduce noise, affecting results.
Future Work
Future research could explore the performance of other model architectures and optimize example selection strategies to improve translation quality.
AI Executive Summary
Recent advancements in large language models (LLMs) have opened new possibilities for in-context learning, particularly in complex tasks like code translation. Traditionally, it is believed that providing more examples enhances model performance. However, this study, through analysis of 90,000 code translation experiments, reveals a 'many-shot paradox': in code translation, functional correctness peaks with 5 to 25 examples, while more examples degrade performance.
The study utilized the Gemini model family to evaluate the effect of prompting from zero to 625 examples. Results show that while static similarity metrics improve with more examples, functional correctness significantly declines in many-shot settings. This finding challenges the 'more is better' assumption, emphasizing the importance of a few high-quality examples in code translation.
These findings have significant implications for software engineering, suggesting that when leveraging LLMs for code translation, the focus should be on the quality rather than the quantity of examples. Future research could further optimize example selection strategies and explore the performance of other model architectures to enhance functional correctness and economic efficiency.
Deep Analysis
Background
With the development of large language models (LLMs), in-context learning has become possible, especially in complex tasks like code translation. Traditionally, it is believed that providing more examples can improve model performance, but in code translation, functional correctness is more important than static similarity.
Core Problem
Code translation requires deep semantic understanding to ensure that the generated code is not only syntactically correct but also functionally equivalent. This strict requirement makes code translation a challenging testbed for in-context learning.
Innovation
The study is the first to reveal the 'many-shot paradox' in code translation, where functional correctness peaks with a few examples, and more examples degrade performance, challenging the 'more is better' assumption.
Methodology
- �� Used Gemini model family for 90,000 translation experiments
- �� Evaluated prompting effects from zero to 625 examples
- �� Analyzed static and dynamic performance metrics, focusing on functional correctness
Experiments
The experiments were based on the CodeTransOcean benchmark, involving 30 translation pairs of six programming languages. Each pair randomly sampled 200 source code snippets, totaling 90,000 translation experiments.
Results
Functional correctness peaks at 5-25 examples, with Pass@1 showing 61.8% at 25 examples. Many-shot prompting increases functional and runtime errors, reducing success rates.
Applications
The findings have significant implications for effectively leveraging LLMs in software engineering, suggesting a focus on the quality rather than the quantity of examples in code translation.
Limitations & Outlook
The study is limited to the Gemini model family; performance of other models remains unclear. Randomly chosen examples may introduce noise, affecting results.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. You have a recipe that tells you what ingredients you need and the steps to follow. Now, imagine you have many recipes, but each is slightly different. You might think having more options is good, but it can actually be confusing, making it hard to choose the right recipe. Similarly, in code translation, providing too many examples can confuse the model, reducing the quality of the translation.
ELI14 Explained like you're 14
Imagine you're playing a game with lots of levels. You need to choose the right tools to pass each level. Now, if you have too many tool options, you might get confused and not know which one to use. Similarly, in code translation, giving too many examples can confuse the model, reducing the quality of the translation. A few high-quality examples are like the best tools in the game, helping you pass the levels smoothly!
Glossary
Large Language Model (LLM)
A type of AI model capable of processing large amounts of text data and generating natural language.
Used in in-context learning for code translation.
In-Context Learning (ICL)
A method where models learn new tasks by being provided examples directly in the prompt.
The study evaluated the effect of different numbers of examples on translation performance.
Many-Shot Prompting
A technique where a large number of examples are provided in the prompt to enhance model performance.
The study found that many-shot prompting may degrade functional correctness in code translation.
Functional Correctness
The ability of generated code to be functionally equivalent to the source code.
A key evaluation metric in the study.
Pass@1
A metric measuring the success rate of code translation in terms of compilation.
Used to evaluate functional correctness.
Open Questions Unanswered questions from this research
- 1 How to optimize example selection strategies without introducing noise?
- 2 How do other model architectures perform under many-shot prompting?
Applications
Immediate Applications
Software Engineering
Optimize prompting strategies in code translation to improve functional correctness and economic efficiency.
Long-term Vision
Cross-Platform Compatibility
Improve code translation techniques to achieve more efficient cross-platform software development.
Abstract
Large Language Models (LLMs) with vast context windows offer new avenues for in-context learning (ICL), where providing many examples ("many-shot" prompting) is often assumed to enhance performance. We investigate this assumption for the complex task of code translation. Through a large-scale empirical study of over 90,000 translations, we systematically evaluate the impact of scaling in-context examples from zero-shot to many-shot configurations of up to 625 examples, with prompts spanning from approximately 100,000 to 800,000 tokens. Our findings reveal a "many-shot paradox": while static similarity metrics may modestly improve with more examples, functional correctness consistently peaks with few-shot prompting (5-25 examples). Providing substantially more examples often degrades this crucial functional performance. This study highlights that for code translation, the quality of a few well-chosen examples outweighs sheer quantity, challenging the universal efficacy of "more is better" for ICL and underscoring the task-dependent nature of optimal prompting strategies. Our results have significant implications for effectively leveraging LLMs in software engineering.