onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction
onPanda uses token-level correction to efficiently annotate on-policy data, reducing time by 52%.
Key Findings
Methodology
onPanda employs token-level correction as its core interaction mechanism. Annotators locate the first inappropriate token in a model response and either select a substitute from the model's candidate tokens or type the correct text. The system truncates everything after that position and continues generation from the corrected prefix until a satisfactory response is obtained.
Key Results
- onPanda reduces annotation time by 52%, with most tokens in the final response generated by the model, preserving the model's sampling distribution.
- Experiments demonstrate onPanda's effectiveness in maintaining on-policy data accuracy and providing fine-grained supervision.
- The release of the Panda-CVL dataset offers a benchmark for token-level correction, fostering community research.
Significance
onPanda significantly reduces the cost of annotating on-policy data and improves efficiency. By using token-level correction, annotators can precisely control model outputs with low cognitive load, making it suitable for constructing on-policy SFT and preference data.
Technical Contribution
onPanda introduces a new annotation tool capable of real-time correction of reasoning and tool calls across diverse environments. It records token-level corrections with precise positions, providing fine-grained supervision signals.
Novelty
onPanda is the first to achieve real-time token-level correction and tool call execution across multiple environments, addressing the limitations of existing tools that cannot correct and execute in real-time.
Limitations
- onPanda may require more human intervention when dealing with very complex dialogues.
- Proper configuration of model candidate tokens is needed to ensure correction accuracy.
Future Work
Future work could explore the application of onPanda to more multimodal data and how to further reduce the need for human intervention.
AI Executive Summary
onPanda is an interactive tool for efficiently annotating LLM alignment data and agent trajectories, using token-level correction as its core interaction mechanism. Annotators can correct inappropriate tokens by selecting candidate tokens or free-form editing. The system continues generating responses until a satisfactory result is achieved. Experiments show that onPanda can reduce annotation time by 52% while maintaining the accuracy of the model's sampling distribution.
The innovation of onPanda lies in its ability to correct reasoning and tool calls in real-time across diverse environments and execute corrected calls. The release of the Panda-CVL dataset provides a benchmark for the community, promoting research on token-level correction data. onPanda also supports multimodal and agent trajectory annotation, offering rich annotation data.
Despite its excellent performance in annotation efficiency and data quality, onPanda may require more human intervention when handling complex dialogues. Future work could explore how to apply onPanda to more multimodal data and further reduce the need for human intervention.
Deep Analysis
Background
With the development of large language models (LLMs), efficiently annotating on-policy data has become a key issue. Traditional annotation methods are costly and struggle to maintain the accuracy and granularity of on-policy data. onPanda offers a new solution through token-level correction.
Core Problem
Existing annotation workflows struggle to reduce costs while maintaining on-policy data accuracy. Manual editing and preference ranking methods have limitations and fail to provide fine-grained supervision signals.
Innovation
onPanda's core innovation is its token-level correction mechanism, allowing annotators to precisely control model outputs with simple clicks and edits. It achieves real-time correction and tool call execution in diverse environments.
Methodology
- �� Annotators locate the first inappropriate token.
- �� Select a substitute from model candidates or free-form edit.
- �� System truncates content after the position and continues generation from the corrected prefix.
- �� Repeat until a satisfactory response is obtained.
Experiments
Experiments used the Panda-CVL dataset, comparing onPanda with manual editing and preference ranking methods in terms of efficiency and accuracy. Results show onPanda excels in reducing annotation time and maintaining data quality.
Results
onPanda reduces annotation time by 52% while maintaining high-quality on-policy data. Most tokens are model-generated, ensuring data's sampling distribution.
Applications
onPanda is suitable for constructing on-policy SFT and preference data, especially in scenarios requiring precise control of model outputs.
Limitations & Outlook
onPanda may require more human intervention when handling complex dialogues. Future work could explore how to further reduce this need.
Plain Language Accessible to non-experts
Imagine you're in a factory where machines produce products. onPanda acts like a factory worker who checks the quality of each product. If a problem is found, they immediately correct it and continue production. This ensures each product meets standards without restarting the entire process.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to correct your character's wrong moves. onPanda is like a super assistant that helps you quickly find mistakes and offers the right move options. This way, you can complete game tasks faster without starting over.
Glossary
Token-Level Correction
Annotators can locate and correct inappropriate tokens in model-generated responses.
Core interaction mechanism of onPanda.
On-Policy Data
Data used to train models to generate responses that align with specific strategies or preferences.
onPanda is used to annotate such data.
Panda-CVL
A Chinese vision-language dataset annotated using onPanda.
Provides a benchmark for token-level correction.
SFT
Supervised Fine-Tuning, refers to fine-tuning models using annotated data.
On-policy data generated by onPanda is used for SFT.
Preference Data
Data used to train models to prefer generating certain types of responses.
Preference data generated by onPanda.
Open Questions Unanswered questions from this research
- 1 How to reduce human intervention in more complex dialogues?
- 2 How to further apply onPanda to multimodal data?
Applications
Immediate Applications
On-Policy Data Annotation
Researchers can use onPanda to efficiently annotate on-policy data, reducing time and cost.
Long-term Vision
Multimodal Data Annotation
onPanda can be extended to annotate more multimodal data, improving data quality.
Abstract
We present onPanda, an interactive tool for efficiently annotating LLM alignment data and agent trajectories. onPanda adopts token-level correction as its core interaction: while reading a model response, the annotator locates the first inappropriate token and either picks a substitute from the model's candidate tokens or types the correct text via free-form editing. The system then truncates everything after that position and continues generation from the corrected prefix, repeating this locate-correct-continue loop until a satisfactory response is obtained. This mechanism lets annotators precisely steer model outputs at low cost: a small controlled study suggests that onPanda reduces median annotation time by 52% over manual post-editing. Since the vast majority of tokens in the final response are generated by the model itself, the resulting data largely preserves the model's sampling distribution and is well suited for constructing on-policy SFT and preference data. Furthermore, the token-level corrections recorded during annotation provide fine-grained supervision with precise positions and naturally paired positive--negative samples. onPanda also connects to external tools and harnesses, enabling interactive trajectory annotation in realistic environments. In addition, we release Panda-CVL, a dataset annotated with onPanda, together with a benchmark for token-level correction.