Large Model Strategic Thinking, Small Model Efficiency: Transferring Theory of Mind in Large Language Models
Fine-tuning small models improves their Theory of Mind task performance, achieving 46% alignment with large models.
Key Findings
Methodology
The study fine-tunes small language models to mimic large models in Theory of Mind tasks. Using LLaMa2-70b's responses and motivations dataset, LLaMa2-7b is fine-tuned, focusing on social dilemma games and contextual decision-making.
Key Results
- The fine-tuned small model showed a 46% improvement across all games, with 18% and 28% improvements in new contexts and games, respectively.
- The small model maintained significant alignment in unseen social contexts, demonstrating good generalization.
- In public goods games, the fine-tuned model exhibited higher pro-social behavior, indicating a better understanding of cooperation.
Significance
The study demonstrates the potential of fine-tuning small models to transfer Theory of Mind capabilities from large models, offering a more cost-effective alternative. This is significant for applications requiring complex social decision-making under resource constraints.
Technical Contribution
The technical contribution lies in demonstrating how to transfer strategic thinking capabilities from large to small models through fine-tuning, particularly in game theory and social context understanding. This provides new avenues for developing efficient, specialized AI models.
Novelty
This study is the first to show the transfer of Theory of Mind capabilities from large to small models through fine-tuning, especially in complex social decision-making tasks. It differs significantly from prior work that relied on direct application of large models.
Limitations
- The fine-tuned model occasionally exhibits overcorrection or bias amplification, potentially affecting stability.
- Lack of comprehensive testing in multi-player games limits applicability in more complex social interactions.
Future Work
Future research could explore the effects of fine-tuning small models in more diverse social contexts and game types, and develop more complex multi-player interaction scenarios to further validate model generalization.
AI Executive Summary
As large language models improve in Theory of Mind tasks, their deployment costs rise. This study proposes a method to transfer large model capabilities to small models through fine-tuning. Using a dataset of responses and motivations from LLaMa2-70b, LLaMa2-7b is fine-tuned, focusing on social dilemma games and contextual decision-making. Experimental results show a 46% improvement in the small model's performance across all games, with 18% and 28% improvements in new contexts and games, respectively. This indicates that fine-tuning not only effectively transfers large model capabilities but also offers a more cost-effective alternative. Although the fine-tuned model occasionally shows overcorrection or bias amplification, it maintains significant alignment in unseen social contexts, demonstrating good generalization. This provides new avenues for developing efficient, specialized AI models, particularly in applications requiring complex social decision-making. Future research could explore the effects of fine-tuning small models in more diverse social contexts and game types, and develop more complex multi-player interaction scenarios to further validate model generalization.
Deep Analysis
Background
Large language models (LLMs) excel in Theory of Mind (ToM) tasks but are costly to deploy. Researchers have been exploring ways to reduce costs while maintaining performance. Fine-tuning small models to transfer large model capabilities is a potential solution.
Core Problem
Large models perform well in strategic thinking tasks but are computationally expensive. The challenge is to reduce computational costs without significantly degrading performance.
Innovation
This study innovatively uses datasets generated by large models to fine-tune small models, enabling them to perform ToM tasks similarly to large models. This method reduces computational costs and demonstrates small models' generalization capabilities in unseen contexts.
Methodology
- �� Use LLaMa2-70b to generate a dataset of responses and motivations for 20 unique scenarios.
- �� Fine-tune LLaMa2-7b to perform similarly to the large model in social dilemma games.
- �� Evaluate the fine-tuned model's performance in unseen contexts and games.
Experiments
Experiments use LLaMa2-70b and LLaMa2-7b, generating and fine-tuning datasets for 20 social dilemma scenarios. Performance is evaluated by comparing pre- and post-fine-tuning results, assessing generalization in unseen contexts.
Results
The fine-tuned small model showed a 46% improvement across all games, with 18% and 28% improvements in new contexts and games, respectively. In public goods games, the fine-tuned model exhibited higher pro-social behavior.
Applications
This method can be applied in applications requiring complex social decision-making, such as automated negotiation and social robotics, particularly under resource constraints.
Limitations & Outlook
While the fine-tuned model performs well in most scenarios, it occasionally exhibits overcorrection or bias amplification. Additionally, lack of comprehensive testing in multi-player games limits applicability in more complex social interactions.
Plain Language Accessible to non-experts
Imagine you're working in a team that needs to make decisions. The large model is like an experienced team leader who can quickly analyze complex situations and make decisions but requires a lot of resources. The small model is like a new team member who, with guidance from the leader, can perform just as well in specific tasks. The study fine-tunes the small model to perform specific tasks like the large model, saving resources while effectively completing tasks.
ELI14 Explained like you're 14
Imagine you're playing a game that requires teamwork. The large model is like a game expert who can quickly find the best strategy but needs a lot of time and effort. The small model is like a newbie player who learns how to play better by watching the expert. The study is about making the small model learn the expert's strategies so it can also perform well in the game!
Glossary
Large Language Model (LLM)
A model based on deep learning capable of processing and generating natural language.
Used to generate response and motivation datasets.
Theory of Mind (ToM)
The ability to understand others' mental states, such as beliefs and intentions.
Used to evaluate model performance in social contexts.
Fine-tuning
The process of retraining a model on a specific dataset to improve its performance on specific tasks.
Used to enhance small model performance in ToM tasks.
Social Dilemma
A situation requiring a choice between individual and collective interests.
Used to test model's strategic decision-making ability.
LLaMa2-70b
A large language model with 70 billion parameters.
Used to generate fine-tuning datasets.
Open Questions Unanswered questions from this research
- 1 How can small models be effectively fine-tuned for multi-player games to better simulate complex social interactions?
- 2 Does the generalization capability of fine-tuned models vary across different cultural contexts?
Applications
Immediate Applications
Automated Negotiation
Fine-tuning small models to perform like large models in negotiations, saving computational resources.
Long-term Vision
Social Robotics
Develop robots capable of understanding and responding to complex social contexts, improving human-robot interaction.
Abstract
As the performance of larger, newer Large Language Models continues to improve for strategic Theory of Mind (ToM) tasks, the demand for these state-of-the-art models increases commensurately. However, their deployment is costly both in terms of processing power and time. In this paper, we investigate the feasibility of creating smaller, highly-performing specialized algorithms by way of fine-tuning. To do this, we first present a large pre-trained model with 20 unique scenarios that combine different social contexts with games of varying social dilemmas, record its answers, and use them for Q&A fine-tuning on a smaller model of the same family. Our focus is on in-context game-theoretic decision-making, the same domain within which human interaction occurs and that requires both a theory of mind (or a semblance thereof) and an understanding of social dynamics. The smaller model is therefore trained not just on the answers provided, but also on the motivations provided by the larger model, which should contain advice and guidelines to navigate both strategic dilemmas and social cues. We find that the fine-tuned smaller language model consistently bridged the gap in performance between the smaller pre-trained version of the model and its larger relative and that its improvements extended in areas and contexts beyond the ones provided in the training examples, including on out-of-sample scenarios that include completely different game structures. On average for all games, through fine-tuning, the smaller model showed a 46% improvement measured as alignment towards the behavior of the larger model, with 100% representing indistinguishable behavior. When presented with out-of-sample social contexts and games, the fine-tuned model still displays remarkable levels of alignment, reaching an improvement of 18% and 28% respectively.