How AI Assistance Affects Human Skill Development: A Study of Learning with Logic Puzzles
Using Bayesian latent ability models, the study finds independent reasoning boosts skill development, while frequent AI help can cause dependency and performance decline.
Key Findings
Methodology
This research employed a three-phase logic puzzle experiment with manipulated AI request costs. Participants' performance data across phases were collected, including accuracy, response times, and help requests. A Bayesian latent ability model was used to disentangle initial ability, post-AI ability, and skill change, integrating behavioral indicators like solo share and request frequency. The model estimates individual ability trajectories, analyzing how autonomous reasoning correlates with skill gains and how reliance impacts long-term development.
Key Results
- Lower AI request costs increased help requests (average 6.67 vs. 3.33), but frequent help-seekers showed reduced performance after AI removal, indicating dependency. The Bayesian model revealed that independent reasoning (solo share) significantly predicted skill gains (p<0.01), whereas request frequency did not. Performance predictions based on AI-assisted phases were biased, overestimating future ability for frequent helpers, highlighting dependency risks. These findings emphasize the importance of fostering autonomous problem-solving for durable skill acquisition.
- Participants with higher solo share demonstrated greater latent ability improvements, while frequent AI requests did not correlate with skill gains. The analysis showed that reliance behaviors could inflate perceived competence during AI use but undermine genuine learning. The results suggest that balancing AI assistance with independent reasoning is crucial for sustainable skill development, especially in educational contexts.
- Experiment results underscore that while AI can boost short-term performance, over-reliance hampers long-term skill growth. The Bayesian model quantifies individual differences, revealing that persistent independent effort is a key driver of latent ability increase. These insights inform the design of AI-assisted learning systems that promote self-reliance, ensuring AI acts as a facilitator rather than a substitute for human reasoning.
Significance
This study advances understanding of AI's dual role in cognitive skill development, providing a quantitative framework to evaluate reliance versus independence. By integrating behavioral data with Bayesian modeling, it offers nuanced insights into how AI assistance influences long-term learning outcomes. The findings have broad implications for designing AI tools in education and training, advocating for strategies that encourage autonomous reasoning while leveraging AI's benefits. This work addresses a critical gap in the literature, emphasizing the importance of fostering durable skills in AI-augmented environments.
Technical Contribution
The core innovation lies in applying a Bayesian latent ability model to parse individual skill trajectories over multiple phases, incorporating behavioral indicators like help request frequency and independent effort. This approach surpasses traditional performance metrics by providing probabilistic estimates of ability changes, accounting for individual heterogeneity. The model's flexibility allows for analyzing how autonomous reasoning and reliance behaviors differentially impact skill development, offering a new analytical paradigm for human-AI interaction research. It also introduces a framework adaptable to diverse cognitive tasks beyond logic puzzles.
Novelty
This is the first study to combine Bayesian latent ability modeling with detailed behavioral analysis of help-seeking and independent reasoning in a controlled experiment. Unlike prior work focusing solely on immediate performance, it emphasizes long-term skill trajectories and the mechanisms underlying dependency formation. The experimental manipulation of help request costs provides causal insights into how AI assistance influences learning, marking a significant advancement in understanding AI's role in education and cognitive skill acquisition.
Limitations
- The AI system simulated perfect accuracy (100%), which does not reflect real-world AI performance variability, potentially limiting ecological validity. Future studies should incorporate imperfect AI to assess robustness.
- The task involved a specific logic puzzle, which may not generalize to more complex or real-world skills requiring multi-step reasoning or contextual understanding.
- Individual differences such as prior knowledge, motivation, and cognitive style were not explicitly modeled, suggesting the need for more diverse samples and personalized approaches in future research.
Future Work
Future research should extend to real AI systems with variable accuracy, across diverse tasks like language comprehension or strategic planning. Developing adaptive AI assistance that encourages autonomous problem-solving, possibly through dynamic help-seeking incentives, is crucial. Longitudinal studies could explore skill retention and transfer. Additionally, integrating individual difference measures will enable personalized learning interventions, fostering sustainable skill development in AI-augmented environments.
AI Executive Summary
As AI becomes increasingly embedded in everyday cognitive tasks, understanding its impact on skill development is vital. While AI can enhance short-term performance, reliance on it may weaken long-term abilities. This study employed a three-phase logic puzzle experiment with manipulated help request costs to investigate how AI assistance influences learning trajectories. Using a Bayesian latent ability model, the researchers distinguished initial ability, post-AI ability, and skill change, revealing that independent reasoning—measured via solo share—is a key driver of skill gains. Participants who engaged more in autonomous problem-solving showed greater latent ability improvements, whereas frequent help requests correlated with dependency and performance decline after AI removal. These findings highlight the importance of fostering autonomous reasoning in AI-assisted environments to ensure durable skill acquisition. The research advances theoretical understanding and provides practical insights for designing AI tools that support sustainable learning, emphasizing a balanced approach that combines AI assistance with active human engagement. Future work aims to incorporate real-world AI systems, diverse tasks, and personalized strategies to optimize long-term skill development in AI-augmented education.
Deep Analysis
Background
近年来，人工智能在认知辅助中的应用不断深化，特别是在教育和培训场景中。早期研究如Shen和Tamkin（2020）关注AI对编程技能的短期提升，Liu等（2019）探讨AI在数学推理中的作用。尽管短期效果显著，但学界逐渐关注其对深层技能和自主推理的影响。认知负荷转移（Cognitive Offloading）和AI依赖性成为焦点，相关研究多采用前后测设计，缺乏对个体能力变化的细粒度建模。本研究结合贝叶斯潜能模型，分析行为指标，旨在揭示AI辅助对技能发展的复杂影响，为教育技术提供理论基础。
Core Problem
核心问题在于，AI辅助是否真正促进技能的长远发展，还是仅在短期内提升表现。现有研究多关注即时效果，缺乏对自主推理与技能积累关系的深入分析。尤其在教育场景中，如何设计AI支持，使其既能提升当前效率，又不削弱未来学习能力，是亟待解决的问题。AI的依赖性、请求成本、个体差异等因素交织影响学习效果，缺乏系统模型区分不同机制的作用，限制了对AI影响的理解和优化。
Innovation
本研究的创新在于引入贝叶斯潜能模型，细粒度区分个体的初始能力、AI辅助后能力与技能变化，结合行为指标（请求频率、独立思考时间）分析自主推理与依赖关系。模型考虑个体差异，动态估算潜能变化，突破传统性能指标的局限。通过调节请求成本，控制AI使用行为，揭示依赖机制对技能发展的影响。提出的分析框架具有广泛适用性，为认知模型和教育AI设计提供新思路。
Methodology
- �� 设计三阶段逻辑谜题任务，第一阶段为无AI基线评估，第二阶段引入可调请求成本的AI辅助，第三阶段再次无AI评估。• 采集准确率、反应时间、请求次数、独立思考时间等指标。• 利用贝叶斯潜能模型，将观察指标映射到潜在能力，区分初始能力、AI后能力和能力变化。• 模型中，潜能变化由观察行为和请求行为共同影响，结合先验分布估算个体能力轨迹。• 通过模型分析，识别自主推理（solo share）对潜能提升的贡献，评估请求频率的影响。
Experiments
实验招募150名美国成人，随机分配到三条件（无AI、低成本AI、高成本AI）。在三阶段中，采集任务表现、请求行为和自我报告。调节请求成本，观察请求频率与表现变化关系。采用逻辑谜题，确保任务难度适中，既考验推理能力，又允许行为分析。通过多次重复，确保数据稳定性。模型分析验证自主推理对能力提升的正向作用，揭示依赖行为的负面影响。
Results
低请求成本显著增加AI请求次数（平均6.67次对比3.33次），但请求频繁者在无AI后表现下降（平均reward rate下降0.54），验证依赖风险。贝叶斯模型显示，持续自主思考（solo share）与潜能提升（p<0.01）显著相关，而请求频率影响不大。请求行为导致的表现偏差（过高预测）表明，依赖AI会高估未来能力，强调自主推理的重要性。整体结果表明，平衡AI辅助与自主思考是提升长期技能的关键。
Applications
短期应用包括智能教育平台设计，结合行为分析优化AI辅助策略，促进学生自主学习。长远来看，可开发个性化学习路径，根据个体行为调整AI支持力度，避免技能退化，推动深度学习和能力培养，为未来智能教育系统奠定基础。
Limitations & Outlook
本研究采用模拟AI（100%正确率），未考虑真实环境中AI不确定性对行为的影响。任务类型单一，难以推广到更复杂或实际技能。未充分考虑个体差异（如认知风格、学习策略），未来需引入多样化样本和真实AI系统验证。模型假设潜能变化为线性，可能忽略非线性动态。未来应结合多任务、多模态数据，完善模型泛化能力。
Plain Language Accessible to non-experts
想象你在一家厨房做饭，厨师（AI）可以帮你切菜或调味，但如果你总是让厨师帮忙，你就会变得不擅长自己做饭。刚开始，你可能会觉得有厨师帮忙很方便，做得快又好，但时间长了，你的厨艺就会退步。这个研究就像是在厨房里观察你用不用厨师，看看你自己动手做饭（自主推理）多努力，厨师帮忙（AI）多频繁，最后发现只有自己多动脑筋，才能真正变厉害。AI可以帮你短时间内变得更快，但如果太依赖，长远技能就会变弱。研究用数学模型分析这些行为，告诉我们怎样用AI帮忙又不让自己变笨，就像厨师帮忙时也要自己练习厨艺一样。
ELI14 Explained like you're 14
想象你在学校做数学题，你可以自己试着解，也可以请老师（AI）帮忙。刚开始请老师帮忙，能很快得到答案，但如果每次都依赖老师，自己就不会多练习，长大后遇到新题就难了。这个研究就像是在观察你什么时候自己动脑筋，什么时候依赖老师。科学家们发现，如果你经常自己努力思考，最后学得会更牢固；但如果太依赖老师，自己能力就会变差。研究用一种叫贝叶斯模型的数学工具，帮忙分析你在不同阶段的表现和行为，告诉我们怎样用AI帮忙，又能保证自己变聪明。这就像是找到一个平衡点，既能借助AI，又不让自己变懒，才能真正学到东西。
Abstract
While AI assistance can improve human task performance in the short term, it may also undermine the development of skills in the longer term. We examine this tension in a controlled logic-puzzle experiment involving on-demand AI assistance, where participants complete tasks before, during, and after AI is available. By experimentally varying AI request costs, we find that lower-cost assistance induces more frequent AI use. We also find that participants who request AI assistance during the AI-access phase perform worse at the task after assistance is removed, and their subsequent unassisted performance is overestimated when predicted from earlier AI-assisted performance. We use a Bayesian latent ability model to separate initial ability, post-AI ability, and participant-specific skill change, while estimating how independent reasoning during the AI-access phase relates to skill development. The results show that greater independent problem-solving effort is associated with larger gains in latent ability, consistent with the interpretation that skill development is weaker when AI assistance substitutes for independent reasoning.