"Like Taking the Path of Least Resistance": Exploring the Impact of LLM Interaction on the Creative Process of Programming
Study finds LLM-assisted programming shortens idea generation time but reduces creative moments.
Key Findings
Methodology
The study involved 20 programmers in a within-subject experiment to explore LLM's impact on programming creativity. Two conditions were tested: LLM-assisted and unassisted, using qualitative and quantitative analyses to identify four collaboration modes.
Key Results
- In LLM-assisted conditions, idea generation time significantly shortened (p=0.0004), but creative moments decreased (p=0.002).
- LLM-generated code was functionally and syntactically superior, but contained similar numbers of ideas as participant-generated code.
- Identified four collaboration modes: brainstormer, implementer, verifier, and co-pilot.
Significance
The study reveals LLM's impact on programming creativity, highlighting its support and limitations, and provides insights for designing more effective creativity support tools.
Technical Contribution
Introduced four collaboration modes and analyzed LLM's role in the creative process, offering suggestions for improving existing creativity support tools.
Novelty
First systematic study on LLM's impact on programming creativity, revealing its support and limitations across different creative stages.
Limitations
- LLM may lead to homogenization of ideas, reducing diversity.
- Small sample size may affect the generalizability of results.
Future Work
Future research could explore optimizing LLM use to enhance user creativity and diversity.
AI Executive Summary
With the advancement of AI technology, LLMs are increasingly used in programming. However, their impact on the creative process remains unclear. This study investigates the role of LLMs in programming creativity through experiments with 20 programmers. Results show that while LLM assistance shortens idea generation time, it also reduces creative moments. Although LLM-generated code is functionally and syntactically superior, it contains a similar number of ideas as participant-generated code.
The study also identifies four collaboration modes: brainstormer, implementer, verifier, and co-pilot. These modes support problem-solving strategies to varying degrees, especially when participants retain agency in idea generation, leading to more creative moments and diverse solutions.
Based on these findings, the study offers recommendations for designing more effective creativity support tools, emphasizing the importance of balancing cognitive effort and solution creativity by providing dynamic and selective support.
Deep Analysis
Background
With AI advancements, LLMs are widely applied in programming. Existing research mainly focuses on LLM's impact on idea generation, with less attention on the entire creative process. This study aims to fill this gap by exploring LLM's role in programming creativity.
Core Problem
The impact of LLM on the programming creative process is unclear, especially its support and limitations across different creative stages. The study aims to reveal how LLM affects idea generation and solution diversity in programming.
Innovation
This study systematically examines LLM's impact on programming creativity, introduces four collaboration modes, and analyzes LLM's support and limitations across different creative stages.
Methodology
- �� Participants: 20 programmers
- �� Experimental conditions: LLM-assisted and unassisted
- �� Data collection: screen recordings, code snippets, interviews
- �� Analysis methods: qualitative and quantitative analysis
Experiments
The experimental design includes two conditions: LLM-assisted and unassisted. Participants completed two programming tasks in each condition, including algorithmic and system design tasks. Data were collected through screen recordings and interviews.
Results
Results indicate that LLM-assisted conditions shorten idea generation time but reduce creative moments. LLM-generated code is functionally and syntactically superior but contains a similar number of ideas as participant-generated code.
Applications
The findings can inform the design of more effective creativity support tools, helping programmers maintain agency in idea generation when using LLMs.
Limitations & Outlook
LLM may lead to homogenization of ideas, reducing diversity. The small sample size may affect the generalizability of results.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. Usually, you need to come up with recipes, choose ingredients, and decide on cooking methods yourself. Now, there's a smart assistant that can help you make these decisions. While it can quickly offer suggestions, you might rely on it instead of thinking for yourself. This is similar to the study's finding that using LLMs in programming shortens idea generation time but reduces creative moments. The assistant saves time but may limit your creativity.
ELI14 Explained like you're 14
Imagine you're playing a game where you usually have to come up with strategies to win. Now, there's a super-smart assistant that can help you think of strategies. While it can quickly offer suggestions, you might rely on it instead of thinking for yourself. This is similar to the study's finding that using LLMs in programming shortens idea generation time but reduces creative moments. The assistant saves time but may limit your creativity.
Glossary
LLM (Large Language Model)
A deep learning-based model capable of generating human-like text.
Used as a tool for assisting programming in the study.
Creative Moment
Refers to a sudden new idea or insight during problem-solving.
Used to measure participants' creativity during programming.
Human-AI Collaboration Mode
Describes different ways humans and machines collaborate.
Four modes identified in the study: brainstormer, implementer, verifier, and co-pilot.
GitHub Copilot
An LLM-based code generation tool that can autocomplete code.
Used as the tool in the LLM-assisted condition of the study.
Qualitative Analysis
A research method that analyzes non-numeric data to understand phenomena.
Used to analyze participants' interview data.
Open Questions Unanswered questions from this research
- 1 How to optimize LLM use to enhance user creativity?
- 2 What are LLM's support and limitations across different creative stages?
Applications
Immediate Applications
Programming Education
LLMs can be used in programming education to help students quickly generate code and improve learning efficiency.
Long-term Vision
Software Development
LLMs can provide intelligent assistance in software development, enhancing development efficiency and creativity.
Abstract
Creativity is fundamentally human. As AI takes on more of the generative work that once required human imagination, despite documented limitations in creative ability, a critical question emerges: How does GenAI affect users' creativity? Through a within-subject study followed by retrospective interviews with (N=20) programmers, we investigated the impact of LLMs on participants' process of creative thinking in programming and the creativity of generated solutions. Across two conditions (LLM-assisted vs. unassisted), participants using LLMs had significantly shorter idea-generation periods (p=0.0004), leading to fewer creative moments (p=0.002). Qualitative analysis of participants' interactions and interviews revealed four different human-LLM collaboration modes supporting various problem-solving strategies. However, a comparative analysis of the generated solutions shows that while LLMs can help generate more correct and functional code, their solutions contain roughly the same number of ideas as participant-generated ones. Based on our findings, we discuss design implications and considerations for effectively using LLMs to support user creativity.