Detecting and Guiding LLM-Generated Korean Poetry with Interpretable Form-level Features
Detecting and guiding LLM-generated Korean poetry using interpretable features, achieving 83.60 AUC-ROC.
Key Findings
Methodology
The study introduces four linguistic dimensions: output length, structure variation, rhythmic irregularity, and normative adherence, operationalized into five interpretable features for detecting and guiding LLM-generated poetry. Detection is achieved using a logistic regression classifier, and generation is guided by these features.
Key Results
- Detection results: Achieved an average AUC-ROC of 83.60 across seven unseen LLMs, improving by 7.76 points over the baseline KatFishNet.
- Generation results: Expert evaluations prefer feature-guided poems over unconstrained baselines.
- Feature validity: Removing any dimension lowers AUC-ROC, especially structure variation.
Significance
This research bridges the gap between human and LLM-generated poetry using interpretable features, offering a novel detection and generation guidance method with significant academic and industrial implications, especially in language-specific poetry generation.
Technical Contribution
Technically, the study provides a new detection framework that surpasses existing baseline methods and guides generation to make LLM-generated poetry closer to human form.
Novelty
This is the first application of language-specific interpretable features for detecting and guiding Korean poetry, differing from previous hard constraint methods.
Limitations
- Detection struggles with certain generator-specific punctuation patterns.
- Generation guidance still has limitations in normative adherence.
Future Work
Future research could explore optimizing feature-guided generation to better match human poetry in normative adherence.
AI Executive Summary
Large language models (LLMs) often struggle with generating modern Korean poetry, producing outputs that resemble "line-broken prose." Existing solutions fall short of addressing this issue effectively. This study proposes a novel approach using interpretable form-level features to detect and guide LLM-generated Korean poetry. The method quantifies the gap between human and LLM-generated poetry through four linguistic dimensions: output length, structure variation, rhythmic irregularity, and normative adherence, operationalized into five interpretable features. Experimental results show that a logistic regression classifier using these features excels in detection tasks, achieving an average AUC-ROC of 83.60, significantly improving over baseline methods. In generation tasks, feature guidance makes generated poems closer to human poetry, as supported by expert evaluations. Despite these achievements, the method has limitations in certain generator-specific punctuation patterns, and future research could further optimize feature-guided generation.
Deep Analysis
Background
With the development of large language models, their performance in generating poetry has attracted attention. Previous research primarily relied on hard constraints like meter and rhyme, which do not align with the free form of modern Korean poetry.
Core Problem
LLM-generated Korean poetry often lacks poetic quality, resembling "line-broken prose" without rhythm or poetic tension. Making LLM-generated poetry closer to human poetry is an important and challenging problem.
Innovation
This study proposes a new method for detection and generation guidance using interpretable form-level features to quantify the gap between human and LLM-generated poetry. These features include output length, structure variation, rhythmic irregularity, and normative adherence.
Methodology
- �� Introduce four linguistic dimensions and operationalize them into five interpretable features
- �� Use a logistic regression classifier for detection
- �� Guide generation using features to make poetry closer to human form
Experiments
Experiments used the KatFishNet dataset and newly collected poetry samples, evaluating detection performance on seven unseen LLMs and conducting expert evaluations for generation guidance.
Results
In detection tasks, classifiers using five features achieved an average AUC-ROC of 83.60 across seven unseen LLMs. In generation tasks, feature-guided poems were closer to human poetry in expert evaluations.
Applications
The method can be used to improve the quality of LLM-generated poetry, making it closer to human poetry, applicable to poetry creation and literary analysis.
Limitations & Outlook
Despite excellent detection performance, the method struggles with certain generator-specific punctuation patterns. Generation guidance still has limitations in normative adherence.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. An LLM is like a novice chef who knows how to make basic dishes but lacks creativity and variation. This study is like providing the chef with new spices and techniques to make dishes closer to those of a professional chef. By analyzing dish length, structure, rhythm, and norms, we help the chef improve their cooking skills, making the final dish more aligned with human tastes.
ELI14 Explained like you're 14
Imagine you're playing a game where the characters need to write poetry. An LLM is like a newbie player whose poems are always plain and unexciting. This study is like giving the player some cheat codes to write more poetic works. By analyzing poem length, structure, rhythm, and norms, we help the player enhance their poetry creation skills, making their works more like human-written poems.
Glossary
LLM (Large Language Model)
A type of AI model capable of generating text, typically trained on large datasets.
Used for generating Korean poetry.
AUC-ROC (Area Under Curve - Receiver Operating Characteristic)
A metric for evaluating classifier performance, higher values indicate better performance.
Used to evaluate detection task performance.
KatFishNet
A baseline method for detecting Korean LLM text using punctuation and spacing cues.
Compared with the method in this study.
Interpretable Features
Metrics used to quantify and analyze text features, aiding in understanding model outputs.
Used for detection and generation guidance.
Rhythmic Irregularity
Variation in line lengths within poetry, affecting the rhythm of the poem.
One of the features used for detection and generation guidance.
Open Questions Unanswered questions from this research
- 1 How to further optimize normative adherence in generation guidance remains unexplored.
- 2 The impact of generator-specific punctuation patterns on detection performance requires further study.
Applications
Immediate Applications
Poetry Creation
Assist poets and writers in using LLMs to generate works closer to human poetry.
Long-term Vision
Literary Analysis
Analyze language-specific poetry features through LLM-generated poetry, advancing literary research.
Abstract
LLMs often struggle with modern Korean poetry, producing outputs that resemble "line-broken prose." We address two coupled tasks: detecting whether a Korean poem is human- or LLM-authored, and guiding LLMs to generate poetry closer in form to human writing. We quantify the human-LLM gap along four form-level linguistic dimensions: output length (Volume), the diversity and connective use of line-final forms (Structure Variation), the irregularity of line lengths (Rhythmic Irregularity), and adherence to standard orthography (Normative Adherence). We operationalize these dimensions as five interpretable features. For detection, a logistic regression classifier over these five features attains an average AUC-ROC of 83.60 in zero-shot out-of-distribution detection across seven unseen LLMs, versus 75.84 for the strongest baseline in our comparison, KatFishNet, an absolute gain of 7.76 AUC points and a 10.23% relative improvement; one generator-specific punctuation pattern outside our taxonomy remains a boundary case. For generation, expert evaluation on GPT-5.2 prefers feature-guided poems over the unconstrained baseline, and analyses across GPT-5.2 and Gemini-3 show that targeted length, rhythm, and ending statistics move toward the human distribution. These results suggest that interpretable, language-specific features can bridge the diagnosis and guidance of LLM-generated poetry.