Read, Grep, and Synthesize: Diagnosing Cross-Domain Seed Exposure for LLM Research Ideation
PaperGym enhances LLM research ideation through cross-domain seed retrieval and method synthesis.
Key Findings
Methodology
PaperGym employs a three-stage pipeline: tool-augmented seed extraction, cross-domain seed retrieval, and method synthesis. Seeds are extracted using read, grep, and bash tools in an isolated paper environment. Cross-domain retrieval is achieved by paraphrasing across seven ML domains. Finally, methods are synthesized and scored by rubric-based judges.
Key Results
- Cross-domain retrieval shows more novelty wins than no-retrieval and same-domain baselines, with 60% and 67% win rates, but no significant difference from a random diverse-seed control.
- Tool-augmented extraction improves seed specificity from 4.22 to 4.76.
- Paraphrase mode achieves complete cross-domain coverage, retrieving seeds spanning all seven domains for every problem.
Significance
The study indicates that LLM ideation systems benefit from diverse seed exposure but do not yet reliably exploit the semantic reasons for seed retrieval. This provides a diagnostic signal for future research on how to more effectively leverage cross-domain seeds to enhance research ideation.
Technical Contribution
PaperGym fundamentally differs from existing SOTA methods, particularly in its tool-augmented seed extraction and retrieval. It introduces new engineering possibilities, allowing for more precise seed extraction and broader domain coverage.
Novelty
This is the first systematic test of LLMs' ability to leverage cross-domain migration during research ideation. Compared to related work, PaperGym extends seed retrieval scope through tool augmentation and paraphrasing.
Limitations
- Current systems do not reliably exploit the semantic reasons for seed retrieval to enhance novelty.
- In some cases, the random diverse-seed control performs comparably.
Future Work
Future research directions include developing a rubric-rewarded RL system to train policies that exploit retrieval content rather than mere presence.
AI Executive Summary
In the field of machine learning, cross-domain technique migration is a key driver of innovation. PaperGym explores cross-domain seed exposure in LLM research ideation through a three-stage pipeline. First, tool-augmented seed extraction uses read, grep, and bash tools in an isolated paper environment to extract specific seeds. Next, cross-domain seed retrieval is achieved by paraphrasing across seven ML domains. Finally, methods are synthesized and scored by rubric-based judges. Experimental results show that cross-domain retrieval has more novelty wins than no-retrieval and same-domain baselines, but no significant difference from a random diverse-seed control. This indicates that LLM ideation systems benefit from diverse seed exposure but do not yet reliably exploit the semantic reasons for seed retrieval. PaperGym provides a diagnostic signal for future research on how to more effectively leverage cross-domain seeds to enhance research ideation.
Deep Analysis
Background
Innovation in machine learning often relies on cross-domain technique migration. For instance, gradient-based adversarial optimization used in vision was later applied to LLM jailbreaking. Similarly, SOLVENT demonstrated how human-annotated research paper labels can support cross-domain analogy retrieval. PaperGym's research background explores whether LLMs can systematically leverage such cross-domain migration in research ideation.
Core Problem
The core problem is whether current LLM ideation systems benefit from targeted cross-domain retrieval or simply from exposure to diverse mechanisms. This problem is important because understanding how LLMs generate ideas can help design more effective research tools.
Innovation
PaperGym's core innovation lies in its three-stage pipeline: tool-augmented seed extraction, cross-domain seed retrieval, and method synthesis. Each stage is carefully designed to maximize seed extraction specificity and retrieval domain coverage.
Methodology
- �� Tool-augmented seed extraction: Uses read, grep, and bash tools in an isolated paper environment to extract seeds.
- �� Cross-domain seed retrieval: Retrieves seeds by paraphrasing across seven ML domains.
- �� Method synthesis: Synthesizes methods and scores them using rubric-based judges.
Experiments
The experimental design evaluates PaperGym's performance on 30 research problems. Benchmarks include no-retrieval, same-domain retrieval, cross-domain retrieval, and a random seed control group. The experiments use 1,167 seeds extracted from 446 conference papers from 2017 to 2025.
Results
Experimental results show that cross-domain retrieval has more novelty wins than no-retrieval and same-domain baselines, with 60% and 67% win rates, but no significant difference from a random diverse-seed control. This indicates that LLM ideation systems benefit from diverse seed exposure.
Applications
PaperGym can enhance research ideation efficiency, particularly when cross-domain knowledge is required. It can help researchers quickly identify potential innovation directions.
Limitations & Outlook
A major limitation of PaperGym is that current systems do not reliably exploit the semantic reasons for seed retrieval to enhance novelty. Additionally, in some cases, the random diverse-seed control performs comparably. Future improvements can focus on better leveraging retrieval content.
Plain Language Accessible to non-experts
Imagine you're in a library looking for inspiration. PaperGym acts like a smart assistant, helping you extract interesting ideas from different books. First, it uses tools to find key sentences in the books, then searches for similar ideas on different shelves. Finally, it combines these ideas into a new story. This process helps you gather inspiration from various fields, not just the ones you're familiar with.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to collect clues from different levels to unlock new challenges. PaperGym is like your game assistant, helping you find important clues in each level and then combining them to unlock new game content. This assistant not only helps you find clues but also tells you why they're important. Isn't that cool?
Glossary
Seed Retrieval
The process of extracting and retrieving key concepts or methods from literature.
Used in PaperGym for cross-domain seed retrieval.
Tool-Augmented
The process of enhancing data extraction using tools like read, grep, and bash.
Used to improve seed extraction specificity.
Paraphrasing
Transforming a problem statement into the language and concept structure of different domains.
Used for cross-domain seed retrieval.
Method Synthesis
Combining retrieved seeds into a new research method.
The third stage of PaperGym.
Novelty
The innovation and uniqueness of a research method or result.
Used to assess the innovation level of synthesized methods.
Open Questions Unanswered questions from this research
- 1 How to more effectively leverage the semantic reasons for seed retrieval to enhance novelty?
- 2 Which factors most promote ideation in diverse seed exposure?
Applications
Immediate Applications
Research Ideation
Helps researchers generate new research directions based on cross-domain knowledge.
Long-term Vision
Automated Research Assistant
Develop intelligent assistants capable of automatically generating research ideas, enhancing research efficiency.
Abstract
The discovery of novel methodologies for emerging problems is a continuing cycle in ML, often driven by the migration of techniques across domains. Building on this observation, we ask whether current LLM ideation systems benefit from targeted cross-domain retrieval or simply from exposure to diverse mechanisms. We study this question through PaperGym, a three-stage pipeline: (1) tool-augmented seed extraction via read, grep, and bash over an isolated paper environment, (2) cross-domain seed retrieval via paraphrasing across seven ML domains, and (3) method synthesis from retrieved seeds, each scored by rubric-based judges. Tool-augmented extraction improves specificity, and paraphrase-based retrieval broadens domain coverage. In synthesis, cross-domain retrieval receives more pairwise novelty wins than no-retrieval and same-domain baselines, but shows no significant difference from a random diverse-seed control. These findings suggest LLM ideation systems benefit from diverse seed exposure, but do not yet reliably exploit the semantic reason particular seeds were retrieved. We release the seed library, rubric prompts, and run scripts at https://github.com/yunjoochoi/PaperGym