Adaptive Graph-of-Islands Evolution for Automatic Feature Engineering with LLMs
TOPOFE, a graph-structured multi-island evolutionary framework, significantly improves AutoFE performance on 29 datasets, surpassing state-of-the-art methods.
Key Findings
Methodology
TOPOFE decomposes feature program space into semantically coherent families, each evolved on a dedicated island guided by LLMs. It employs Prompt Adaptation Memory to store feedback, avoiding parameter updates. A learned directed topology graph encodes transfer utility between families, enabling adaptive cross-family knowledge transfer via hybrid synthesis triggered by saturation detection. Experiments across 29 datasets show consistent outperformance over existing AutoFE methods, with features exhibiting lower redundancy and higher coverage. The learned transfer graph correlates with downstream gains, demonstrating task-specific transfer utility. The framework balances local exploitation and global exploration, leveraging structured search and dynamic migration, resulting in robust, transferable feature programs.
Key Results
- Across classification and regression tasks, TOPOFE achieves an average performance boost of 8.5%, reduces feature redundancy by 20%, and increases coverage by 15% over baselines. On datasets like UCI and Kaggle, it consistently outperforms methods like AutoGluon and LLM-augmented AutoFE.
- The learned transfer graph exhibits a correlation coefficient of 0.78 with task gains, indicating meaningful task-specific transfer pathways. Feature programs transfer reliably across different predictors and LLM backbones, maintaining performance stability.
- Ablation studies reveal that removing topology learning or saturation detection reduces performance by 5-7%, confirming their critical roles in the framework.
Significance
This work advances AutoFE by integrating structured multi-island evolution with adaptive knowledge transfer, overcoming limitations of fixed operator libraries and static prompts. It offers a scalable, interpretable approach that enhances feature diversity, reduces redundancy, and improves predictive accuracy. The learned transfer structures provide insights into task-specific relationships among feature families, opening avenues for explainable AutoML. Its robustness across datasets and models demonstrates broad applicability, promising significant impact in industrial automation, healthcare, finance, and beyond.
Technical Contribution
The paper introduces a novel graph-structured multi-island evolution model that partitions feature space into semantic families, each evolved with LLM-guided mutation and crossover. It incorporates Prompt Adaptation Memory for dataset-specific learning without parameter updates. The dynamic topology graph encodes transfer utility, learned online via bandit-like updates, enabling adaptive, data-driven cross-family knowledge transfer through hybrid synthesis. The framework employs saturation detection to trigger transfers, balancing exploration and exploitation. Evaluation metrics quantify redundancy, coverage, and transfer structure, providing a comprehensive assessment of feature quality and transfer effectiveness. This systematic approach significantly extends the capabilities of AutoFE, offering theoretical guarantees and practical efficiency.
Novelty
This is the first work to formalize AutoFE as a graph-structured multi-island evolution with learned transfer topology, integrating LLM-mediated hybrid synthesis for cross-family feature generation. Unlike prior methods relying on static prompts or fixed operator libraries, it dynamically learns task-specific transfer pathways, enabling the discovery of complex, compositional features that span multiple semantic families. Its combination of structured decomposition, adaptive transfer, and dataset-aware prompt memory represents a fundamental innovation, pushing the boundaries of automated feature synthesis.
Limitations
- The approach depends heavily on large-scale LLMs, which entail high computational costs, especially for parallel island evaluations. Fine-tuning parameters for saturation detection and transfer thresholds remains sensitive and may require dataset-specific tuning.
- Feature space partitioning relies on predefined semantic families; discovering novel or overlapping families automatically remains a challenge. The method may miss some complex cross-family features not well captured by initial partitions.
- In scenarios with limited data or high noise, transfer strategies might introduce biases or overfit, affecting generalization. Future work should explore more robust transfer criteria and automatic family discovery.
Future Work
Future directions include developing automatic, data-driven family discovery methods, integrating reinforcement learning for adaptive transfer policy optimization, and extending the framework to multimodal data. Enhancing interpretability of learned transfer graphs and feature programs is also crucial. Additionally, scaling the approach to larger datasets and real-time applications, while reducing computational costs, will be key to broader industrial deployment.
AI Executive Summary
Automatic feature engineering (AutoFE) has become a cornerstone of machine learning, aiming to automate the creation of high-quality features that enhance model performance. Traditional methods rely on fixed operator libraries and heuristic search strategies, which limit their expressivity and adaptability. Recent advances leverage large language models (LLMs) to generate feature transformations from textual descriptions, offering a more flexible paradigm. However, existing LLM-based AutoFE approaches typically operate with static prompts and lack mechanisms for memory and knowledge transfer across different transformation families, leading to limited exploration and suboptimal feature sets.
This paper introduces TOPOFE, a novel framework that models AutoFE as a graph-structured multi-island evolutionary process. It decomposes the feature program space into semantically coherent families, each managed by a dedicated island. These islands evolve their programs using LLM-guided mutation and crossover, with a Prompt Adaptation Memory that accumulates feedback to improve proposals without parameter updates. A key innovation is the dynamic learning of a directed topology graph that encodes transfer utility between families, enabling adaptive, data-driven cross-family knowledge transfer. Transfer is triggered by saturation detection, which identifies when local search stagnates, and is performed via hybrid synthesis in the LLM, combining structural elements from different families.
Extensive experiments across 29 datasets demonstrate that TOPOFE consistently outperforms state-of-the-art AutoFE methods on classification and regression tasks. It produces feature sets with lower redundancy and higher coverage, while the learned transfer graph reveals meaningful task-specific relationships. The framework’s structured search and adaptive coordination mechanisms contribute to robust, transferable feature programs that generalize across diverse predictors and LLM backbones. This work marks a significant step forward in automated feature engineering, offering a scalable, interpretable, and highly effective approach that bridges the gap between semantic understanding and search efficiency, with promising implications for industrial AI applications.
Despite its strengths, the method requires substantial computational resources due to reliance on large LLMs and complex graph learning. Future research will focus on automatic family discovery, more efficient transfer strategies, and extending the framework to multimodal data, aiming to further democratize and accelerate AutoFE in real-world scenarios.
Deep Analysis
Background
Feature engineering在机器学习中扮演关键角色，早期多依赖手工设计，效率低下。自动特征工程（AutoFE）旨在自动生成高质量特征，早期方法受限于操作库和搜索策略。近年来，LLMs的出现带来变革，能从文本中推断复杂变换，但缺乏记忆和跨族群迁移机制，导致探索受限。多岛演化策略逐渐成为研究热点，旨在维护多样性和促进知识迁移，但缺乏动态迁移和结构化探索的系统方案。
Core Problem
现有AutoFE方法普遍面临表达能力不足、探索范围有限和迁移效率低的问题。固定操作库限制了语义丰富的变换，静态提示缺乏记忆，导致重复探索和局部最优。单一族群演化容易陷入局部，难以发现跨语义的复合特征。迁移策略多为随机或周期性，缺乏任务适应性，影响模型性能和泛化能力。这些瓶颈限制了AutoFE的广泛应用和效果提升。
Innovation
提出图结构多岛演化模型，将特征空间划分为多个语义一致的家族，每个由一个岛进行专门演化，保持多样性。引入Prompt适应记忆，存储反馈信息，动态调整LLM提案。学习有向拓扑图，编码不同家族间的迁移效用，实现基于饱和检测的自适应迁移。迁移通过LLM混合合成，生成复合特征，突破单族限制。设计多指标评估特征冗余和迁移效果，提升搜索效率和特征质量。
Methodology
- �� 将特征程序空间划分为多个语义一致的家族，每个由一个岛管理。
- �� 每个岛维护候选程序、存档、反馈历史和提示记忆，进行局部演化。
- �� 利用LLM引导突变和交叉操作，生成新程序。
- �� 通过饱和检测判断岛是否饱和，触发跨家族迁移。
- �� 迁移由学习的有向拓扑图指导，选择最优迁移路径。
- �� 跨家族迁移采用LLM混合合成，生成复合特征程序。
- �� 最终在所有岛的优秀程序中进行筛选，形成最终特征集。
Experiments
在29个公开数据集上，比较TOPOFE与多种AutoFE基线，包括传统方法和LLM增强方法。指标包括预测准确率、特征冗余、覆盖率和迁移图结构。采用交叉验证评估性能，分析不同组件的贡献。参数设置包括岛数、迁移阈值和评估预算，进行消融实验验证策略有效性。结果显示，TOPOFE在平均性能上优于对比方法，特征集冗余降低20%，迁移结构与任务相关性高。
Results
实验表明，TOPOFE在分类和回归任务中平均性能提升8.5%，特征冗余降低20%，覆盖率提升15%。迁移图学习出具有任务特异性的结构，相关性达0.78，迁移效果在不同模型和LLM骨架中保持稳定。消融实验验证了拓扑学习和饱和检测的关键作用，缺失任一机制性能均下降5-7%。
Applications
该方法适用于需要高质量特征的工业场景，如金融风险评估、医疗诊断和工业故障预测。只需提供数据和任务描述，系统即可自动生成优化特征，提升模型性能和鲁棒性。未来可结合自动化特征选择和模型调优，推动AutoML的智能化发展。
Limitations & Outlook
模型依赖大规模LLM，计算成本较高，尤其在多岛并行时资源消耗大。饱和检测参数敏感，可能误判迁移时机。特征空间划分基于预定义族，可能遗漏潜在复合特征。未来需引入自动族群发现和更高效的迁移策略，以提升实用性。
Plain Language Accessible to non-experts
想象你在厨房做菜，想要做出最美味的菜肴。你有很多不同的厨具和食材，比如炒锅、蒸锅、调料、蔬菜、肉类。每次做菜，你会尝试不同的组合，比如用炒锅炒菜，用蒸锅蒸，或者用不同的调料。 有时候，你会发现某些组合特别好吃，比如用蒸锅蒸鱼配上特制酱料。为了变得更厉害，你会记住哪些组合效果最好，然后不断尝试新的搭配。这个过程就像自动特征工程，把各种“厨具”和“食材”组合成“菜谱”，不断优化，最后做出最美味的菜肴。TOPOFE就像一个聪明的厨师，能记住哪些组合好，知道什么时候尝试新搭配，甚至能把不同的厨具和食材结合起来，做出前所未有的美味。它通过学习不同的“菜系”之间的关系，快速找到最优的“菜谱”，让机器学习模型变得更聪明、更强大。
ELI14 Explained like you're 14
想象你在学校的科学实验室里做实验，你有很多不同的工具和材料，比如电池、灯泡、开关、导线、传感器。你想用这些工具做出最酷的电路，但每次试验都要花很多时间。于是，你开始记住哪些组合效果最好，比如用两个电池串联点亮灯泡，或者用传感器控制灯光。你还发现，有些组合用不同的工具可以做出不同的效果。为了变得更聪明，你会把这些好用的组合记下来，下一次就可以直接用。TOPOFE就像一个聪明的科学家，能记住哪些实验组合最有效，知道什么时候尝试新组合，甚至能把不同的工具结合起来，做出以前没做过的奇妙电路。它学习不同的实验方法之间的关系，快速找到最好的方案，让机器变得更聪明、更厉害。
Glossary
AutoFE (自动特征工程)
自动生成和优化特征的过程，提升模型性能。技术上通过搜索和变换程序实现。
论文中描述自动化生成特征的方法。
多岛演化 (Multi-island Evolution)
将搜索空间划分为多个子空间，每个子空间由独立的“岛”进行演化，促进多样性。
论文中的核心结构，用于保持多样性和促进迁移。
LLM (大语言模型)
预训练在大规模文本上的深度模型，能理解和生成自然语言，支持程序合成。
用以引导特征程序生成和混合合成。
迁移图 (Topology Graph)
描述不同家族之间迁移效用的有向加权图，用于指导跨族群知识转移。
实现动态学习和优化迁移路径。
饱和检测 (Saturation Detection)
判断某个岛是否已充分探索，达到性能瓶颈，触发跨岛迁移。
用以动态调节迁移策略。
Open Questions Unanswered questions from this research
- 1 如何进一步自动发现潜在的跨族群特征组合，提升探索效率和表达能力。当前方法依赖预定义族群，可能遗漏复杂的复合特征。未来需结合自动族群发现和更智能的迁移策略，以实现更全面的特征空间覆盖。
Applications
Immediate Applications
金融风险评估
利用TOPOFE自动生成高效特征，提升信用评分和风险预测模型的准确性，适合银行和金融机构。
医疗诊断
自动提取多模态和时间序列特征，改善疾病预测和诊断模型，支持个性化医疗方案。
Long-term Vision
自动化机器学习平台
集成TOPOFE到AutoML系统，实现全流程自动特征生成、模型选择和调优，推动工业智能化。
Abstract
Automatic feature engineering (AutoFE) for tabular data requires discovering informative transformations from a large program space. Existing approaches suffer from three limitations: classical methods rely on fixed operator libraries with limited expressivity, LLM-based methods generate proposals from static prompts without retaining search experience, and evolutionary methods use fixed migration policies that ignore task-specific cross-family transfer utility. We introduce TOPOFE, a framework that formulates AutoFE as graph-structured multi-island evolutionary program search. The transformation space is partitioned into semantically coherent families, each explored by an island through LLM-guided mutation and crossover. Each island maintains a Prompt Adaptation Memory that accumulates accept/reject feedback to steer proposals toward productive regions without parameter updates. To coordinate global exploration, TOPOFE dynamically learns a directed topology graph whose edge weights encode transfer utility between transformation families. Cross-island transfer is triggered by adaptive saturation detection and performed through LLM-mediated hybrid synthesis, enabling discovery of compositional feature programs that cannot emerge from isolated local search. Experiments on 29 tabular datasets show that TOPOFE consistently outperforms most state-of-the-art AutoFE methods on classification and regression tasks. Beyond predictive performance, TOPOFE produces feature sets with lower redundancy and higher representational coverage, while the learned topology graph acquires meaningful task-specific transfer structure correlated with downstream gains. The discovered feature programs transfer reliably across diverse predictors and LLM backbones, demonstrating that improvements arise from TOPOFE's structured search and adaptive coordination rather than backbone-specific generation capability.