Peer-to-Peer Energy Trading in Dairy Farms using Multi-Agent Reinforcement Learning
Combining PPO and DQN in multi-agent RL enables P2P energy trading in dairy farms, reducing costs by 14.2% and peak demand by 55.5%.
Key Findings
Methodology
This study integrates Proximal Policy Optimization (PPO) and Deep Q-Networks (DQN) algorithms within a multi-agent system tailored for rural dairy farms. The framework incorporates auction-based market clearing via a double auction (DA), a price advisor agent, load, and battery management modules. Agents observe local states—load demand, renewable generation, battery status—and optimize energy transactions to minimize costs and peak demand while respecting dairy operational constraints. The market mechanism ensures privacy by exchanging only bids and public prices. The models are trained and validated on Irish and Finnish datasets, demonstrating superior performance over rule-based approaches, with cost reductions of 14.2% and 5.16%, and revenue increases of 7.24% and 12.73%. Peak demand reductions achieved by PPO (55.5%) and DQN (50%, 27.02%) significantly alleviate grid stress, showcasing the algorithms' adaptability and scalability in rural energy systems.
Key Results
- DQN reduces electricity costs by 14.2% in Ireland and 5.16% in Finland, while increasing revenue by 7.24% and 12.73%, respectively, outperforming traditional rule-based methods.
- PPO achieves a peak demand reduction of 55.5% in Ireland, with DQN reducing peak demand by 50% in Ireland and 27.02% in Finland, effectively mitigating grid peak loads.
- The auction-based market clearing and price advisor enhance trading efficiency, with models maintaining robustness across different regions and seasonal variations, confirming broad applicability.
Significance
This work advances rural energy management by integrating deep reinforcement learning with P2P market mechanisms, addressing high peak loads, cost inefficiencies, and environmental concerns. It demonstrates that intelligent, decentralized control can significantly improve energy utilization, reduce operational costs, and support sustainable rural development. The framework's validation across Irish and Finnish datasets underscores its robustness and potential for widespread deployment in rural microgrids, contributing to the global shift toward smart, green energy systems. The approach also offers a scalable solution for integrating renewable energy sources into rural communities, fostering economic growth and environmental sustainability.
Technical Contribution
The core technical innovation lies in combining PPO and DQN algorithms with auction-based market clearing and a privacy-preserving policy-to-market interface. The framework models dairy-specific operational constraints, such as synchronized peak loads and battery management, within a multi-agent RL setting. It introduces a price advisor agent that dynamically adjusts market prices, enhancing trading efficiency and revenue. The system employs multi-objective optimization to balance cost, peak demand, and revenue, demonstrating robustness across different regions and seasonal conditions. This integrated approach surpasses traditional rule-based and single-agent RL methods, providing a scalable, adaptive, and privacy-aware solution for rural energy trading.
Novelty
This is the first study to apply both PPO and DQN algorithms within a community-based P2P energy trading framework tailored for rural dairy farms. The integration of auction-based market clearing, a dynamic price advisor, and dairy-specific constraints represents a novel approach that addresses the unique operational challenges of rural energy systems. Unlike prior work limited to urban or residential settings, this research explicitly considers high-load, cyclical demands and rural grid limitations, offering a new paradigm for decentralized, intelligent energy management in agricultural communities.
Limitations
- The models rely on accurate load and generation forecasts; inaccuracies can reduce effectiveness, especially under unexpected operational or environmental changes.
- Training deep RL models involves high computational costs, limiting real-time deployment and scalability to larger communities without further optimization.
- Validation is limited to Irish and Finnish datasets; applicability to other regions with different grid conditions and market regulations requires additional adaptation.
Future Work
Future research will focus on enhancing computational efficiency for real-time deployment, extending the framework to multi-energy systems, and integrating IoT-based sensing for autonomous operation. Exploring multi-region coordination and market design improvements will further increase scalability and robustness. Additionally, incorporating user behavior models and demand response strategies can improve economic benefits and system resilience, paving the way for widespread adoption of intelligent rural microgrids.
AI Executive Summary
Rural dairy farms face significant energy management challenges, including high peak loads during milking and cooling, leading to elevated costs and grid stress. Traditional rule-based strategies lack adaptability in dynamic environments, necessitating more intelligent solutions. This study introduces a novel multi-agent reinforcement learning (MARL) framework that combines Proximal Policy Optimization (PPO) and Deep Q-Networks (DQN) algorithms with auction-based market mechanisms to optimize energy trading and consumption in dairy communities.
The framework models each farm as an autonomous agent observing local load, renewable generation, and battery status, while a central price advisor dynamically adjusts market prices. The auction mechanism ensures privacy and market efficiency, enabling agents to bid and ask energy quantities based on local conditions. The algorithms learn optimal strategies through trial-and-error interactions, balancing costs, peak demand, and revenue.
Experimental validation on Irish and Finnish datasets demonstrates the system's effectiveness: DQN reduces electricity costs by 14.2% in Ireland and 5.16% in Finland, while increasing revenue by over 7%. Peak demand is significantly curtailed—by 55.5% with PPO in Ireland, and by 50% and 27.02% with DQN in Ireland and Finland respectively—alleviating grid stress.
This innovative approach offers a scalable, privacy-preserving solution for rural energy management, supporting sustainable development and renewable integration. It opens pathways for deploying intelligent microgrids in agricultural settings, promoting economic savings, environmental benefits, and energy resilience. Future work will focus on real-time implementation, multi-region coordination, and broader energy system integration, advancing rural energy sustainability.
Deep Analysis
Background
随着可再生能源技术的快速发展，微电网和点对点（P2P）能源交易逐渐成为研究焦点。早期工作如G. K.等（2018）采用规则基础策略优化农村能源调度，但在高变异性环境中效果有限。深度强化学习（DRL）如DQN和PPO的引入，极大提升了智能调度能力，尤其在复杂、多变的市场环境中。多智能体系统（MAS）在智能电网中的应用逐步成熟，实现了分散控制和协作优化。现有研究多集中于城市或工业场景，农村奶牛场的高峰负荷、负载不稳定和信息有限等特殊性尚未充分解决。本研究结合P2P市场机制，创新性地将MARL应用于农村奶牛场，填补了该领域的空白。
Core Problem
农村奶牛场的能源管理面临多重难题：其高峰负荷集中在挤奶和冷却环节，导致电网压力大；能源供需不平衡，传统调度难以应对突发剩余或短缺；负载高度周期性，难以通过需求响应调节；此外，农村地区信息不对称和市场机制不完善，限制了能源的高效利用。这些问题严重制约农村能源的可持续发展，亟需智能化、适应性强的解决方案。
Innovation
本研究的核心创新包括：1）设计结合奶牛场特殊需求的MARL调度框架，考虑生产周期和储能特性；2）引入竞价市场清算机制，实现隐私保护和高效交易；3）开发价格顾问代理，动态调节市场价格，优化收益；4）结合深度强化学习算法（PPO和DQN）提升学习能力和适应性。这些创新共同解决了农村能源管理中的高峰控制、市场效率和隐私保护难题，显著优于传统规则和单智能体方法。
Methodology
- �� 构建多智能体系统，每个奶牛场作为独立代理，观察局部状态（负载、发电、储能）
- �� 设计基于PPO和DQN的强化学习模型，优化能源买卖策略
- �� 引入竞价机制，利用双边拍卖（DA）实现市场清算
- �� 设立价格顾问代理，调节市场价格以平衡供需
- �� 结合奶牛场的生产周期和储能限制，制定调度策略
- �� 通过观察市场价格、负载变化，持续学习和调整策略
- �� 采用多目标优化，兼顾成本、峰值需求和收益
- �� 在Irish和Finnish数据集上训练和测试模型，进行参数调优和性能评估
Experiments
采用Irish和Finnish地区的实际负载、发电和市场数据，设置不同的市场环境和季节条件。基线模型包括规则策略和单智能体强化学习。指标包括电费成本、峰值需求、售电收益和市场效率。超参数如学习率、折扣因子和交易频率经过调优。通过消融实验验证价格顾问和竞价机制的贡献，评估模型在不同地区、季节和规模下的表现。模型训练采用离线模拟，测试则在不同环境中进行，确保鲁棒性。
Results
结果显示，DQN在Irish地区降低电费14.2%，Finnish地区降低5.16%，同时售电收益提升7.24%和12.73%。PPO在Ireland峰值需求降低55.5%，DQN在两地区分别降低50%和27.02%。引入竞价和价格顾问机制显著提升交易效率和收益，模型在不同环境中表现出良好的适应性和扩展性。实验还验证了多智能体学习在高峰控制和成本优化中的优势，展示了未来农村微电网智能调度的潜力。
Applications
该模型适用于农村偏远地区的微电网管理，特别是奶牛场、养殖场等高峰负荷场景。依赖于负载、发电和市场数据的实时监测，结合智能调度算法，实现成本降低和峰值控制。可推广至其他农村能源系统，提升能源利用效率，减少碳排放，支持绿色农业发展。
Limitations & Outlook
模型依赖准确的负载预测和市场信息，实际应用中可能受数据不完整或误差影响。训练深RL模型涉及高计算成本，限制实时部署和大规模应用。模型主要在Irish和Finnish数据验证，跨地区推广需考虑环境差异和政策变化。未来需优化算法效率，增强模型的适应性和实用性。
Plain Language Accessible to non-experts
想象你在一个大厨房里做饭，每个厨师代表一个奶牛场，他们都需要用电来准备食材和冷藏牛奶。厨房里有一个智能助手（算法），可以观察每个厨师的需求和厨房的电力供应情况，帮他们决定什么时候用电、什么时候储存食物。这个助手还会和其他厨师交流，告诉他们什么时候可以一起用电，或者交换食材。通过这样合作，厨房里的每个人都能用更少的电费，避免同时用电过多导致电力紧张。这个系统就像用智能算法调度厨房的电力一样，帮助农村奶牛场合理用电，节省成本，同时保护环境。
ELI14 Explained like you're 14
想象你在学校的食堂里吃饭，每个学生代表一个奶牛场，他们都需要用电来做饭和冷藏食物。有时候，很多学生同时用电，电费就会变得很贵。现在，学校请来了一个聪明的机器人（算法），它可以观察每个学生的用电情况，帮他们安排用电时间，比如让一些学生在用餐高峰期少用电，或者让他们交换食材。这个机器人还会和学生们交流，告诉他们什么时候用电最便宜，什么时候可以多卖点电。这样一来，大家都能用更少的钱吃到好饭，还能帮忙减少电力紧张。就像这个机器人帮学校合理安排用电一样，研究里的算法帮农村奶牛场合理用电，既省钱又环保！
Abstract
The integration of renewable energy resources in rural areas, such as dairy farming communities, enables decentralized energy management through Peer-to-Peer (P2P) energy trading. This research highlights the role of P2P trading in efficient energy distribution and its synergy with advanced optimization techniques. While traditional rule-based methods perform well under stable conditions, they struggle in dynamic environments. To address this, Multi-Agent Reinforcement Learning (MARL), specifically Proximal Policy Optimization (PPO) and Deep Q-Networks (DQN), is combined with community/distributed P2P trading mechanisms. By incorporating auction-based market clearing, a price advisor agent, and load and battery management, the approach achieves significant improvements. Results show that, compared to baseline models, DQN reduces electricity costs by 14.2% in Ireland and 5.16% in Finland, while increasing electricity revenue by 7.24% and 12.73%, respectively. PPO achieves the lowest peak hour demand, reducing it by 55.5% in Ireland, while DQN reduces peak hour demand by 50.0% in Ireland and 27.02% in Finland. These improvements are attributed to both MARL algorithms and P2P energy trading, which together results in electricity cost and peak hour demand reduction, and increase electricity selling revenue. This study highlights the complementary strengths of DQN, PPO, and P2P trading in achieving efficient, adaptable, and sustainable energy management in rural communities.