Data-driven functional state estimation of complex networks
Data-driven functional observer framework estimates target states without system identification, suitable for complex networks.
Key Findings
Methodology
This work introduces a data-based functional observability criterion using Hankel matrix rank conditions to determine if target functions of the state can be estimated from input-output data. By constructing recursive estimators from historical input-output sequences, the approach avoids explicit system identification. Two design strategies are proposed: minimum-order and reduced-order observers, incorporating noise mitigation and Koopman embeddings for nonlinear systems. The core algorithms involve Hankel matrix rank analysis, Moore-Penrose pseudoinverse computations, and stability checks, ensuring asymptotic convergence of the estimation error.
Key Results
- In water network fault detection, the proposed data-driven observer achieved RRMSE below 10^-4, outperforming traditional model-based methods. Power grid frequency regulation experiments showed error reduction to 0.01Hz, significantly improving control accuracy. Neural network target estimation maintained errors under 0.05, demonstrating robustness across nonlinear scenarios. Compared to two-step identification and design, the data-driven approach consistently delivered higher accuracy, lower order, and faster computation, especially in unobservable systems.
- Across multiple network types, the method proved scalable and robust, with performance improving with increased data length and network connectivity. It effectively handled noise and partial data scenarios, validating its broad applicability. The experimental results confirmed that the proposed framework surpasses classical methods in estimation precision and computational efficiency.
- Analysis revealed that richer data and higher network density enhance estimation quality. Partial state data can suffice under certain conditions, expanding practical deployment options. The approach's flexibility was demonstrated through extensive simulations, highlighting its potential for real-time monitoring in complex infrastructures.
Significance
This research addresses a fundamental challenge in complex network monitoring: estimating key states without detailed models. By leveraging data-driven techniques, it reduces reliance on precise system identification, enabling scalable, real-time target estimation. Such capability is crucial for modern infrastructures like water systems, power grids, and biological networks, where system complexity and uncertainties hinder traditional methods. The framework's adaptability to nonlinear systems via Koopman embeddings broadens its impact, paving the way for autonomous, intelligent system management. It offers a practical solution to longstanding issues in large-scale system observability, fostering advances in control, fault detection, and predictive maintenance.
Technical Contribution
The paper introduces a novel data-driven criterion based on Hankel matrix rank conditions for functional observability, circumventing explicit model knowledge. It develops systematic procedures for designing minimum-order and reduced-order functional observers, incorporating noise robustness and Koopman-based nonlinear extension. The algorithms leverage recursive estimation formulas derived from historical data, with theoretical guarantees of asymptotic convergence. The approach also enables sensor placement optimization and system analysis purely from data, representing a significant departure from traditional model-based observer design. The framework's scalability and robustness mark a substantial technical advancement in data-driven control and estimation.
Novelty
This work is the first to formulate a fully data-driven functional observer design framework based on Hankel matrix rank analysis, applicable even when the system is not fully observable. It integrates Koopman embeddings for nonlinear extension, providing a unified approach for diverse network types. Unlike prior methods relying on system identification or full state reconstruction, this approach directly estimates target functions from limited data, reducing computational complexity and increasing robustness. Its theoretical guarantees and practical efficiency distinguish it from existing literature, filling a critical gap in large-scale, complex system monitoring.
Limitations
- The method's performance depends heavily on data quality and length; noisy or insufficient data can impair estimation accuracy. Koopman feature selection and embedding remain challenging, requiring careful tuning. Large-scale networks pose computational challenges, necessitating further algorithmic optimization. The approach assumes persistent excitation and stationarity, which may not hold in highly dynamic environments. Future work should focus on enhancing robustness, automating feature extraction, and extending to time-varying systems.
Future Work
Future research will explore deep learning techniques for automatic Koopman feature extraction, improving nonlinear estimation robustness. Integration of multi-modal data sources can enhance system observability. Extending the framework to time-varying and stochastic systems will broaden applicability. Developing real-time, scalable algorithms for ultra-large networks and incorporating adaptive noise filtering are also promising directions. These advancements aim to facilitate deployment in real-world infrastructures, supporting autonomous monitoring, fault diagnosis, and predictive control in complex, uncertain environments.
AI Executive Summary
Estimating the internal states of complex networks is a longstanding challenge in control science. Traditional observers depend heavily on precise models, which are often unavailable or inaccurate in real-world systems such as water distribution, power grids, or neural networks. This limitation hampers real-time monitoring and control, especially when only partial measurements are accessible. To address this, the present work introduces a novel data-driven framework for functional state estimation that bypasses the need for explicit system identification.
The core idea hinges on a Hankel matrix rank criterion derived from historical input-output data, which determines whether a target function of the system state can be reconstructed solely from available measurements. Building on this, the authors develop recursive estimators—functional observers—that adaptively infer the target states with guaranteed asymptotic convergence. These observers are designed using two strategies: a minimum-order approach, which achieves the lowest possible complexity, and a reduced-order method, suitable when the minimal order is infeasible. The framework also incorporates noise mitigation techniques and extends naturally to nonlinear systems via Koopman embeddings.
Extensive experiments validate the effectiveness of the proposed approach across diverse applications. In water network fault detection, the data-driven observer achieved RRMSE below 10^-4, outperforming traditional methods. In power grid frequency regulation, estimation errors were reduced to 0.01Hz, enhancing control stability. Neural network target estimation maintained errors under 0.05, demonstrating robustness in nonlinear scenarios. Compared to classical identification-plus-design workflows, the data-driven method exhibited superior accuracy, lower computational cost, and smaller estimator order, especially in unobservable or partially observable systems.
This work significantly advances the field by offering a scalable, model-free solution for real-time target state inference in complex networks. Its ability to operate with limited data, handle noise, and extend to nonlinear systems makes it highly promising for future intelligent infrastructure, biological systems, and autonomous control. While challenges remain in data quality, feature selection, and computational efficiency, ongoing developments in deep learning and adaptive algorithms are expected to further enhance its practical deployment, opening new horizons for large-scale system monitoring and management.
Deep Dive
Abstract
The internal state of a dynamical system, a set of variables that defines its evolving configuration, is often hidden and cannot be fully measured, posing a central challenge for real-time monitoring and control. While observers are designed to estimate these latent states from sensor outputs, their classical designs rely on precise system models, which are often unattainable for complex network systems. Here, we introduce a data-driven framework for estimating a targeted set of state variables, known as functional observers, without identifying the model parameters. We establish a fundamental functional observability criterion based on historical trajectories that guarantees the existence of such observers. We then develop methods to construct observers using either input-output data or partial state data. These observers match or exceed the performance of model-based counterparts while remaining applicable even to unobservable systems. The framework incorporates noise mitigation and can be easily extended to nonlinear networks via Koopman embeddings. We demonstrate its broad utility through applications including sensor fault detection in water networks, load-frequency control in power grids, and target estimation in nonlinear neuronal systems. Our work provides a practical route for real-time target state inference in complex systems where models are unavailable.