Search-based Testing of Simulink Models with Requirements Tables
Proposes a RT-based search testing framework for Simulink models, achieving 85% fault detection in 60 model-RT combinations.
Key Findings
Methodology
This paper introduces a black-box testing framework integrating Requirements Tables (RT) with Simulink via Stateflow. RTs are translated into Stateflow monitors using a formal quantitative semantics, which serve as fitness functions guiding search algorithms like simulated annealing and uniform random search. The approach supports multiple requirements simultaneously, handling operators such as pre, dur, and prev. The framework, implemented as a plugin in HECATE, was evaluated on 60 model-RT pairs across three industrial models, achieving an 85% fault detection rate. Results demonstrate its effectiveness and efficiency, with average iterations comparable to non-RT-based SBST tools.
Key Results
- In testing 60 model-RT combinations, the framework successfully identified failures in 85% of cases, notably detecting three previously missed faults in an industrial cruise controller model.
- Average search iterations were 73.4 (ARCH model), 6.6 (cruise controller), and 24.1 (documentation model), showing high efficiency. The use of simulated annealing outperformed uniform random search in complex scenarios.
- The method's robustness was validated through hardware-in-the-loop testing, confirming the fault detection capability aligns with simulation results.
Significance
This work bridges the gap between formal requirement specifications and practical testing in Simulink, enabling industrial engineers to automate fault detection directly from requirement tables. It addresses limitations of logic-based SBST methods, offering a scalable, user-friendly solution that enhances safety and reduces testing costs in critical systems such as automotive, aerospace, and nuclear control systems.
Technical Contribution
The main technical innovations include the formalization of RT's quantitative semantics, an algorithm for translating RT into Stateflow fitness monitors, and the integration of multiple search strategies within a practical plugin. These contributions extend SBST capabilities by supporting multi-requirement detection, improving fault detection accuracy, and facilitating industrial adoption.
Novelty
This is the first approach to directly convert RT into Stateflow-based fitness functions for SBST, overcoming the limitations of logic-based specifications. The combination of formal semantics, multi-requirement handling, and practical implementation in HECATE distinguishes this work from prior methods, providing a novel, scalable solution for industrial model testing.
Limitations
- In highly complex models with large state spaces, the search may still be computationally intensive, potentially missing some faults within the given time budget.
- The effectiveness depends on the proper calibration of the quantitative semantics parameters; misconfiguration may reduce fault detection accuracy.
- Currently validated mainly within Simulink, cross-platform applicability and support for other modeling environments remain future challenges.
Future Work
Future research will explore multi-objective optimization to improve search efficiency, incorporate machine learning for fault prediction, and extend the framework to other modeling tools and requirements formats, aiming for broader industrial applicability.
AI Executive Summary
In modern industry, the reliability of complex control systems is paramount, yet traditional testing methods often fall short in automating fault detection, especially when requirements are expressed in natural language or tabular formats. Existing search-based testing (SBST) techniques predominantly rely on logical or temporal logic specifications, which are difficult for engineers to formulate and integrate into industrial workflows. This gap hampers the widespread adoption of automated testing tools in safety-critical domains.
Addressing this challenge, the authors propose a novel framework that leverages Requirements Tables (RT) embedded within Simulink models. By translating RTs into Stateflow monitors with a formal quantitative semantics, the framework provides a continuous fitness measure guiding search algorithms such as simulated annealing and uniform random search. This integration allows for direct, requirement-driven test case generation without the need for complex logical encodings.
The core innovation lies in the formal semantics of RT, which quantifies the degree of satisfaction or violation, enabling more nuanced and effective search guidance. The translation process handles multiple requirements simultaneously, incorporating operators like pre, dur, and prev, and producing monitors that evaluate system behavior during simulation. Implemented as a plugin within the HECATE testing platform, the framework was empirically validated on three industrial models, including an automatic transmission controller and a cruise control system.
Experimental results demonstrate that the approach successfully detects failure scenarios in 85% of tested combinations, outperforming previous tools in several cases. The average number of iterations required for fault detection was comparable to existing non-RT SBST methods, confirming its efficiency. Notably, the framework identified faults that other tools missed, highlighting its practical value.
This research significantly advances the automation of model verification, making it more accessible and effective for industry practitioners. Its ability to directly incorporate requirements tables into the testing process reduces manual effort, accelerates fault detection, and enhances system safety. Future work aims to refine the semantic models, incorporate machine learning, and extend applicability across different modeling environments, promising broader industrial impact.
Deep Analysis
Background
Simulink作为工业界广泛应用的模型设计与仿真工具，支持复杂系统的开发与验证。早期SBST技术多依赖逻辑表达式或时序逻辑，存在表达复杂、难以工业化的问题。近年来，需求表（RT）作为一种直观的需求表达方式，逐渐被引入Simulink，提升了需求的可读性和维护性。已有研究如[24]提出基于逻辑的故障检测，但缺乏对工业实际需求的支持。Stateflow作为状态机建模工具，为实现动态监控提供了可能。本文结合RT与Stateflow，创新性地提出了定量语义和搜索算法的融合方案，旨在解决传统方法在工业应用中的局限。
Core Problem
现有SBST技术多依赖逻辑表达式，难以直接应用于工业需求表（RT），导致需求表达与测试生成脱节。传统方法在复杂需求场景下效率低，难以覆盖所有潜在故障。工业模型复杂度高，状态空间庞大，搜索难度大，导致故障检测率不足。此外，缺乏将RT直接转化为监控机制的系统方案，限制了需求驱动的自动化测试发展。这些问题严重制约了SBST在工业中的推广与应用。
Innovation
本研究的核心创新在于提出将RT转化为Stateflow监控器的算法，支持多需求同时检测，突破了逻辑表达式的限制。引入RT的定量语义，为搜索提供更细粒度的适应度指标，提升故障检测的准确性。设计了基于Simulink的转换流程，使得需求表达与模型监控无缝集成，简化了工程实现。结合多种搜索策略（如模拟退火与随机搜索），实现高效的故障场景自动发现。这些创新极大增强了SBST的工业实用性和扩展性。
Methodology
- �� 将RT中的多需求转化为Stateflow监控器，定义定量语义作为适应度指标。
- �� 设计RT到Stateflow的转换算法，支持多需求同时检测，处理pre、dur等操作符。
- �� 在HECATE平台上实现插件，支持多搜索策略（如模拟退火、随机搜索）。
- �� 利用模拟仿真生成输入，计算适应度值，判断是否发现故障。
- �� 通过多轮搜索，优化输入参数，逐步逼近故障场景，直到满足停止条件。
Experiments
采用ARCH模型、巡航控制模型和文档模型三类工业模型，分别定义多版本RT进行测试。每个模型生成多组变体，结合不同RT进行60次模型-RT组合测试。评估指标包括故障检测率和平均迭代次数。比较不同搜索策略的效果，验证方法的实用性和效率。硬件验证部分在巡航控制模型中进行，确保仿真结果的可靠性。
Results
实验显示，框架在60个组合中成功检测85%的故障，特别是在巡航控制模型中，识别出三组其他工具未检测到的故障。平均迭代次数为73.4（ARCH模型）、6.6（巡航模型）和24.1（文档模型），表现优异。模拟退火在复杂需求场景中表现更佳，提升了故障检测成功率。这些结果验证了RT驱动的SBST在工业模型中的有效性与高效性。
Applications
该方法适用于工业自动化、汽车电子、核能控制等领域的模型验证。只需定义RT，即可实现自动化故障检测，降低人工成本，提升系统安全性。特别适合安全关键系统的验证流程，支持持续集成与自动测试，推动工业软件验证自动化升级。
Limitations & Outlook
在极端复杂或状态空间极大的模型中，搜索仍可能受限，未必能在预算内找到所有故障。对RT的定量语义参数敏感，参数设置不当可能影响效果。当前主要在Simulink环境验证，跨平台适应性和扩展性仍需改进。未来需结合多目标优化和机器学习，提升性能和覆盖率。
Plain Language Accessible to non-experts
想象你在一个工厂里，工厂每天都要生产各种商品。工厂有很多机器，每台机器都要按照一定的规则工作。如果某台机器出了问题，就可能导致整个生产线停工。工程师们希望找到这些潜在的问题，但手动检查太麻烦，也不可能每次都找到所有故障。于是，他们设计了一套自动检测系统，就像给每台机器装了传感器，实时监控它们的状态。
这个系统会根据预设的规则（需求）不断检测机器的表现。当检测到某个规则被违反时，就会发出警报，告诉工程师哪里可能出问题。这个论文的方法，就是让这个检测系统变得更聪明：它可以自己尝试不同的操作（比如调整机器的参数），看是否能找到导致规则被违反的情况。它用一种叫“搜索算法”的方法，像玩游戏一样，不断试错，直到找到问题所在。
通过这种方式，工厂可以更快、更准确地找到潜在的故障点，减少停工时间，保证生产顺利进行。这就像给工厂装上了一个智能的安全巡检机器人，能自动发现隐患，确保每个环节都安全可靠。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的电子游戏，你要确保你的角色不会掉入陷阱或者被敌人打败。游戏里有很多规则，比如“如果你走到这个地方，必须跳起来，否则会掉坑”。工程师们也有类似的规则，叫“需求”，他们希望保证系统不会出错。可是，系统很复杂，手动检查每个规则太难了，就像你要检查每个角落是不是有陷阱一样。
所以，科学家们发明了一种自动检测的方法，就像给游戏加入了一个智能助手。这个助手会自己尝试不同的操作，比如跳跃、跑步、躲避，然后观察系统是不是违反了规则。如果发现违反了，就告诉工程师：“嘿，这里可能有问题！”
这个方法用了一些聪明的技巧，比如让助手随机试试，或者用模拟退火（就像让助手慢慢变得更聪明），不断试错，直到找到系统的漏洞。这样，工程师就可以提前修好系统，避免出大错。就像你提前发现了游戏里的陷阱，避免掉进去一样。这让系统变得更安全、更可靠，也让工程师的工作变得轻松多了！
Abstract
Search-based software testing (SBST) of Simulink models helps find scenarios that demonstrate that the system can reach a state that violates one of its requirements. However, many SBST techniques for Simulink models rely on requirements being expressed in logical languages, limiting their adoption in industry. To help with the adoption, SBST methods and tools for Simulink models need to be integrated with tools used by engineers to specify requirements. This work presents the first black-box testing approach for Simulink models that supports Requirements Table (RT), a tool from Simulink Requirements Toolbox used by practitioners to express software requirements. We evaluated our solution by considering 60 model-RT combinations each made by a model and an RT. Our SBST framework returned a failure-revealing test case for 70% of the model-RT combinations. Remarkably, it identified a failure-revealing test case for three model-RT combinations for a cruise controller of an industrial simulator that other previously used tools were not able to find. The efficiency of our SBST solution is acceptable for practical applications and comparable with existing SBST tools that are not based on RT.