Action-Aligned Retrieval with Pairwise Multimodal Reranking for Text-Based Person Anomaly Search
ActPair framework combines action-aligned retrieval and pairwise multimodal reranking, significantly improving text-based person anomaly search accuracy.
Key Findings
Methodology
The proposed three-stage ActPair framework includes: first, fine-tuning a vision-language model (VLM) with an action-aligned multi-task objective to enhance behavior semantics encoding; second, employing parallel late-fusion retrieval using original and scene-grounded rewritten queries, enriching semantic recall; third, implementing an off-the-shelf pivot-promote pairwise multimodal reranking module that compares candidate images directly, reducing computational costs. Experiments on the Pedestrian Anomaly Behavior (PAB) dataset show superior performance over state-of-the-art methods, with strong transferability to unseen datasets.
Key Results
- On PAB test, ActPair achieves R@1 of 88.62% and mAP of 93.97%, outperforming methods like IRRA and RaSa, especially in fine-grained ranking. Fine-tuning improves R@1 by 3.7%, demonstrating enhanced ordering of similar candidates.
- On RSTPReid, a non-anomaly dataset, R@1 reaches 55.25%, showing robust cross-domain generalization, validating the model’s sensitivity to behavioral cues.
- Ablation studies reveal that action-aligned multi-task learning and late-fusion contribute about 2-3% each, with pairwise reranking adding 1.5%, confirming the effectiveness of the integrated approach.
Significance
This work advances behavior understanding by focusing on fine-grained action semantics, addressing the pose-semantic gap that hampers traditional appearance-based methods. It broadens the application scope of vision-language retrieval to anomaly detection, surveillance, and security, providing a more nuanced and reliable recognition system. The integration of multimodal cues and efficient pairwise comparison sets a new standard for detailed behavior analysis in complex scenes.
Technical Contribution
Key innovations include the multi-task fine-tuning of VLM with explicit action supervision, scene-grounded query rewriting with late fusion, and a pivot-promote pairwise reranking algorithm. These components collectively improve behavior discrimination, recall, and computational efficiency, enabling scalable, fine-grained behavior retrieval. The framework bridges pose-semantic gaps and enhances the interpretability of behavior semantics, setting new benchmarks for multimodal anomaly search.
Novelty
This is the first comprehensive system integrating action-aligned multi-task learning, scene-aware query rewriting, and pairwise multimodal comparison within a coarse-to-fine pipeline for person anomaly search. Unlike prior works relying solely on appearance or pose, this approach explicitly models behavior semantics and scene context, achieving superior fine-grained discrimination and efficiency.
Limitations
- The approach depends on predefined action and scene tags, which may limit generalization to unseen behaviors or environments. Complex interactions like multi-person occlusion still pose challenges.
- Pairwise reranking, although efficient, remains quadratic in worst-case complexity, requiring further optimization for large-scale, real-time applications.
- The reliance on labeled action datasets and language models introduces potential biases, necessitating future research into unsupervised or weakly supervised methods.
Future Work
Future directions include integrating temporal dynamics from videos to capture motion patterns, developing unsupervised action semantics learning, and optimizing pairwise comparison algorithms for real-time large-scale deployment. Exploring broader scene understanding and multi-agent interactions will further enhance behavior anomaly detection capabilities.
AI Executive Summary
Detecting abnormal behaviors in surveillance footage is a critical challenge in public safety. Traditional methods relying on appearance features or static pose cues often fall short in capturing subtle, context-dependent actions. Existing approaches struggle with pose-semantic gaps, scene ambiguities, and computational inefficiencies, limiting their effectiveness in real-world scenarios.
To address these issues, this study introduces ActPair, a novel framework that unifies action-aligned retrieval with pairwise multimodal reranking. The system begins by fine-tuning a vision-language model (VLM) with an action-aware multi-task objective, which enhances the model’s ability to encode behavior semantics beyond mere pose. Next, it employs a parallel late-fusion retrieval strategy, combining the original query with a scene-grounded rewrite generated by a large language model (LLM), thus capturing both fine details and contextual cues.
The core innovation lies in the pairwise reranking module, which uses a pivot-promote algorithm to compare candidate images directly, rather than relying on absolute relevance scores. This approach significantly reduces computational costs while improving the discrimination of subtle behavioral differences. Extensive experiments on the Pedestrian Anomaly Behavior (PAB) dataset demonstrate that ActPair outperforms existing methods, achieving an R@1 of 88.62% and a mean Average Precision (mAP) of 93.97%. The model also exhibits strong transferability to unseen datasets like RSTPReid, confirming its robustness.
This work marks a substantial step forward in fine-grained behavior understanding, with broad implications for surveillance, security, and intelligent monitoring systems. Despite its success, challenges remain in handling complex multi-person interactions and scaling for real-time deployment. Future research will focus on incorporating temporal cues, unsupervised learning, and algorithmic optimization to further enhance performance and applicability.
Deep Dive
Abstract
Text-based person anomaly search requires distinguishing individuals based on fine-grained, context-dependent behaviors rather than mere appearance. Existing methods struggle to capture these context-conditioned actions, frequently relying on isolated skeletal geometry, discarding raw query details during reformulation, or utilizing absolute pointwise scoring for multimodal verification. To address these limitations, we propose \textbf{ActPair}, a unified three-stage coarse-to-fine framework that combines action-aligned retrieval with pairwise multimodal reranking to bridge the pose-semantic gap. First, we fine-tune a vision-language model (VLM) with an action-aligned multi-task objective that encourages the representations to encode action-discriminative semantics. Second, we perform parallel late-fusion retrieval using the original query and a large language model (LLM)-generated context-grounded rewrite, retaining complementary details from both semantic views. Finally, we propose an efficient off-the-shelf reranking module that leverages a pivot-promote algorithm to perform direct pairwise visual comparisons, mitigating residual spatial and compositional ambiguities without the prohibitive inference costs of exhaustive evaluation. Extensive experiments demonstrate that our framework achieves the best results among the compared methods on the Pedestrian Anomaly Behavior (PAB) public test and transfers effectively to an unseen, non-anomaly-specific dataset.