Scene Text Detection and Recognition "in light of" Challenging Environmental Conditions using Aria Glasses Egocentric Vision Cameras
Combining EAST+CRNN with image upscaling reduces CER from 0.65 to 0.48 in egocentric scene text recognition under challenging conditions.
Key Findings
Methodology
This study employs Meta Project Aria smart glasses to collect scene text images under varied lighting, distance, and resolution conditions. A custom dataset was built, with text detection via EAST, followed by heuristic merging of character boxes. OCR was performed using CRNN and PyTesseract. The analysis focused on how environmental factors influence Character Error Rate (CER), evaluated according to ICDAR 2024 standards. Image upscaling significantly improved recognition, reducing CER from 0.65 to 0.48. Eye-gaze data was integrated to focus detection on user attention zones, optimizing processing efficiency.
Key Results
- Resolution and distance significantly impact recognition accuracy; low resolution (1408×1408) at 1m distance caused CRNN to fail, but upscaling reduced CER to 0.48, a 26% improvement.
- PyTesseract demonstrated more stable performance across conditions, with less CER variation, but overall accuracy was lower than CRNN. Lighting variables showed weak correlation with CER.
- Preprocessing, especially image enlargement, effectively mitigated low-quality image issues, notably in distant, low-resolution scenarios.
Significance
This work is the first systematic evaluation of environmental impacts on egocentric scene text recognition using wearable cameras. It demonstrates how combining image preprocessing and gaze-based region focusing enhances robustness, paving the way for personalized AR applications like asset inspection and nutrition analysis. The findings address long-standing challenges of environmental variability, contributing to the development of adaptive, user-aware recognition systems in real-world settings.
Technical Contribution
The paper introduces a multi-stage framework combining EAST detection with heuristic character box merging, augmented by image upscaling and gaze-based region selection. This approach improves robustness against low resolution and distance, and integrates multimodal data for efficiency. It advances scene text recognition by emphasizing environmental adaptation and user-centric processing, offering a practical solution for wearable AR devices.
Novelty
This is the first comprehensive study on the influence of environmental variables on scene text recognition in egocentric vision captured by AR glasses. It innovatively combines detection, heuristic merging, image upscaling, and gaze tracking, demonstrating significant performance gains in real-world conditions, surpassing prior static dataset evaluations.
Limitations
- Limited dataset size restricts generalizability; scenarios with complex backgrounds or dynamic scenes were not extensively tested.
- Gaze data retrieval introduces latency, hindering real-time application; offline processing was used in experiments.
- Preprocessing strategies need further refinement for diverse environments; model robustness across varied real-world conditions remains a challenge.
Future Work
Future efforts will expand dataset diversity, including dynamic scenes and complex backgrounds. Developing real-time processing pipelines with edge computing and optimizing gaze detection for instant feedback are key directions. Further research will explore multimodal fusion, adaptive models, and broader environmental factors like background clutter and text orientation to enhance robustness and applicability.
AI Executive Summary
The rapid proliferation of wearable AR devices has opened new frontiers for scene text detection and recognition (STDR), yet real-world environments pose significant challenges. Variations in lighting, distance, and resolution often degrade the performance of traditional models, limiting their practical deployment. This study leverages Meta’s Project Aria smart glasses to systematically investigate these environmental impacts. By collecting a comprehensive dataset across multiple controlled conditions, the researchers evaluate the robustness of state-of-the-art algorithms, particularly EAST combined with CRNN and PyTesseract. Results reveal that low resolution and increased distance notably impair recognition accuracy, with CER rising to 0.65 under challenging conditions. However, applying image upscaling as a preprocessing step reduces CER to 0.48, demonstrating substantial performance gains. The integration of eye-gaze tracking further refines detection by focusing computational resources on user-attention zones, significantly improving efficiency. These innovations mark a step forward in adaptive, user-aware AR systems capable of functioning reliably in complex environments. Despite limitations such as dataset size and latency in gaze data retrieval, the findings provide valuable insights for future development. The work underscores the importance of environmental adaptation in scene text recognition, offering practical pathways toward robust, real-time AR applications in asset management, nutrition analysis, and beyond. Overall, this research bridges the gap between laboratory models and real-world deployment, emphasizing the critical role of multimodal data and preprocessing strategies in overcoming environmental variability.
Deep Analysis
Background
场景文本检测与识别（STDR）作为文档分析的核心任务，经历了从传统图像处理到深度学习模型的快速发展。早期方法如基于边缘检测和模板匹配，受环境干扰较大，鲁棒性不足。近年来，深度学习架构如EAST、CTPN、Transformer等显著提升了检测与识别性能，特别在自然场景中表现优异。然而，这些模型多在理想环境下训练，面对低光、远距离和低分辨率等复杂条件时，性能明显下降。Meta的Project Aria提供了高质量的可穿戴设备数据，为研究真实环境下的文本识别提供了新平台。此前，相关工作多集中在静态图片或视频中的文本检测，缺乏对环境变量影响的系统分析。本研究填补了在Egocentric视觉环境中，光照、距离、分辨率等因素对识别性能影响的空白，推动了个性化、场景感知的AR文本识别技术发展。
Core Problem
现有模型在复杂环境中表现不稳定，尤其在低光、远距离和低分辨率条件下识别率大幅下降。传统方法多依赖静态数据集，难以应对动态、个性化的AR场景。如何在真实、多变的环境中实现鲁棒、实时的文本识别，成为亟待解决的问题。环境因素的复杂交互进一步加剧了模型的适应难度，尤其是在有限数据和计算资源限制下，提升模型的泛化能力和效率成为关键挑战。
Innovation
本研究的创新点包括：1）利用Meta Aria眼镜采集多环境数据，构建真实场景的定制数据集；2）提出结合EAST检测器与启发式字符框合并的多阶段检测方法，有效提升文本区域的完整性；3）引入图像放大预处理策略，缓解低分辨率带来的识别性能下降；4）融合眼动追踪信息，动态调整检测区域，实现用户关注区域的优化。这些创新解决了传统模型在复杂环境下鲁棒性不足的问题，推动了场景文本识别的实用化和智能化。
Methodology
- �� 数据采集：使用Meta Aria智能眼镜在不同光照、距离和分辨率条件下录制场景文本图像。• 数据预处理：对低分辨率图像进行放大，提升细节信息。• 文本检测：采用EAST模型检测文本区域，利用启发式算法合并字符框，形成完整文本块。• 识别模型：分别使用CRNN和PyTesseract进行字符识别，比较性能差异。• 结合眼动追踪：获取用户关注点，缩小检测范围，提升效率。• 评估指标：采用ICDAR 2024标准的CER和IOU指标，分析环境变量对性能的影响。• 改进策略：基于初步分析，调整预处理参数，优化模型鲁棒性。• 实地测试：在实际场景中验证模型效果，结合用户关注区域进行动态检测。
Experiments
实验使用ICDAR 2024 Aria数据集和自建定制数据集，涵盖不同光照、距离和分辨率条件。采用F1、Precision、Recall和CER指标，比较EAST+CRNN与EAST+PyTesseract的性能。通过调节图像放大、亮度增强等预处理技术，观察识别效果变化。设置不同距离（50cm、1m）和分辨率（1048×1408、2880×2880），进行多场景测试。引入眼动追踪数据，评估检测区域优化效果。采用交叉验证确保结果的稳健性，并进行统计分析以确认环境变量的影响程度。
Results
低分辨率和远距离显著降低CRNN的识别性能，CER由0.65提升至0.78，放大后降至0.48，提升约26%。PyTesseract表现更稳定，CER变化较小，但整体准确率低于CRNN。光照变量对识别影响较弱，相关性较低。预处理（放大）有效缓解低质量图像的识别困难，尤其在远距离和低分辨率场景中表现优异。结合眼动追踪实现区域缩小，提升检测效率，验证了多模态融合的潜力。
Applications
本技术可应用于智能AR导航、资产管理、食品标签识别等场景，特别适合在复杂环境中实现高效、鲁棒的文本识别。结合眼动追踪，实现个性化、动态检测，提升用户体验。未来可扩展至自动驾驶、机器人导航等领域，推动场景理解的智能化发展。
Limitations & Outlook
数据集规模有限，难以覆盖所有复杂背景和动态变化；眼动追踪存在延时，限制实时应用；预处理策略对不同环境适应性不足，模型泛化能力有待提升。未来需扩大数据、多模态融合，优化算法效率，增强系统的实用性和鲁棒性。
Plain Language Accessible to non-experts
想象你在一家繁忙的超市里买东西，货架上满是标签和商品。用眼镜看东西时，光线可能明亮，也可能昏暗，商品远近不同，标签的清晰度也会变化。传统的识别方法就像用放大镜看标签，但在光线暗或标签模糊时就会出错。这个研究就像给放大镜加了智能助手，它可以自动放大模糊的标签，还能根据你盯着的区域集中注意力，快速找到你需要的商品。通过让眼镜“知道”你在看哪个标签，它可以更快、更准确地帮你识别信息。这就像你有个聪明的助手，帮你在复杂环境中找到正确的标签，节省时间，避免出错。这个技术未来可以让我们在任何环境下都能轻松识别标签、菜单、路标等信息，无论光线多暗、距离多远，都能帮我们搞定。
ELI14 Explained like you're 14
想象你在超市里买东西，看到很多标签和商品。用普通相机拍照时，有时候光线不好，标签又远又模糊，就很难看清楚。这就像用放大镜看东西，但如果光线太暗或距离太远，放大镜也帮不上忙。这个研究就像给普通放大镜装上了聪明的眼睛，它可以自动放大模糊的标签，还能知道你在看哪个标签，帮你更快找到需要的商品。它还会根据你盯着的地方，集中注意力，省掉看不清的部分，让识别变得更快更准。就像你有个超级助手，帮你在繁忙的超市里轻松找到商品信息，不管光线多暗、距离多远，都能帮你搞定。这项技术未来可以用在很多地方，比如帮盲人看菜单、导航，甚至让机器人更聪明地理解环境。是不是很酷？
Abstract
In an era where wearable technology is reshaping applications, Scene Text Detection and Recognition (STDR) becomes a straightforward choice through the lens of egocentric vision. Leveraging Meta's Project Aria smart glasses, this paper investigates how environmental variables, such as lighting, distance, and resolution, affect the performance of state-of-the-art STDR algorithms in real-world scenarios. We introduce a novel, custom-built dataset captured under controlled conditions and evaluate two OCR pipelines: EAST with CRNN, and EAST with PyTesseract. Our findings reveal that resolution and distance significantly influence recognition accuracy, while lighting plays a less predictable role. Notably, image upscaling emerged as a key pre-processing technique, reducing Character Error Rate (CER) from 0.65 to 0.48. We further demonstrate the potential of integrating eye-gaze tracking to optimise processing efficiency by focusing on user attention zones. This work not only benchmarks STDR performance under realistic conditions but also lays the groundwork for adaptive, user-aware AR systems. Our contributions aim to inspire future research in robust, context-sensitive text recognition for assistive and research-oriented applications, such as asset inspection and nutrition analysis. The code is available at https://github.com/josepDe/Project_Aria_STR.