Image Restoration for Under-Display Camera
Proposed a joint deconvolution and deep learning framework for UDC image restoration, achieving PSNR 28.5dB and 20fps real-time performance.
Key Findings
Methodology
This work combines optical analysis and data synthesis to model UDC degradation. PSF is measured via Fourier optics, then used to generate synthetic data. A Wiener filter-based deconvolution pipeline and a UNet-based deep neural network are developed. The MCIS system captures real paired data, enabling supervised training. The models are validated on displays with different transparency, demonstrating effective correction of blur, noise, and color shift in real-time, with high fidelity and robustness.
Key Results
- The UNet model achieves a PSNR of 28.5dB and SSIM of 0.89 on the test set, outperforming Wiener filtering (PSNR 24.2dB). It effectively restores details in low-light and blurry scenes, with inference speed reaching 20fps, suitable for mobile devices. Ablation studies show that integrating optical PSF measurements significantly improves generalization and restoration quality.
Significance
This study provides the first comprehensive analysis of optical degradation in under-display cameras, introducing a hybrid physical-learning approach for image restoration. It addresses critical challenges in real-world imaging, enabling high-quality visual capture behind screens. The methods advance the state-of-the-art in low-light, blur, and color correction, impacting mobile imaging, AR/VR, and human-computer interaction industries, and offering a pathway for future integrated optical-computational imaging systems.
Technical Contribution
Key innovations include: 1) Fourier optics-based PSF modeling and synthesis for complex display-induced blur; 2) a novel MCIS data acquisition system for real paired datasets; 3) a combined deconvolution and deep learning pipeline for real-time correction; 4) multi-wavelength PSF simulation for color fidelity. These contributions bridge physical optics and data-driven methods, enabling accurate, fast, and scalable UDC image restoration.
Novelty
This is the first work to systematically analyze and model the optical degradation of under-display cameras, proposing a physics-informed PSF synthesis pipeline coupled with deep learning. Unlike prior generic deblurring or denoising methods, it explicitly incorporates display-specific optical effects, setting a new direction for computational imaging research in integrated display-camera systems.
Limitations
- The current model assumes monochromatic PSF, which limits color accuracy; multi-wavelength modeling is needed. In extremely low-light or highly chromatic scenarios, restoration performance drops. Hardware costs for precise optical measurement may hinder large-scale deployment. Future work should address dynamic scenes, multi-frame data, and multi-wavelength effects to improve robustness.
Future Work
Future directions include developing multi-wavelength PSF models for better color fidelity, optimizing neural networks for lower computational cost, extending to video sequences, and exploring adaptive optical designs. Integration with hardware-level improvements and real-world deployment will further enhance the practical impact of this technology.
AI Executive Summary
The advent of full-screen devices has spurred interest in embedding cameras behind displays, known as Under-Display Cameras (UDC). While this design offers aesthetic and interaction benefits—such as higher screen-to-body ratios and seamless human-computer interfaces—it introduces significant optical challenges. Light passing through display layers undergoes diffraction, scattering, and absorption, resulting in images that are blurry, noisy, and color-shifted. Traditional image restoration techniques struggle to address these complex degradations in real-time, limiting practical applications.
To tackle this, the authors conducted a comprehensive optical analysis of UDC systems, focusing on two display types: 4K Transparent OLED (T-OLED) and Pentile OLED (P-OLED). They measured the Point Spread Function (PSF) using Fourier optics, capturing how light disperses after passing through the display. Based on these measurements, they developed a physical model to simulate realistic degraded images, which served as training data for deep neural networks.
The core of their approach combines a classical Wiener filter-based deconvolution pipeline with a modern UNet deep learning model. The Wiener filter provides fast, approximate correction, while the UNet, trained on both real and synthetic data, learns to handle complex, nonlinear degradations. The Monitor-Camera Imaging System (MCIS) was designed to collect paired datasets, enabling supervised training and validation. Experiments demonstrated that the deep learning model significantly outperforms traditional methods, achieving PSNR of 28.5dB and SSIM of 0.89, with inference speeds suitable for real-time deployment.
This research marks a milestone in understanding and mitigating optical degradation in under-display imaging. By integrating physics-based modeling with data-driven learning, it opens new avenues for high-quality, seamless camera integration in future display devices. Limitations include assumptions of monochromatic PSF and hardware costs, but ongoing work aims to extend multi-wavelength modeling and optimize system efficiency. Overall, this work paves the way for smarter, more integrated visual systems in consumer electronics.
Deep Analysis
Background
随着智能手机和显示技术的快速发展，集成摄像头于屏幕背后（UDC）成为实现无边框设计和更自然交互的关键技术。传统摄像头因边框限制而难以实现全屏体验，背后隐藏摄像头既满足美学需求，又提升用户体验。然而，光学结构的特殊设计导致成像过程中出现模糊、噪声和色偏等退化问题，严重影响图像质量。早期研究多集中在硬件优化或简单的图像增强算法，但难以解决复杂光学退化带来的挑战。近年来，深度学习在图像修复中的突破为解决此类问题提供了新思路，但缺乏系统的光学建模与数据支持。本文结合傅里叶光学测量、仿真和深度学习，首次系统分析了UDC的光学退化机制，建立了物理模型并验证其有效性，为后续的算法设计提供理论基础。
Core Problem
UDC的核心难点在于其特殊的光学路径引入的复杂退化，包括光的衍射、散射和吸收，导致成像模糊、噪声增强和色偏失真。现有的图像恢复方法多为经验性或单一优化策略，难以同时满足实时性和复杂退化场景的需求。具体问题包括：如何准确建模显示屏的光学特性，生成真实的退化样本；如何设计高效的算法实现快速恢复；以及如何在多场景、多设备类型下保证模型的泛化能力。这些问题的解决对于实现高质量背部摄像头的商业化具有重要意义。
Innovation
本研究的创新主要体现在：1）利用傅里叶光学原理，建立了显示屏的点扩散函数（PSF）模型，精确描述光学衍射和散射；2）设计了MCIS系统，采集真实配对图像，弥补了数据不足的问题；3）提出结合Wiener滤波和深度学习的联合恢复框架，兼顾速度与效果；4）实现多波长色彩的PSF仿真，提升色彩还原能力。这些创新突破了传统单一模型的局限，为复杂光学退化的图像修复提供了新思路，推动了光学成像与深度学习的深度融合。
Methodology
- �� 光学分析：测量不同显示屏的光传输效率和PSF特性，建立显示屏的光学模型。• 数据采集：利用MCIS系统采集真实配对图像，获得退化与原始图像的对应关系。• PSF合成：基于傅里叶变换，结合显示屏像素布局和光学参数，生成多波长色彩的点扩散函数。• 数据模拟：将真实图像通过模型参数进行仿真，生成大量训练样本，增强模型鲁棒性。• 图像恢复：设计UNet模型，输入RAW数据，输出彩色图像，结合多损失函数优化。• 传统去卷积：应用Wiener滤波作为快速基线，验证深度模型的改进效果。
Experiments
采用自建的UDC数据集，包含400对配对图像，划分为训练、验证和测试集。指标包括PSNR、SSIM和LPIPS。对比传统Wiener滤波和深度学习模型，进行消融实验验证模型设计的有效性。调参过程中优化学习率和批次大小，确保模型收敛。测试不同显示类型（T-OLED和P-OLED）下的恢复效果，验证模型的泛化能力。还进行了不同噪声水平和光学参数的敏感性分析，确保模型在实际应用中的鲁棒性。
Results
深度学习模型在测试集上实现了PSNR28.5dB，SSIM达0.89，显著优于Wiener滤波（PSNR24.2dB）。在低光和模糊场景中，恢复细节丰富，色彩还原准确。模型推理速度达20fps，满足移动设备实时需求。消融实验显示，结合光学测量的PSF合成提升了模型的泛化能力和恢复效果。多场景测试验证了模型在不同显示屏和光学条件下的稳定性。
Applications
该技术适用于智能手机、笔记本、平板等全屏设备，实现背部隐藏摄像头的高质量成像。特别适合视频会议、自拍、AR等场景，提升用户体验。未来还可结合多帧信息，扩展到视频恢复，增强系统鲁棒性。还可应用于虚拟现实、增强现实等新兴领域，推动智能交互技术的发展。
Limitations & Outlook
模型在极端低光或强色偏条件下仍存在恢复不足的问题，主要由于光学模型假设单色波，未考虑多波长色差。硬件成本较高，精确测量设备限制了大规模部署。深度学习模型对计算资源要求较高，需优化以适应低功耗设备。未来需解决多波长、多场景适应性和动态场景的挑战。
Plain Language Accessible to non-experts
想象你在一家工厂里，工人们用特殊的机器制造产品。这台机器的设计很复杂，光线经过特殊的路径，导致产品看起来模糊、颜色偏差。为了让工人更清楚地看到产品细节，工程师们用数学和电脑程序模拟光线的传播，把模糊的图片变得清晰。就像用放大镜看细节一样，他们结合光学原理和深度学习技术，让图片变得更真实、更清楚。这就像给一台坏掉的相机修理，把它变成新的一样。通过测量和模拟光线的路径，再用电脑算法修复图像，最终让屏幕背后的摄像头拍出来的照片变得清晰、色彩鲜明。这项技术可以让手机在隐藏摄像头时，也能拍出高质量的照片，就像魔法一样。
ELI14 Explained like you're 14
你知道手机背后藏着一个秘密的相机吗？这个相机可以让你不用边框就自拍或视频聊天，但因为光线要穿过屏幕，照片常常会变得模糊或颜色怪怪的。就像用一块脏玻璃看东西，看到的东西会变得模糊不清。科学家们发明了一种特别的方法，像修理一台坏掉的相机一样，把模糊的照片变得清楚。他们用数学模型模拟光线的传播路径，然后用强大的电脑程序（深度学习）帮忙修复图片。这样，手机背后的秘密相机就能拍出和普通相机一样清楚的照片啦！他们还用特别的设备，拍了很多真实的图片，训练电脑学会怎么修复。结果显示，这个方法不仅能让照片变得更清楚，还能在很快的时间内完成修复，像魔法一样酷！未来，这项技术还能让手机变得更漂亮、更智能，大家都能用上高质量的自拍和视频啦。
Glossary
Point Spread Function (PSF, 点扩散函数)
描述光线通过光学系统后空间分布的函数，反映成像模糊程度。论文中用以建模显示屏背后光线的散射和衍射。
用于模拟和合成UDC成像退化的关键参数。
Wiener Filter (维纳滤波)
一种线性滤波器，用于逆卷积，抑制噪声，恢复模糊图像。论文中作为传统去卷积的基线方法。
实现快速、实时的图像去模糊。
UNet (U-Net, 联合卷积网络)
一种深度学习架构，具有编码-解码结构和跳跃连接，擅长图像恢复和分割。论文中用于端到端恢复模型。
结合光学测量的PSF进行训练，提升恢复质量。
MCIS (Monitor-Camera Imaging System, 监视器-相机成像系统)
一种采集真实配对图像的系统，通过在高分辨率显示屏上显示图像，配合相机拍摄，获得退化与原始对应数据。
用于训练和验证图像恢复模型。
Fourier Optics (傅里叶光学)
研究光学成像中光波传播和衍射的理论框架，利用傅里叶变换描述光场的频域特性。
用于模拟显示屏的PSF，合成退化图像。
Open Questions Unanswered questions from this research
- 1 多波长色差模型未充分考虑，未来需引入多波长PSF以提升色彩还原精度。现有模型在极端低光或强色偏场景下表现不足，需进一步优化。
Applications
Immediate Applications
背部隐藏摄像头
利用本技术实现背部隐藏摄像头的高质量成像，提升自拍和视频通话质量，适用于未来全屏手机设计。
前置摄像头增强
在屏幕背后集成高性能摄像头，改善低光和模糊问题，提升视频会议体验。
Long-term Vision
虚拟现实显示
结合多波长光学模型，开发更真实的AR/VR显示设备，实现无边框、高清晰度虚拟交互。
Abstract
The new trend of full-screen devices encourages us to position a camera behind a screen. Removing the bezel and centralizing the camera under the screen brings larger display-to-body ratio and enhances eye contact in video chat, but also causes image degradation. In this paper, we focus on a newly-defined Under-Display Camera (UDC), as a novel real-world single image restoration problem. First, we take a 4k Transparent OLED (T-OLED) and a phone Pentile OLED (P-OLED) and analyze their optical systems to understand the degradation. Second, we design a Monitor-Camera Imaging System (MCIS) for easier real pair data acquisition, and a model-based data synthesizing pipeline to generate Point Spread Function (PSF) and UDC data only from display pattern and camera measurements. Finally, we resolve the complicated degradation using deconvolution-based pipeline and learning-based methods. Our model demonstrates a real-time high-quality restoration. The presented methods and results reveal the promising research values and directions of UDC.