Data Overfitting for On-Device Super-Resolution with Dynamic Algorithm and Compiler Co-Design
Proposes Dy-DCA, a content-aware single-model super-resolution framework with dynamic routing and compiler optimization, achieving 33FPS on mobile with 1.7× speedup.
Key Findings
Methodology
This work introduces a content-aware data preprocessing pipeline combined with a dynamic neural network architecture. Video frames are split into uneven patches based on PSNR-guided complexity, reducing the number of models to one. A routing node directs patches to different paths, enabling the network to handle diverse textures efficiently. The approach leverages operator classification (e.g., shape, convolution) for shape inference and data-flow analysis, facilitating compiler-level optimizations like fused code generation and static scheduling. These techniques collectively enable real-time inference on resource-constrained devices, significantly reducing overheads.
Key Results
- On UVG and VSD4K datasets, the proposed method outperforms multi-model approaches by at least 0.5dB PSNR at 2×, 3×, 4× scaling, while achieving 33FPS on mobile phones. Overall, the system accelerates inference by 1.7× and reduces memory usage by 1.61×, demonstrating practical deployment feasibility.
- Deployments on devices like OnePlus 11 show inference latency around 30ms, with memory consumption dropping below 60MB, validating efficiency gains. The content-aware patching reduces model switching by 4× and I/O overhead by 7×, leading to lower energy consumption and improved real-time performance.
- Compiler optimizations based on shape inference enable fused kernels and static scheduling, further boosting speed and reducing system overhead, making the approach suitable for edge applications requiring high throughput and low latency.
Significance
This research addresses the critical bottleneck of multiple model switching in edge-based super-resolution, proposing a unified, content-adaptive solution. It bridges the gap between high-quality super-resolution and real-time deployment on mobile devices, facilitating applications like video streaming, conferencing, and AR/VR. The integration of algorithmic and compiler-level optimizations paves the way for scalable, efficient video enhancement systems that can operate under strict resource constraints, advancing both academic research and industry deployment.
Technical Contribution
The core innovation lies in combining content-aware patch segmentation with a dynamic routing neural network, supported by a shape inference framework based on operator classification. This allows the system to adapt dynamically to content complexity, reducing model switching and I/O. Additionally, the compiler-level shape analysis and fusion techniques enable static scheduling and memory optimization, overcoming challenges posed by dynamic input shapes. These contributions collectively enable a highly efficient, scalable super-resolution pipeline suitable for edge devices.
Novelty
This is the first work to integrate content-aware patching with a single dynamic neural network guided by shape inference and compiler optimizations for super-resolution. Unlike prior multi-model or static approaches, it adapts to content complexity in real-time, significantly reducing overhead while maintaining high PSNR. The combination of operator classification-based shape inference and compiler-aware optimization represents a novel, holistic approach to edge super-resolution.
Limitations
- The reliance on PSNR for patch complexity assessment may not fully capture perceptual quality, potentially affecting subjective results.
- Dynamic routing and shape inference introduce some computational overhead, which might impact performance under highly dynamic or complex control flows.
- The current design is tailored to specific backbone architectures like EDSR and WDSR; generalization to other models requires further validation.
Future Work
Future directions include integrating semantic content understanding for more accurate patch segmentation, extending shape inference to more complex operators, and exploring hardware-aware optimization for specialized accelerators. Additionally, improving robustness against content variability and reducing inference latency further will be key to broader adoption.
AI Executive Summary
As high-resolution video content becomes ubiquitous, delivering high-quality, real-time super-resolution on mobile and edge devices remains a significant challenge. Existing multi-model approaches, though effective in improving PSNR, suffer from frequent model switching, high memory footprint, and energy consumption, severely limiting their practicality. This paper introduces Dy-DCA, a novel framework that leverages content-aware patch segmentation and a dynamic routing neural network to unify super-resolution into a single adaptable model.
The core idea is to split video frames into patches of varying sizes based on content complexity, guided by PSNR metrics. Simultaneously, a dynamic neural network with a routing node directs patches to different paths optimized for their texture detail. This approach minimizes the total number of models needed, reducing switching overhead and I/O bottlenecks. To support this dynamic input, the authors develop a shape inference framework based on operator classification, enabling static analysis of intermediate tensor shapes. This facilitates compiler-level optimizations such as fused code generation and static scheduling, which further accelerate inference.
Experimental results demonstrate that on standard datasets UVG and VSD4K, the proposed method surpasses multi-model baselines by at least 0.5dB PSNR at 2×, 3×, and 4× scales. On a commercial mobile device (OnePlus 11), the system achieves 33FPS, with inference latency around 30ms and memory usage below 60MB. The overall speedup reaches 1.7×, and memory consumption drops 1.61×, validating the approach’s efficiency and practicality. The content-aware patching strategy reduces model switching by 4× and I/O overhead by 7×, significantly lowering energy consumption.
This work marks a substantial step toward scalable, real-time super-resolution on resource-constrained devices, bridging the gap between high-quality video enhancement and practical deployment. Future work aims to incorporate semantic content understanding, extend shape inference to more operators, and optimize for specialized hardware, broadening the scope and robustness of edge super-resolution solutions.
Deep Analysis
Background
Video super-resolution技术经历了从传统插值到深度学习的快速发展。早期方法如SRCNN、VDSR追求高性能，但模型庞大难以部署。后续如EDSR、WDSR引入模块化设计，提升效率。近年来，利用模型过拟合实现个性化超分成为趋势，但多模型切换带来巨大开销，限制实际应用。边缘设备对模型大小和实时性要求更高，促使研究转向单模型、多内容适应方案，结合内容感知与编译优化成为新方向。
Core Problem
多模型超分虽然效果优异，但模型切换频繁导致延迟和能耗剧增，难以满足移动端实时需求。现有方法多依赖静态模型或简单划分，未能充分利用内容信息优化Patch划分和模型调度，存在I/O瓶颈和系统复杂性。如何在保证高质量的同时，减少模型数量和系统开销，成为亟待解决的难题。
Innovation
提出内容感知Patch划分策略，根据PSNR动态调整Patch大小，减少冗余信息。引入动态路由网络，单模型覆盖多内容场景，避免频繁切换。结合Operator分类的形状推断，支持动态特征的编译优化，包括融合代码和静态调度。实现算法、软件与硬件的协同优化，显著提升边缘设备推理速度和能效，突破动态特征在实际部署中的瓶颈。
Methodology
- �� 视频帧根据内容复杂度进行不均匀划分，利用PSNR指标引导Patch大小调整。
- �� 使用动态路由网络，将不同Patch分配到不同路径，避免模型切换。
- �� 利用Operator分类（如Shape、Conv、Add）进行形状推断，支持动态特征的静态分析。
- �� 通过数据流分析实现融合代码、调度优化，减少运行时开销。
- �� 编译器结合形状推断结果，进行静态调度和内存优化，确保实时推理。
- �� 在边缘设备部署，验证速度、能耗和内存节省效果。
Experiments
采用UVG和VSD4K数据集，比较多模型方案与单模型内容感知方案的PSNR、速度和内存。设置不同放大倍数（X2、X3、X4），调节PSNR阈值（40、30）控制Patch划分。测试在手机端的推理速度和能耗，验证模型切换和I/O开销的降低效果。进行消融实验验证编译优化的贡献。
Results
在多场景、多倍数下，单模型方案PSNR超越多模型方案0.5dB，手机端实现33FPS实时推理。整体速度提升1.7倍，内存节省1.61倍。Patch划分策略减少模型切换次数4倍，I/O开销降低7倍。编译优化使推理延迟降至30ms，能耗降低50%以上，验证方案的实用性和高效性。
Applications
适用于移动视频通信、云端边缘协作、智能终端等场景，满足低延迟高质量视频传输需求。内容感知Patch划分和动态调度可适应不同内容复杂度，提升用户体验。未来结合多模态信息，支持更复杂场景。
Limitations & Outlook
依赖PSNR指标，可能对某些内容变化不敏感。动态路由在极端场景下鲁棒性不足。模型结构偏向特定架构，泛化能力待验证。未来需增强内容感知的准确性和模型鲁棒性。
Plain Language Accessible to non-experts
想象你在厨房做饭，面对不同的食材，有的需要切得细一些，有的可以大块放。为了节省时间和材料，你会根据食材的不同，采用不同的切割方式。这里，视频就像食材，Patch是切好的块，内容越丰富的部分需要更细的处理。传统方法像用很多不同的厨具，每次切换都要换工具，既麻烦又慢。本文提出一种聪明的办法，只用一种厨具，根据食材的内容自动调整切割大小，既快又省材料。这样，厨师（算法）可以更快地完成菜肴（视频超分），而厨房（设备）也不会堆满工具（模型），效率大大提高。整个过程就像厨房里的智能助手，知道什么时候用大刀，什么时候用细刀，帮你节省时间和精力，做出更好吃的菜。
Abstract
Deep neural networks (DNNs) are frequently employed in a variety of computer vision applications. Nowadays, an emerging trend in the current video distribution system is to take advantage of DNN's overfitting properties to perform video resolution upscaling. By splitting videos into chunks and applying a super-resolution (SR) model to overfit each chunk, this scheme of SR models plus video chunks is able to replace traditional video transmission to enhance video quality and transmission efficiency. However, many models and chunks are needed to guarantee high performance, which leads to tremendous overhead on model switching and memory footprints at the user end. To resolve such problems, we propose a Dynamic Deep neural network assisted by a Content-Aware data processing pipeline to reduce the model number down to one (Dy-DCA), which helps promote performance while conserving computational resources. Additionally, to achieve real acceleration on the user end, we designed a framework that optimizes dynamic features (e.g., dynamic shapes, sizes, and control flow) in Dy-DCA to enable a series of compilation optimizations, including fused code generation, static execution planning, etc. By employing such techniques, our method achieves better PSNR and real-time performance (33 FPS) on an off-the-shelf mobile phone. Meanwhile, assisted by our compilation optimization, we achieve a 1.7$\times$ speedup while saving up to 1.61$\times$ memory consumption. Code available in https://github.com/coulsonlee/Dy-DCA-ECCV2024.