PACE: Perceived-Latency-Aware Cascading Service Routing and Filler Control for QoE-Efficient Retrieval-Augmented Dialogue Serving
PACE reduces P95 PTFR from 0.53s to 0.29s via perceived-latency-aware dialogue routing.
Key Findings
Methodology
PACE integrates a load-adaptive cascading router, joint path-filler controller, and volatility-aware cache admission. It minimizes perceived time-to-first-response (PTFR) under quality and cost constraints.
Key Results
- Result 1: On 75,000 CarQA requests, PACE reduced pure-LLM P95 PTFR from 0.53s to 0.29s.
- Result 2: Adaptive controller achieved 0.41s P95, outperforming RAG by 2.4x under high load with comparable quality.
- Result 3: Filler controller cut small-model calls by 94% with a 0% filler-answer conflict rate.
Significance
This study formalizes perceived time-to-first-response (PTFR) as a QoE objective for dialogue services. It introduces a novel method to jointly optimize answer source selection and waiting window fillers, significantly enhancing user experience in real-world deployments like automotive retail.
Technical Contribution
PACE introduces load-adaptive cascading routing and joint path-filler control, uniquely optimizing PTFR. It also incorporates volatility-aware cache admission to reduce stale answers, advancing both latency and quality in dialogue services.
Novelty
PACE is the first to quantify filler-answer conflict risk and jointly optimize routing and fillers for perceived latency minimization, offering significant innovation over RAG and semantic caching methods.
Limitations
- Limitation 1: PACE relies on specific load signals like TTFT and arrival rate, which may be unstable under extreme load fluctuations.
- Limitation 2: Filler model selection and training introduce additional computational overhead.
- Limitation 3: Volatility-aware caching may add overhead for non-volatile queries.
Future Work
Future work could explore more sophisticated load prediction models and extend filler mechanisms to multimodal dialogue systems.
AI Executive Summary
In dialogue services, perceived time-to-first-response (PTFR) is critical for user experience. Traditional optimizations focus on speed or cost but often neglect perceived latency. PACE addresses this gap by introducing a novel framework that minimizes PTFR under quality and cost constraints.
PACE integrates three key mechanisms: a load-adaptive cascading router, a joint path-filler controller, and a volatility-aware cache admission system. Deployed in an automotive retail setting, PACE significantly reduces perceived latency while maintaining high-quality responses and low computational costs.
Experiments on 75,000 CarQA requests show that PACE reduces pure-LLM P95 PTFR from 0.53s to 0.29s. The adaptive controller outperforms RAG by 2.4x under high load, while the filler controller cuts small-model calls by 94% with zero conflict. Despite limitations like reliance on load signals, PACE offers a robust foundation for future dialogue service optimizations.
Deep Analysis
Background
Large language models (LLMs) are widely used in dialogue services like chatbots and virtual assistants. However, existing methods often optimize for speed, throughput, or cost while neglecting perceived latency, a critical factor in user experience, especially in human-robot interactions.
Core Problem
Current methods like cascading routing, semantic caching, and adaptive retrieval optimize cost, hit rate, or quality individually but fail to jointly control answer source selection and waiting window fillers. Minimizing perceived latency under quality and cost constraints remains a key challenge.
Innovation
PACE introduces several innovations:
- �� Formalizes perceived time-to-first-response (PTFR) as a QoE optimization objective.
- �� Implements a load-adaptive cascading router to dynamically adjust semantic cache and retrieval thresholds.
- �� Designs a joint path-filler controller to decide filler usage dynamically.
- �� Introduces volatility-aware cache admission to reduce stale answers.
Methodology
PACE's implementation involves:
- �� Dynamically adjusting semantic cache and retrieval thresholds using load signals (TTFT and arrival rate).
- �� Running a filler model in parallel with LLM generation to occupy the waiting window.
- �� Using a lightweight lexical classifier to identify volatile queries and applying short time-to-live or denial policies.
- �� Logging all request paths, latencies, and decisions as structured data for analysis.
Experiments
Experiments were conducted on 75,000 CarQA requests, comparing PACE against traditional RAG, semantic caching, and static threshold methods. Metrics included P95 PTFR, answer quality, and filler-answer conflict rate.
Results
Key findings include:
- �� PACE reduced pure-LLM P95 PTFR from 0.53s to 0.29s.
- �� Adaptive controller achieved 0.41s P95, outperforming RAG by 2.4x under high load.
- �� Filler controller cut small-model calls by 94% with a 0% conflict rate.
Applications
PACE is ideal for low-latency, high-quality dialogue services in retail robots, customer support, and online education. Its mechanisms can also extend to multimodal dialogue systems.
Limitations & Outlook
PACE's limitations include reliance on load signals, additional computational costs for filler models, and potential overhead from volatility-aware caching. Future work could address these challenges.
Plain Language Accessible to non-experts
Imagine you're at a restaurant. The chef needs time to prepare your main dish, but the waiter brings you bread and water to keep you occupied while you wait. In the same way, PACE uses a small filler model to provide a quick response while the main answer is being prepared in the background. It also adjusts its strategy based on how busy the restaurant (system) is, prioritizing faster responses during busy times.
ELI14 Explained like you're 14
Imagine you're playing a video game, and it suddenly freezes. Annoying, right? Now imagine the game shows you a fun mini-game while it loads. That's what PACE does! It gives you a quick response (like a mini-game) while preparing the main answer in the background. Plus, it knows when to give you the quick response and when to just give you the answer directly. Cool, huh?
Glossary
PTFR (Perceived Time-to-First-Response)
The time from a user's query to the first meaningful response, a key QoE metric.
Used to evaluate user experience in dialogue services.
RAG (Retrieval-Augmented Generation)
A technique combining knowledge retrieval with LLM generation for high-quality answers.
Used in PACE's L1 and L2 paths.
Semantic Cache
A caching mechanism based on query semantic similarity for fast answer retrieval.
Used in PACE's L0 path for near-zero latency.
Filler Model
A small model that provides short social acknowledgments while the main answer is being generated.
PACE uses a filler model to occupy the waiting window.
Volatility-Aware Cache
A mechanism that dynamically adjusts caching policies based on query volatility.
Used to reduce stale answers in PACE.
Open Questions Unanswered questions from this research
- 1 How can PACE maintain stability under extreme load fluctuations?
- 2 Can more efficient filler models be designed to reduce computational overhead?
Applications
Immediate Applications
Retail Robots
Provide low-latency, high-quality customer dialogue services in automotive showrooms and similar settings.
Online Education
Quickly respond to student queries on educational platforms, reducing perceived wait times.
Long-term Vision
Multimodal Dialogue Systems
Extend PACE to voice and visual dialogue scenarios for more natural human-computer interactions.
Abstract
We present the PACE, a framework for retrieval-augmented dialogue serving that formalizes Perceived Time-to-First-Response (PTFR) as a QoE objective and minimizes it under quality/cost constraints. Unlike prior work on cascaded routing, semantic caching, or adaptive retrieval, PACE jointly controls which answer source composes the response and what fills the waiting window. Deployed on a humanoid-robot sales service, it combines three mechanisms: a load-adaptive cascading router, a joint path-filler controller, and volatility-aware cache admission. On 75k CarQA requests, the cascade halves pure-LLM PTFR at P95 (0.29 vs 0.53s at c16). The adaptive controller reaches 0.41s P95, outperforming RAG by 2.4 times at high load with equal quality. The filler controller cuts calls by 94% with zero conflict. Volatility-aware admission reduces stale answers from 86% to 0%. A gating rule ensures the controller never worse than the baseline, with exposure bounded by one hold period. This is the first quantification of filler-answer conflict risk in deployed services.