Efficient Test-Time Finetuning of LLMs via Convex Reconstruction and Gradient Caching

The evolution of large-scale language models (LLMs) such as GPT-3, BERT, and T5 has transformed NLP by enabling models with billions of parameters to perform a wide range of tasks with minimal task-specific tuning. These models are trained on web-scale corpora, capturing broad linguistic and factual knowledge. However, their deployment often faces challenges related to adaptability and efficiency. Fine-tuning on task-specific data improves performance but is computationally expensive and slow, especially for large models. Test-time fine-tuning (TTFT) emerged as a promising paradigm, allowing models to adapt dynamically to each input by retrieving relevant data, updating parameters, and evaluating. Early approaches like gradient-based meta-learning (e.g., MAML) provided theoretical foundations but were impractical at scale. Recent methods such as kNN retrieval and SIFT focus on selecting relevant samples for quick adaptation, yet they suffer from redundancy, high computational cost, and limited diversity. These issues hinder real-time deployment, especially in latency-sensitive applications like chatbots, real-time translation, and content moderation. The challenge remains to develop methods that can efficiently select diverse, relevant data and perform rapid updates without sacrificing accuracy.

The core problem addressed in this work is how to efficiently perform test-time fine-tuning on large language models under strict latency constraints. Existing methods either rely on simple nearest-neighbor retrieval, which often results in redundant samples and limited diversity, or on complex selection heuristics that are computationally expensive. Additionally, frequent model updates via gradient descent incur high computational costs, making real-time adaptation infeasible. The key bottlenecks are: 1) selecting a relevant and diverse support set quickly; 2) performing fast fine-tuning with minimal resource consumption; 3) avoiding redundant computations during repeated samples. Overcoming these bottlenecks is critical for deploying adaptive models in real-world, latency-sensitive environments such as conversational agents, where response time directly impacts user experience. The challenge is to balance relevance, diversity, and computational efficiency simultaneously, requiring novel algorithmic solutions that leverage geometric insights and optimization techniques.

The main innovations introduced in this paper are threefold: 1) convex geometric support set selection—by formulating the data selection as a sparse convex approximation problem solved via the Frank–Wolfe algorithm, which naturally balances relevance and diversity without explicit penalties; 2) geometric integerization—converting fractional convex weights into an integer multiset that preserves approximation quality and induces natural sample repetition, facilitating efficient gradient caching; 3) gradient reuse—exploiting repeated samples in the multiset to cache and reuse gradients across multiple optimizer steps, dramatically reducing the number of forward-backward passes. These innovations collectively enable a highly efficient TTFT pipeline that maintains high adaptation quality while significantly reducing computational costs. The approach leverages the geometry of convex hulls, avoiding heuristic or greedy sample selection, and introduces a principled way to handle sample diversity and redundancy simultaneously.

• �� Define the problem as finding a sparse convex combination of candidate samples that approximates the query embedding, minimizing the squared L2 distance.
• �� Use Frank–Wolfe algorithm: starting from a single vertex, iteratively add the candidate that maximally reduces residual error, forming a sparse convex combination with at most d+1 support points.
• �� Solve the convex approximation efficiently via inner product evaluations, avoiding projections, and stopping when the residual error falls below a threshold or support size cap is reached.
• �� Geometric integerization: convert fractional weights into integer counts through three steps—floor allocation, greedy fill, and local swap refinement—ensuring the total sample size equals N.
• �� Construct the support multiset from the integerized weights, where each sample appears multiple times according to its count.
• �� During fine-tuning, process the multiset in blocks, caching gradients for repeated samples, and refresh the cache periodically to balance accuracy and speed.
• �� Conduct experiments on datasets like The Pile, comparing BPB metrics, total runtime, and support set sizes, validating the speedup and quality improvements over baselines.

• �� Datasets: 12 subsets from The Pile, including domains like arXiv, DM Math, Enron, GitHub, and others, covering academic, legal, and technical texts.
• �� Baselines: compare against kNN retrieval, SIFT, and existing TTFT methods.
• �� Metrics: primarily bits-per-byte (BPB) to measure compression efficiency, along with total runtime, selection speed, and fine-tuning effectiveness.
• �� Hyperparameters: candidate pool size, support set cap, Frank–Wolfe tolerance, gradient cache refresh interval.
• �� Protocol: evaluate at multiple time budgets (e.g., T.75s, T.0s), analyze the impact of support set size, integerization, and gradient reuse through ablation studies. Measure the speedup factors and BPB improvements, and visualize the diversity of selected support sets using t-SNE plots.

• �� Across all datasets, HullFT consistently outperforms kNN and SIFT, with an average BPB reduction of 3.83% at T.75s and 3.44% at T.0s, with maximum improvements exceeding 6%.
• �� The convex hull-based support set selection is 25.8× faster than SIFT, and gradient caching accelerates fine-tuning by 1.48×, saving nearly 90 seconds in total.
• �� The support set's geometric diversity naturally emerges, avoiding explicit diversity penalties, and the repeated samples enable effective gradient reuse, leading to substantial speedups without compromising quality.

• �� Real-time adaptive NLP systems: enabling chatbots, virtual assistants, and content moderation tools to quickly adapt to new inputs with minimal latency.
• �� Edge computing: deploying models on resource-constrained devices, where fast support set selection and gradient caching reduce computational load.
• �� Personalized AI: dynamically customizing models based on user interactions, improving relevance and user experience in applications like recommendation systems.
• �� Cross-modal adaptation: extending the geometric approach to multi-modal models, facilitating rapid adaptation across different data types.

• �� Dependence on the quality of the candidate retrieval pool; limited recall or diversity in the pool constrains the support set quality.
• �� The geometric diversity achieved may not fully capture complex semantic nuances in highly intricate tasks.
• �� Gradient caching assumes high sample repetition; in highly diverse or novel scenarios, the benefits diminish, potentially leading to bias.
• �� The approach's scalability to extremely large models or multi-task settings remains to be validated, and high-dimensional convex optimization may incur additional computational costs.