DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation

Recent progress in large language models (LLMs) such as GPT-4, LLaMA, and PaLM has revolutionized natural language understanding and generation. However, these models often struggle with factual accuracy and knowledge coverage. Retrieval-augmented generation (RAG) systems address this by integrating external knowledge sources, enabling models to access up-to-date information. Early approaches like BM25 and dense retrieval methods (e.g., Contriever) improved retrieval quality, but static ranking strategies limited adaptability. Recent works incorporated reinforcement learning to refine retrieval, yet most relied on relevance scores without considering generation feedback. This gap motivated research into adaptive, feedback-driven reranking mechanisms that can dynamically optimize retrieval content based on downstream response quality.

Despite advances, existing RAG systems face challenges in balancing the number of retrieved documents. Fixed thresholds often lead to missing critical info or introducing noise, reducing answer accuracy and interpretability. Moreover, static relevance scoring fails to adapt to query complexity, especially in knowledge-intensive tasks requiring nuanced retrieval. The core problem is designing a mechanism that can dynamically adjust both the order and quantity of documents in response to query characteristics, leveraging feedback from the generation process itself. Achieving this requires integrating reinforcement learning with effective reward signals that reflect the true quality of generated responses, which remains an open challenge.

DynamicRAG's key innovations include: 1) modeling the reranker as an RL agent that learns to adaptively select and order documents based on generation feedback; 2) employing a multi-dimensional reward function combining exact match, semantic similarity, fluency, length penalty, and LLM-based scoring; 3) integrating behavior cloning with exploration strategies to stabilize training; 4) utilizing a DPO-based reward optimization framework for robust policy updates. These innovations enable the system to self-tune retrieval strategies in real-time, significantly improving answer accuracy and relevance compared to static approaches. The method also reduces reliance on large annotated datasets, demonstrating high efficiency.

• �� Retrieve top-N documents using a fixed retriever like Contriever.
• �� Encode query and documents with transformer-based models (e.g., LLaMA).
• �� Initialize the reranker as a policy network trained via behavior cloning from expert trajectories.
• �� During exploration, sample multiple document orderings and quantities, generate responses, and evaluate quality.
• �� Compute multi-dimensional rewards (EM, SS, TF, LP, LLM-Eval) for each trajectory.
• �� Use DPO to compare trajectories and update the policy to favor higher-reward actions.
• �� Fine-tune the generator to leverage the dynamically selected documents, closing the feedback loop.
• �� Repeat the process iteratively, balancing exploration and exploitation for optimal performance.

The evaluation involved seven datasets covering question answering, multi-hop reasoning, long-form generation, and fact verification. Baselines included static relevance models, RL-based methods, and recent SOTA like RankRAG. Metrics such as EM, R@K, and recall were used. Models were trained on approximately 150k instruction-output pairs, with hyperparameters tuned for stability. Ablation studies tested the impact of RL, dynamic adjustment, and shared parameters. Results showed consistent improvements across datasets, with EM scores exceeding 48% on NQ and R@5 surpassing 79% on HotpotQA, outperforming baselines by significant margins.

DynamicRAG achieved an EM of 48.4% on NQ with LLaMA3-8B, outperforming static models and non-retrieval baselines. On HotpotQA, R@5 reached 79.4%, demonstrating robustness across tasks. Ablation experiments confirmed that removing RL or dynamic adjustment reduced performance by at least 5%. The model required only 20k training samples, yet matched or exceeded the performance of models trained on 50k+ samples, indicating high data efficiency. These results validate the effectiveness of feedback-driven dynamic reranking in knowledge-intensive tasks.

This framework can be deployed in real-time QA systems, intelligent assistants, and knowledge bases, where adaptive retrieval improves accuracy and relevance. It is particularly suited for scenarios requiring up-to-date information, such as medical diagnosis, legal research, and scientific literature review. The ability to self-adjust retrieval content based on query complexity makes it valuable for personalized and context-aware applications. Future integration with multi-modal data could further enhance its utility in multimedia information retrieval.

Despite promising results, the approach faces challenges in highly ambiguous queries due to sparse reward signals and computational overhead. The reliance on high-quality expert trajectories limits scalability. Real-time deployment requires further optimization for inference speed. Additionally, extending the framework to multi-modal or multi-task settings introduces complexity that needs addressing. Future work should focus on improving reward efficiency, reducing training costs, and ensuring robustness across diverse environments.