Bi'an: A Bilingual Benchmark and Model for Hallucination Detection in Retrieval-Augmented Generation

The rapid development of LLMs like GPT-3, BERT, and their successors has significantly advanced NLP capabilities. However, hallucinations—producing unsupported or false information—persist as a major obstacle, especially in knowledge-intensive tasks. Early efforts focused on model fine-tuning and knowledge distillation, but lacked systematic, multilingual evaluation tools. Benchmarks like HaluBench and RAGTruth provided initial datasets, yet they lacked coverage across multiple tasks and languages. With retrieval-augmented generation (RAG) gaining popularity, integrating external knowledge sources, the need for robust hallucination detection has become urgent. Existing methods often rely on proprietary large models, which are costly and less accessible. Therefore, developing lightweight, scalable, and multilingual evaluation frameworks is crucial for practical deployment and trustworthiness.

Current hallucination detection approaches are limited by their dependence on large, expensive models like GPT-4, hindering scalability. The absence of comprehensive, multilingual benchmarks hampers systematic evaluation across diverse tasks and domains. Additionally, models trained solely on large-scale data often struggle with knowledge conflicts and domain-specific hallucinations. This creates a bottleneck for deploying reliable RAG systems in real-world applications, where efficiency, cost, and robustness are critical. Addressing these issues requires constructing diverse, high-quality datasets and designing lightweight models capable of accurate hallucination detection across languages and tasks.

The core innovations include: 1) constructing Bi'anBench, a large-scale bilingual benchmark covering four key RAG tasks, with synthetic and counterfactual data generation pipelines; 2) employing GPT-4o for semantic perturbation and counterfactual QA synthesis, enriching test scenarios; 3) developing a two-stage training pipeline combining supervised fine-tuning with DPO, leveraging LoRA for parameter efficiency; 4) implementing an ensemble-based sample construction method that enhances data diversity and model robustness. These innovations enable high detection accuracy with lightweight models, reducing reliance on costly large models and facilitating scalable deployment.

• �� Data collection: Curate bilingual datasets for question answering, summarization, data-to-text, and translation from open sources.
• �� Synthetic data generation: Use GPT-4o to introduce semantically plausible but factually inconsistent perturbations, creating hallucination samples.
• �� Counterfactual QA pipeline: Generate question-context-answer triplets with supporting or contradictory evidence, expanding the dataset.
• �� Sample construction: Apply stratified sampling from source datasets, then use ensemble models (GPT-4o, Qwen2.5, Mini) to produce reasoning and judgments, selecting high-quality samples.
• �� Model training: Conduct supervised fine-tuning with LoRA, followed by DPO-based preference learning, on 7B and 14B models.
• �� Training process: Merge LoRA adapters with base models, perform distributed training, and validate performance across multiple tasks and languages.

Experiments evaluate models on Bi'anBench's bilingual subsets, measuring accuracy in hallucination detection. Baselines include GPT-4o, Qwen2.5, and Llama variants. Hyperparameters are tuned for optimal performance, with three runs averaged. Ablation studies analyze the impact of training stages. The models are tested on both standard and counterfactual subsets, assessing generalization and knowledge conflict sensitivity. Results show the 14B Bi'an model achieves 88.2% average accuracy, outperforming larger baselines, validating the effectiveness of the training pipeline and data augmentation strategies.

The lightweight Bi'an models, especially the 14B variant, outperform larger models like Qwen2.5-72B in hallucination detection, with accuracy gains of 5-10%. In bilingual tasks, scores exceed 84%, with 93.3% on counterfactual QA. Ablation results confirm that the two-stage training significantly boosts performance. Knowledge conflict analysis indicates that models with richer parametric knowledge sometimes struggle, highlighting the importance of training strategies. Overall, the models demonstrate a promising balance of efficiency and accuracy, suitable for practical deployment.

This framework enables reliable content verification in AI-powered chatbots, search engines, and knowledge bases. Its low computational cost makes it suitable for edge devices and real-time systems. The bilingual nature supports global applications, especially in multilingual environments. Future integration with multimodal data could extend its use to images and videos, broadening trustworthiness in AI systems.

Models still face challenges with long-context reasoning and numerical accuracy. Synthetic data reliance on GPT-4o incurs high costs, limiting scalability. The dataset lacks subjective tasks like creative writing, which are important in real-world scenarios. Knowledge conflicts, especially in models with extensive parametric knowledge, can impair detection accuracy. Further research is needed to improve robustness, reduce computational costs, and expand coverage to subjective and multimodal tasks.