MATCHA: Matching Text via Contrastive Semantic Alignment

The evolution of NLP evaluation metrics reflects a shift from simple lexical overlap methods like ROUGE and BLEU to embedding-based approaches such as BERTScore and BLEURT. While early metrics provided quick assessments, they lacked semantic depth. Embedding metrics improved alignment with human judgments but remained vulnerable to adversarial inputs and failed to detect contradictions effectively. The rise of large language models (LLMs) increased the demand for more nuanced evaluation tools capable of handling complex semantic nuances, factual correctness, and adversarial robustness. Despite progress, existing metrics often produce high similarity scores for semantically unrelated or contradictory texts, limiting their reliability in real-world applications. This gap motivates the development of more sophisticated, contrastive, and interpretable evaluation methods.

Current evaluation metrics struggle to reliably distinguish between semantically correct and contradictory texts, especially in adversarial or nuanced scenarios. Lexical overlap metrics cannot capture deep semantic differences, while embedding-based metrics often assign high similarity scores to unrelated or contradictory pairs. This leads to over-optimistic assessments of model outputs, masking fundamental errors. The core challenge is designing a metric that is both sensitive to subtle semantic contradictions and interpretable at the token level, ensuring alignment with human judgments and robustness against adversarial attacks. Addressing this problem is crucial for trustworthy NLP system deployment, especially in knowledge-intensive tasks like fact verification and question answering.

MATCHA introduces several key innovations: 1) A contrastive learning framework that explicitly penalizes contradictions by training on positive (correct) and negative (contradictory) pairs; 2) Multi-view semantic measurement, combining proximity to reference texts with distance from adversarial contradictions; 3) Token-level interpretability via semantic alignment, enabling detailed error diagnosis; 4) Margin-based contrastive loss that sharpens decision boundaries, improving discriminability. These innovations collectively enable MATCHA to outperform traditional metrics in robustness, interpretability, and alignment with human judgments, setting a new standard for semantic evaluation.

• �� Texts are encoded into token embeddings using GPT-2; • Embeddings are projected into a shared semantic space via a linear layer; • Document-level representations are obtained through mean pooling; • Similarity is computed using cosine similarity; • During training, positive pairs (reference and correct text) are contrasted with negative pairs (reference and contradictory text) using a margin-based loss; • Adversarial samples are generated to enhance sensitivity to contradictions; • At inference, texts are encoded and scored based on similarity, with higher scores indicating better semantic alignment. This process ensures the model learns to differentiate subtle semantic nuances effectively.

The evaluation involves eight datasets covering natural language inference, question-answering, factuality, image captioning, and semantic similarity. Baselines include ROUGE, METEOR, BERTScore, BLEURT, and MAUVE. The training uses contrastive sampling with hyperparameters tuned for margin and learning rate. Metrics are assessed via similarity gaps, macro-F1, Wasserstein distances, and correlation with human judgments. Ablation studies analyze the impact of multi-view comparison and adversarial training. Results demonstrate that MATCHA consistently achieves larger semantic gaps, better discrimination, and higher correlation with human ratings across tasks.

MATCHA outperforms all baselines, with an 18.38% improvement over ROUGE-L and 20.82% over BERTScore on TruthfulQA. It achieves the highest separation margins in multiple datasets, with average differences exceeding 35%. The macro-F1 scores surpass 70% in most benchmarks, indicating reliable classification. The Wasserstein distances confirm superior distributional separation. Embedding comparisons show MATCHA's ability to assign negative similarity to incorrect pairs, a feat unmatched by other models, confirming its enhanced discriminative and interpretative capabilities.

MATCHA can be integrated into automated evaluation pipelines for large language models, content moderation, and fact-checking systems. It is particularly useful in scenarios requiring high semantic fidelity, such as summarization, question answering, and dialogue systems. Its interpretability allows developers to diagnose errors at the token level, facilitating targeted improvements. Long-term, the framework can be extended to multi-modal data, supporting applications in vision-language tasks, and adapted for multilingual evaluation, broadening its industrial and research impact.

Despite its robustness, MATCHA's computational cost remains high, limiting real-time deployment in resource-constrained environments. Its performance on highly ambiguous, multi-meaning, or low-resource language texts needs further validation. The current training relies on large contrastive datasets, which may not be available for all languages or domains. Future work should focus on optimizing efficiency, reducing data requirements, and extending multi-modal and multilingual capabilities.