RAG em domínio normativo: avaliação de recuperadores e diagnóstico de viés em dados sintéticos do PPC-CC

Abstract

Retrieval-Augmented Generation (RAG) systems enhance the reliabi- lity of Large Language Models (LLMs) by grounding responses in external sources. However, RAG effectiveness critically depends on the information retrieval (IR) com- ponent, whose behavior in specific technical domains remains underexplored. This work conducts an experimental and qualitative investigation of the factors influencing the performance of lexical (BM25, TF-IDF), dense (specialized embeddings), and hy- brid (SPLADE) retrievers in a real-world case: a question-answering system for the Computer Science Course Pedagogical Project (PPC) at UFFS. Results demonstrate that, in this normative domain, traditional lexical methods performed equally or supe- riorly to more complex neural approaches. The analysis further revealed: (i) structural biases in synthetic evaluation datasets, which artificially inflate metrics for term-based methods; and (ii) significant practical limitations in transferring the hybrid SPLADE model to Brazilian Portuguese, due to noisy lexical expansions. The study highlights that the choice of IR method must consider the intrinsic characteristics of the target domain, challenging the assumed automatic superiority of neural approaches.