Validação automática de respostas técnicas automotivas utilizando modelos de linguagem e recuperação de informação

Abstract

Advances in large language models have enabled the automation of complex tasks involving the evalu- ation of technical content. This study investigates the use of Large Language Models to validate the coherence of Question–Answer (QA) pairs extracted from an automotive forum, comparing three approaches: zero-shot classification, retrieval based on semantically similar questions (few-shot), and retrieval based on excerpts from technical manuals. Experimental results indicate a progressive improvement in performance as more structured context is introduced. While the zero-shot approach yields F1-scores below 50%, the use of similar questions as contextual examples provides moderate gains. The best results are achieved with manual-based retrieval, which increases the F1-score to over 70%, highlighting the importance of authoritative technical documentation for reliable automatic answer validation.