Dipartimento di Ingegneria informatica, automatica e gestionale

In this talk, we investigate the robustness and reliability of benchmark-based evaluation for Large Language Models by systematically paraphrasing benchmark questions. We show that small wording changes can alter model predictions and reduce accuracy across multiple benchmarks and models, even when overall rankings remain stable. These findings suggest that current benchmarks may overestimate true generalization and motivate paraphrase-aware evaluation practices.

 

Kevin Roitero is a tenure-track Assistant Professor at the University of Udine. His research focuses on AI, NLP, and Information Retrieval, with work published at top venues such as SIGIR, WSDM, WWW, and CIKM, and recognized with multiple awards.