Home » Publication » 26152

Dettaglio pubblicazione

2022, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Pages 4724-4737 (volume: 1)

Nibbling at the Hard Core of Word Sense Disambiguation (04b Atto di convegno in volume)

Maru Marco, Conia Simone, Bevilacqua Michele, Navigli Roberto

With state-of-the-art systems having finally attained estimated human performance, Word Sense Disambiguation (WSD) has now joined the array of Natural Language Processing tasks that have seemingly been solved, thanks to the vast amounts of knowledge encoded into Transformer-based pre-trained language models. And yet, if we look below the surface of raw figures, it is easy to realize that current approaches still make trivial mistakes that a human would never make. In this work, we provide evidence showing why the F1 score metric should not simply be taken at face value and present an exhaustive analysis of the errors that seven of the most representative state-of-the-art systems for English all-words WSD make on traditional evaluation benchmarks.In addition, we produce and release a collection of test sets featuring (a) an amended version of the standard evaluation benchmark that fixes its lexical and semantic inaccuracies, (b) 42D, a challenge set devised to assess the resilience of systems with respect to least frequent word senses and senses not seen at training time, and (c) hardEN, a challenge set made up solely of instances which none of the investigated state-of-the-art systems can solve. We make all of the test sets and model predictions available to the research community at https://github.com/SapienzaNLP/wsd-hard-benchmark.
Gruppo di ricerca: Natural Language Processing
© Università degli Studi di Roma "La Sapienza" - Piazzale Aldo Moro 5, 00185 Roma