Séminaire de Recherche en Linguistique

The True Promises of Large Language Models

I will present the work and outputs of the project SNF Advanced Disentangling Linguistic Intelligence, a project that aims to investigate the ability of large language models to learn and represent linguistic rules, presented as regularities underlying extensional paradigms. I will present the novel task that we have develop to study this ability, the Blackbird language matrices, a language-oriented task, modelled on the well-known Raven progressive matrices, visual tests of IQ and intelligence for humans. I will then present the implementation of this task and the templates and corresponding datasets that we have developed for many problems and languages. The different machine experiment results show that  LLMs can solve the task, learn phrases and their attributes and some of the underlying rules, but struggle with cross-linguistic abstraction. I will briefly conclude with an opinionated contextualisation of the meaning of this work in the current debate about the relevance of large language models for the study of language.