Séminaire de Recherche en Linguistique

All speakers can understand a sentence never heard before, or derive the meaning of a word or a sentence from its parts. And yet, these basic linguistic skills have proven very hard to reach by computational models. The current reported success of machine learning architectures is based on computationally expensive algorithms and prohibitively large amounts of data that are available for only a few, non-representative languages.  To reach better, possibly human-like, abilities in neural networks, we need to develop tasks and data that help us understand their current generalisation abilities and help us train them to more complex compositional skills.

 

One likely reason why people generalise better is that they have a strong prior bias, grounded in the actual structure of the problem. Experimental work has demonstrated that the human mind is predisposed to extract regularities and generate rules from data, in a way that is distinct from the patterns of activation of neural networks. One possible approach to develop more robust methods, then, is to pay more attention to the decomposition of complex observations, disentangling the factors in the generative process that gives rise to the data. 

 

To learn more disentangled linguistic representations, that reflect the underlying linguistic rules of grammar, we have developed a new linguistic task. In this presentation, I will illustrate a novel grammatical dataset generatively constructed to support investigations of current models' linguistic mastery of grammatical agreement rules and their ability to generalise them. We present the logic of the dataset, the method to automatically construct data on a large scale and the architecture to learn them. Through error analysis and several experiments on variations of the dataset, we demonstrate that this language task and the data that instantiate it provide a new challenging testbed to understand generative processes of generalisation and abstraction.

 

This is work done in collaboration with Aixiu An and Maria Rodriguez.