Multi-modal emotion measurement in an interactive computer-game: 
A pilot- study.

Kaiser, S., Wehrle, T., & Edwards, P. (1994)
kaiser@fapse.unige.ch , wehrle@fapse.unige.ch
In N., H. Frijda (Ed.), Proceedings of the VIIIth Conference of the International Society for Research on Emotions (pp. 275-279). Storrs, CT: ISRE Publications.

 Introduction*

This poster presents a pilot-study of a research project proposing dynamic man- computer interactions as a research method in emotion psychology. The objective of the project is to study the dynamics and interactive nature of emotion with a multi-modal approach. An interactive computer game setting is presented, being more complex than classical experiments but better controllable than real interactions. The computer game was implemented in our group, thus allowing to change and control important parameters (like the probability to win or loose) and to develop experimental settings involving controlled manipulations. During the game, the subjects' facial expressions are videotaped serving as on-line indicators of their emotional involvement.

Research context

Common emotion induction procedures consist of presentation of emotional video sequences as stimuli, or mental imagery, where subjects are ask to imagine, as vividly as possible, an occasion during which they felt a strong emotion in the past. These two approaches have serious drawbacks in common: The dynamic properties of emotion episodes are not included or studied in these experimental settings, and the subject is not involved in an interaction but either looking at a film as an observer or remembering an emotional situation. With a computer game we have an experimental situation which is interactive. The player is involved and not only observer. Moreover, the computer game situation lend itself to the study of the dynamics of emotional processes because all data on the game constellation as well as all data on the subject's manipulations and interventions are recorded continuously. These information can be utilized as context for the interpretation of simultaneously recorded facial behavior and vice versa. The mentioned multi- modality of the approach can be achieved in collaboration with Klaus Scherer, allowing us to measure not only facial behavior and subjective reports, but also vocal and psychophysiological indicators of emotions.

Feasibility study

A major goal of the project is to develop and implement an interactive computer game, whose scenarios can be varied systematically. The design of this computer game will be guided by the theoretical framework of so called appraisal theories of emotion and especially Scherer's component process theory of emotion. In the study presented here, we used a pilot version of the game to test the feasibility of our approach and to gain experiences for the development of the final tool. More concretely, we wanted to test the capacity of such a game to induce distinct emotions and to elicit facial expressions. The prototype of the game was adapted to mainly induce irritation/anger. While playing the game, subjects were videotaped and facial expressions were coded with the Facial Action Coding System (FACS; Ekman & Friesen). Facial expressions supposed to be indicators of irritation/anger were identified by applying Ekman & Friesen's Emotion Prediction Table, specifying the Action Units that are part of the universally found anger expression. The general hypothesis was that if we were able to induce anger/irritation within this experimental setting we should find facial indicators for anger/irritation and subjects should report more anger/irritation at the end of the game than before starting the experimental game. This poster presents first results for facial expressions and discusses their relevance for our further research.

Method

Procedure
40 subjects (female psychology students at the University of Geneva) participated in the experiment. They filled out a questionnaire, in which they were asked to indicate their current emotional state. In order to motivate the subjects to take the experimental situation seriously they were told that they would participate in an experiment on motor- visual coordination and that they would first - as an aptitude test for the real experimentation - exercise their skills in a computer game. Moreover, they were told that 85% of subjects tested at another university had managed the game successfully. After a 10 minute training phase the experimental phase of the computer game began and facial expressions were videotaped. After five minutes the game stopped automatically and subjects' emotional state was assessed with Scherer's computerized questionnaire (a part of the GENESE system, Scherer 1993).

The computer game
In the computer game the subjects had to navigate a vehicle as fast as possible through a kind of tunnel, a constantly changing obstacle course to obtain a special bonus in a given time period. This involved the risk of running into walls or of crashing into suddenly occurring obstacles or objects of different kinds. An object can have a negative or a positive character. Hitting them either costs or provides points, energy, competence etc., so that the player should either avoid or try to hit them actively.

Experimental manipulations
Among a variety of possible antecedent for anger, Ekman mentions "frustration resulting from interference with your activity or the pursuit of your goals" and "verbal attacks which causes you to feel psychologically hurt". Furthermore he assumes that the intensity of anger can vary, ranging from simple irritation to hot rage. In order to provoke irritation/anger we operationalized Ekmans predictions using 4 concrete experimental manipulations:

  1. The difficulty of avoiding harmful encounters increased continually after the first two minutes of the game (higher velocity, more narrow passages in the tunnel, more negative obstacles).
  2. After a crash, messages appeared on the screen which became more and more urgent (Hurry up!!) or mocking (You are really doing a good job !!).
  3. During the fifth minute the control of the vehicle was blocked three times for ten seconds.
  4. After 5 minutes the game stopped automatically and a message appeared telling the subject that she had failed like 15% of the students of the first study.

FACS coding
Subjects' videotaped facial behavior was coded using FACS for the first and the last minute of the computer game. For the purpose of this pilot-study only the apex of Action Units or Action Unit combinations was coded. Since the anger inducing manipulations peek in the final phase of the game, we expected to find more facial indicators for anger in the fifth minute than in the first minute.

Concretely, the following differences were hypothesized:

  1. The frequency of facial expressions (Action Units) will be higher during the fifth minute than in the first minute of the game.
  2. The frequency of Action Units indicating irritation/anger (as postulated by Ekman & Friesen), namely AU 4 (brow lower), 5 (upper lid raise), 7 (lids tight), 10 (upper lip raise), 17 (chin raise), 23 (lip funnel), 24 (lip press) will be higher during the fifth minute than in the first minute of the game.
  3. Subjects reporting irritation/anger will show more Action Units indicating irritation/anger in the fifth minute of the game than subjects reporting another emotion.

Results

In accordance with hypothesis 1, subjects showed significantly more Action Units during the fifth minute than during the first minute of the game (Wilcoxon test; p = 0.02). Contrary to hypothesis 2, we found no significant difference for the frequency of Action Units indicating irritation/anger. In line with hypothesis 3, subjects reporting anger (N = 7) showed significantly more anger relevant Action Units than subjects reporting other emotions (N = 33; T test; p = 0.008). The amount of anger relevant Action Units shown by subjects reporting anger was as well significantly higher than the amount shown by subjects reporting positive emotions (N = 12; T test; p = 0.037) as the amount shown by subjects reporting other negative emotions (N = 21; T test; p = 0.015). One explanation for these results could be that not only anger but also other emotions had been induced. Another, complementary explanation could be that this list of anger relevant Action Units is only a subset of a persons anger repertory. Indeed, there is a correlation between these Action Units and the experience of anger if anger is verbally reported (labeled) by the subject.


Figure 1: Examples of Action Units indicating anger (as postulated by Ekman & Friesen).

In addition, we looked in a more descriptive manner on which other Action Units occurred. Action Unit 12 (lip corner pull) and Action Unit combination AU1 + AU2 (inner and outer brow raise) were shown by most of the subjects (often combined with various other AUs). Other Action Units or Action Unit combinations did not occur regularly. Facial patterns involving AU1 + AU2 look rather different, depending on the co-occurring Action Units (see Figure 2). Intuitively, the patterns can be described as raging from more or less neutral surprise to expressions of negative emotions like consternation or contempt.


Figure 2: Examples of facial expressions involving AU1 (inner brow raise) + AU2 (outer brow raise).


Figure 3: Examples of facial expressions involving AU12 (lip corner pull).

As can be seen in figure 3, AU12 (lip pulling) was found in combination with AU6 (cheek raise), commonly interpreted as felt smile, indicating that the game was not only frustrating but also amusing. But there were also asymmetric smiles, smiles that were too abrupt, or smiles combined with signs of negative emotions, i.e. smiles used to mask or control negative emotions. Moreover, subjects showed a variety of Action Units indicating negative emotions (e.g. disgust, fear), and also AUs which are not part of an emotion prediction table (e.g. blowing, etc. see figure 4). In summary, we found on the one hand a large inter- individual variance and a variety of facial expressions. On the other hand we found a considerable intra- individual consistency, i.e. each subject showed a kind of "typical" facial repertory (see first column in figure 5). These results reflect a well known problem in emotion research concerning the interpretation of facial behavior in terms of a global, context free, and inter-individually valid emotion prediction table.

Discussion

The results of this pilot- study are quite promising. Although general facial activity may be lower in the computer game situation than in person- person interactions, the facial expressions which did occur were rather intense and indicators of emotional involvement. Even this rather simple game elicited a relatively large range of facial expressions.


Figure 4: Examples of various Action Unit combinations.


Figure 5: The first column shows an example of a "typical" expression repertory for one subject. The second column shows the dynamics of a rapid follow up of different facial expressions.

A positive side effect of the computer game setting is that subjects do not talk (at least most of the time), because speech is always a problem for the FACS coding. Although only seven subjects out of 40 explicitly described their emotional experience during the game as irritation/anger, the relative frequency of anger relevant Action Units was rather high for all subjects (44% of the overall facial activity for subjects reporting anger, 36% for subjects reporting other negative emotions, and 34% for subjects reporting positive emotions). One possible explanation for this discrepancy between facial indicators and verbal report could be the following: During the game, subjects did not only experience one emotion, but rather a flow of emotional processes (see second column of figure 5). Such sequences of different emotional experiences cannot be captured by asking subjects post hoc for a single verbal label. However, there are also other explanations, like the effect of social desirability etc., which also may play a role. The preliminary data reported here reinforce our belief that a process oriented and multi-modal approach is required if one wants to learn more about the dynamics and mechanisms characterizing emotional episodes. We are optimistic that the computer game approach can also be used to induce other - negative and positive - emotions.

 * - This project is supported by grants from the Swiss National Science Foundation (FNRS 11-39551.93 "Dynamic man-computer interactions as a paradigm in emotion research") granted to S. Kaiser. We would like to thank C. Delaloye, A. Gay, C. Ludwig and S. Veraguth for their help in carrying out the experiments.