Publications

This page contains a list of all publications that followed from the DATA2GAME research project.

• Modeling Behavioral Competencies In Crisis Management Scenarios
  Paris Mavromoustakos-Blom, Johannes Steinrücke, Sander Bakkes, Pieter Spronck. Psychonomics Society 2018. Amsterdam, The Netherlands.
• Personalised Crisis Management Training on a Tablet
  Paris Mavromoustakos, Sander Bakkes, Pieter Spronck. Proceedings of Foundations of Digital Games 2018 (FDG2018). Malmö, Sweden. August 7–10 2018.
  Abstract: In this paper, we propose a framework for personalised crisis management training through the use of an applied game. The framework particularly focuses on ubiquitously assessing and manipulating player stress levels during training, and evaluating player performance by providing personalised feedback. To achieve these goals, the framework leverages techniques for multi-modal player modeling through physiological sensors, in-game events and selfreport data. Specifically, the present paper (1) discusses design decisions for the personalised crisis management training framework, and (2) presents the game prototype with which user-studies will be performed. Presently, the game prototype is being developed in close collaboration with actual crisis management experts. [Full text]
• Towards Generating Textual Game Assets from Real-World Data
  Judith van Stegeren, Mariët Theune. Proceedings of Foundations of Digital Games 2018 (FDG2018). Malmö, Sweden. August 7–10, 2018.
  Abstract: We propose using real-world datasets to generate textual game assets for serious games. As an example, we used a dataset of P2000 crisis event messages to generate descriptive texts that can be transformed into new game assets by game writers, thereby reducing the writing effort required during the development phase of an adaptive serious game. In this paper we describe this first attempt and we discuss the challenges and possibilities of using open data for textual asset generation. [Full text]
• Determining the effect of stress on analytical skills performance in digital decision games: Towards an unobtrusive measure of experienced stress in gameplay scenarios
  Steinrücke, J., Veldkamp, B. P. & De Jong, T.. Computers in Human Behavior. 16 May 2019.
  Abstract: This study aims to develop an unobtrusive measure for experienced stress in a digital serious gaming environment involving decision making in crisis management, using only in-game measures in a digital decision game called the Mayor Game. Research has shown that stress has an influence on a decision-maker's behavior, and also on the learning experience in training scenarios. Being able to assess unobtrusively the level of stress experienced would allow manipulation of the game so as to improve the learning experience. An experiment was conducted with two conditions, one paced and one non-paced. In the paced condition, participants were exposed to in-game changes that aimed to induce stress by creating information overload, uncertainty and time pressure. While pacing caused differences between the conditions with respect to in-game performance for analytical skills, several simple unobtrusive in-game measures were not consistent enough to serve as indicators for experienced stress. Further, physiological measurements of stress did not show significant differences between the conditions, indicating that the employed methods to induce stress did not work sufficiently. These results call for testing of more sophisticated methodologies to unobtrusively assess experienced stress in the given type of serious game. [Full text]
• Churnalist: Fictional Headline Generation for Context-appropriate Flavor Text
  Judith van Stegeren, Mariët Theune. International Conference on Computational Creativity, Charlotte, NC, USA. June 17 - June 21, 2019.
  Abstract: We present Churnalist, a headline generator for creating contextually-appropriate fictional headlines that can be used as ‘flavor text’ in games. Churnalist creates new headlines from existing headlines with text modification. It extracts seed words from free text input, queries a knowledge base for related words and uses these words in the new headlines. Churnalist’s knowledge base consists of a dataset of pre-trained word embeddings, thus requiring no linguistic expertise or hand-coded models from the user. [Full text]
• Narrative Generation in the Wild: Methods from NaNoGenMo
  Judith van Stegeren, Mariët Theune. Second workshop on Storytelling (StoryNLP), ACL. Florence, Italy. August 1, 2019.
  Abstract: In text generation, generating long stories is still a challenge. Coherence tends to decrease rapidly as the output length increases. Especially for generated stories, coherence of the narrative is an important quality aspect of the output text. In this paper we examine how narrative coherence is attained in the submissions of NaNoGenMo 2018, an online text generation event where participants are challenged to generate a 50,000 word novel. We list the main approaches that were used to generate coherent narratives and link them to scientific literature. Finally, we give recommendations on when to use which approach. [Full text]
• Remixing Headlines for Context-Appropriate Flavor Text
  Judith van Stegeren, Mariët Theune. IEEE Conference On Games, London, UK. August 20-23, 2019.
  Abstract: We describe a prototype of Churnalist, a headline generator for creating contextually-appropriate fictional headlines that can be used as flavor text in games. Churnalist creates new headlines by remixing existing headlines. It extracts seed words from free text input, searches for related words in a dataset of word embeddings and uses these words in the new headlines. The system requires no linguistic expertise or hand-coded language models from the user. [Full text]
• Towards Multi-modal Stress Response Modelling in Competitive League of Legends
  Paris Mavromoustakos-Blom, Sander Bakkes, Pieter Spronck. IEEE Conference On Games, London, UK. August 20-23, 2019.
  Abstract: With the constant rise in popularity of competitive video gaming (also known as Esports), Esports analytics has been a field of growing scientific interest in the recent years. Studies discussing player behaviour, skill learning and team performance have been conducted through Multiplayer Online Battle Arena games such as League of Legends. In this paper, we propose a multi-modal approach towards stress response modeling in competitive LoL games. We collect wearable physiological sensor data, mouse & keyboard logs and in-game data in order to study the relationship between player stress responses and in-game behaviour. We discuss the design criteria and propose future studies using the collected dataset. [Full text]
• Modeling and adjusting in-game difficulty based on facial expression analysis
  Paris Mavromoustakos-Blom, Sander Bakkes, Pieter Spronck. Entertainment Computing. September 20, 2019.
  Abstract: In this paper we introduce Facial Expression Analysis (FEA) both as a means of predicting in-game difficulty and as a modeling mechanism, based on which we develop in-game difficulty adjustment algorithms for single player arcade games. Our main contribution is the implementation of an online and unobtrusive game personalisation system. On the basis of FEA, our system is able to adapt the difficulty level of the game to the individual player, without interruptions, during actual gameplay. Specifically, we study (a) how perceived in-game difficulty can be measured through facial expression analysis, and (b) how facial expression data can model player behavior and predict their affective state. Experimental findings reveal that different in-game difficulty settings can be correlated to distinct player emotions (revealed in facial expressions). Furthermore, a model based on facial expression analysis is successfully applied to calculate an appropriate difficulty setting, tailored to the individual player. From these results, we may conclude that efficient game personalisation is achievable through FEA. [Full text]
• Andromeda: A Personalised Crisis Management Training Toolkit
  Paris Mavromoustakos Blom, Sander Bakkes, Pieter Spronck. In: Liapis A., Yannakakis G., Gentile M., Ninaus M. (eds) Games and Learning Alliance. GALA 2019. Lecture Notes in Computer Science, vol 11899. Springer, Cham. November 1, 2019.
  Abstract: Over the last decades, technological advancements have enabled the gamification of many of modern society’s processes. Crisis management training has benefited from the introduction of human-machine interfaces (HMIs) and wearable monitoring sensors. Crisis responders are nowadays able to attend training sessions through computer-simulated crisis scenarios while simultaneously receiving real-time feedback on their operational and cognitive performance. Such training sessions would require a considerable amount of resources if they were to be recreated in the real world. We introduce Andromeda, a toolkit designed to allow remote-access, real-time crisis management training personalisation through an applied game. Andromeda consists of a browser-based dashboard which enables real-time monitoring and adaptation of crisis management scenarios, and a remote server which securely stores, analyses and serves training data. In this paper, we discuss Andromeda’s design concepts and propose future studies using this toolkit. Our main focal points are player stress response modelling and automated crisis management training adaptation. [Full text]
• Creating a Sentiment Lexicon with Game-Specific Words for Analyzing NPC Dialogue in The Elder Scrolls V: Skyrim
  Thérèse Bergsma, Judith van Stegeren, Mariët Theune. Games and NLP Workshop, LREC 2020. May 11, 2020.
  Abstract: A weak point of rule-based sentiment analysis systems is that the underlying sentiment lexicons are often not adapted to the domain of the text we want to analyze. We created a game-specific sentiment lexicon for video game Skyrim based on the E-ANEW word list and a dataset of Skyrim’s in-game documents. We calculated sentiment ratings for NPC dialogue using both our lexicon and E-ANEW and compared the resulting sentiment ratings to those of human raters. Both lexicons perform comparably well on our evaluation dialogues, but the game-specific extension performs slightly better on the dominance dimension for dialogue segments and the arousal dimension for full dialogues. To our knowledge, this is the first time that a sentiment analysis lexicon has been adapted to the video game domain. [Full text]
• The Effect of Self-Reflection on Information Literacy Performance in a Digital Serious Game (abstract)
  Johannes Steinrücke, Bernard P. Veldkamp, Ton de Jong. Book of Accepted Abstracts for the EARLI SIG14 Conference 2020 in Barcelona. June 30, 2020.
  Abstract: Information literacy is a skill consisting of multiple facets. Being information literate is particularly beneficial in decision making processes in crisis management, where decision makers have to work with limited time and data about the situation. In order to train decision makers on information literacy, digital serious games offer the advantage of being an accessible training method, which the trainees can use on more frequent basis than traditional, analog training methods. Given that self-reflection is already part of the post-training discussion and reflection sessions in traditional crisis management training and that it has proven to be a beneficial instructional intervention in digital serious gaming environments, self-reflection moments in a digital serious game will be used in this study to trigger the trainees to rethink and thereby adjust their in-game training behavior. Consequently, this study investigates the effect of a self-reflection moment in a digital serious game for crisis management decision making training. [Full text]
• Information literacy skills assessment in digital crisis management training for the safety domain: Developing an unobtrusive method
  Johannes Steinrücke, Bernard Veldkamp, Ton de Jong. Frontiers in Education. July 13, 2020.
  Abstract: This study aims to develop an unobtrusive assessment method for information literacy in the context of crisis management decision making in a digital serious game. The goal is to only employ in-game indicators to assess the players’ skill level on different facets of information literacy. In crisis management decision making it is crucial to combine an intuitive approach to decision making, build up by experience, with an analytical approach to decision making, taking into account contextual information about the crisis situation. Situations like these have to be trained frequently, for example by using serious games. Adaptivity can improve the effectiveness and efficiency of serious games. Unobtrusive assessment can enable game developers to make the game adapt to the players current skill level without breaking the flow of gameplay. Participants played a gameplay scenario in the Dilemma Game. Additionally, participants completed a questionnaire that was used as a validation measure for the in-game information literacy assessment. Using latent profile analyses, unobtrusive assessment models could be identified, most of which correlate significantly to the validation measure scores. Although inconsistencies in correlations between the information literacy standards, which call for broader testing of the identified unobtrusive assessment models, have been observed, the results display a good starting point for an unobtrusive assessment method and a first step in the development of an adaptive serious game for information literacy in crisis management decision making. [Full text]
• Multi-Modal Study of the Effect of Time Pressure in a Crisis Management Game
  Paris Mavromoustakos Blom, Sander Bakkes, Pieter Spronck. Proceedings of Foundations of Digital Games (FDG) Conference 2020, Valetta, Malta. 15 september 2020.
  Abstract: In this paper, we study the effect of time pressure on player behaviour during a dilemma-based crisis management game. We analyse the effects of time pressure onto player behaviour through physiological sensors, in-game actions and self-reports. We employ time pressure as an artificial stressor during gameplay. Our analysis focuses on multi-modal modelling of player behaviour, measuring correlation across modalities and estimating expected player stress levels during gameplay. We were able to create descriptive models of time pressure's effect on player behaviour based on their physiological responses and in-game actions, despite these two modalities not showing a notable correlation. Furthermore, we enable the prediction of player behaviour in real-time by estimating future values of sensor- and in-game-derived features during gameplay. The method presented in this paper can be employed in crisis management training, aiming at assessing players' responses to stressful conditions and manipulating player stress levels to provide personalised training scenarios. [Full text]
• Exploring Peak-End Effects in Player Affect through Hearthstone
  Agner Piton, Paris Mavromoustakos Blom and Pieter Spronck. Proceedings of GAME-ON Conference 2020, Aveiro, Portugal. September 23, 2020.
  Abstract: Peak-end theory suggests that when remembering an experience, people tend to focus on the moments of highest emotional variance and the last moments of the experience. In this paper, we study whether peak-end effects occur in gaming experiences, by comparing real-time to retrospective measurements of player affect. We ran an experiment where each of 26 participants played two games of Hearthstone while their affective state was monitored in real time through self-reporting and facial expression recognition. Additionally, participants submitted a retrospective report on their emotions 24 hours after the experiment took place. Strong correlation was found between the self-reported peak- end and retrospective emotion values, while no correlation was found between the retrospective self-reports and player facial expressions. The results of this study validate that the peak-end rule can be leveraged in order to identify players’ retrospective emotions towards a game experience. [Full text]
• Multi-Modal Study of the Effect of Information Complexity in a Crisis Management Game
  Paris Mavromoustakos Blom, Sander Bakkes, Pieter Spronck. Proceedings of GAME-ON Conference 2020, Aveiro, Portugal. September 23, 2020.
  Abstract: In this paper, we study the effect of information complexity on player in-game behaviour and physiological responses during a dilemma-based crisis management game. We run a user study, where players attempt to solve a crisis scenario while their in-game and physiological activity is being monitored through game logs and wearable physiological sensors. Results show that information complexity has noticeable effects on players' decision making and physiological responses, while moderate correlation was found between specific in-game- and physiology-based behavioural features. This study is focused on exploring behavioural patterns correlated to various levels of information complexity. Our findings can be applied in future studies aiming at designing personalised crisis management training scenarios. [Full text]
• Fantastic Strings and Where to Find Them: The Quest for High-Quality Video Game Text Corpora
  Judith van Stegeren, Mariët Theune. Intelligent Narrative Technologies 2020 workshop. Oct 19, 2020.
  Abstract: High-quality video game text corpora can be used as resources for many types of research, including but not limited to text generation for games. However, these corpora are scarce. We address this issue by proposing a number of quality criteria for video game text corpora, and describing from where such corpora can be obtained. We also present three datasets with game texts from popular video games Torchlight II, Star Wars: Knights of the Old Republic and The Elder Scrolls, together with examples of how these corpora can be used in research. [Full text]
• Improving Dutch sentiment analysis in Pattern
  Lorenzo Gatti, Judith van Stegeren. Computational Linguistics in the Netherlands journal. 12 December 2020.
  Abstract: In this paper we investigate methods for improving the sentiment analysis functionality of Pattern.nl, the Dutch submodule of Pattern, an open-source library for web mining and natural language processing. We discuss the impact on performance of three different potential improvements: extending the module’s internal sentiment lexicon; removing subsets of neutral words from the sentiment lexicon; and improving the algorithm for combining multiple word-level sentiment ratings into a sentence-level sentiment rating. We evaluated the improvements on datasets from the product review domain (books, clothing and music) and a dataset of short emotional stories. The experiments show that lexicon expansion does not lead to better results; new normalization techniques, on the other hand, show a limited but consistent performance increase for sentiment ratings. [Full text]
• Data2Game: Towards an Integrated Demonstrator
  Johannes Steinrücke, Paris Mavromoustakos-Blom, Judith van Stegeren, Ymko Attema, Sander Bakkes, Thomas de Groot, Johan de Heer, Dirk Heylen, Rafal Hrynkiewicz, Ton de Jong, Tije Oortwijn, Pieter Spronck, Mariët Theune, Bernard Veldkamp. AHFE 2021: Advances in Usability, User Experience, Wearable and Assistive Technology. 8 July 2021.
  Abstract: The Data2Game project investigates how the efficacy of computerized training games can be enhanced by tailoring training scenarios to the individual player. The research is centered around three research innovations: (1) techniques for the automated modelling of players’ affective states, based on exhibited social signals, (2) techniques for the automated generation of in-game narratives tailored to the learning needs of the player, and (3) validated studies on the relation of the player behavior and game properties to learning performance. This paper describes the integration of the main results into a joint prototype. [Full text]
• Fine-tuning GPT-2 on annotated RPG quests for NPC dialogue generation
  Judith van Stegeren and Jakub Myśliwiec. The 16th International Conference on the Foundations of Digital Games (FDG) 2021. 2 Augustus 2021.
  Abstract: GPT-2, a neural language model trained on a large dataset of English web text, has been used in a variety of natural language generation tasks because of the language quality and coherence of its outputs. In order to investigate the usability of GPT-2 for text generation for video games, we fine-tuned GPT-2 on a corpus of video game quests and used this model to generate dialogue lines for quest-giver NPCs in a role-playing game. We show that the model learned the structure of quests and NPC dialogue, and investigate how the temperature parameter influences the language quality and creativity of the output artifacts. We evaluated our approach with a crowdsource experiment in which human judges were asked to rate hand-written and generated quest texts on language quality, coherence and creativity. [Full text]
• Correlating Facial Expressions and Subjective Player Experiences in Competitive Hearthstone
  Paris Mavromoustakos Blom, Mehmet Kosa, Sander Bakkes & Pieter Spronck. The 16th International Conference on the Foundations of Digital Games (FDG) 2021. 2 Augustus 2021.
  Abstract: In this study, we used recordings of players’ facial expressions that are captured during competitive Hearthstone games to analyse the correlation between in-game player affective responses and subjective post-game self-reports. With this, we aimed to examine whether eye gaze, head pose and emotions gathered as objective data from face recordings would be associated with subjective experiences of players which were collected in the form of a post-game survey. Data was collected during a live offline Hearthstone competition, which involved a total of 17 players and 31 matches played. Correlation analyses between in-game and post-game variables show that players’ facial expressions and eye gaze measurements are associated with both players’ attention to the opponent and their mood influenced by the opponent. In future research, these results may be used to implement predictive player models. [Full text]