Abstract Current review papers in the area of Affective Computing and Affective Gaming point to a number of issues with using their methods in out-of-the-lab scenarios, making them virtually impossible to be deployed. On the contrary, we present a game that serves as a proof-of-concept designed to demonstrate that—being aware of all the limitations and addressing them accordingly—it is possible to create a product that works in-the-wild. A key contribution is the development of a dynamic game adaptation algorithm based on the real-time analysis of emotions from facial expressions. The obtained results are promising, indicating the success in delivering a good game experience.
InfFusion2023
J. M. Górriz et al., “Computational approaches to Explainable Artificial Intelligence: Advances in theory, applications and trends,” Inf. Fusion, vol. 100, p. 101945, 2023.
Abstract Deep Learning (DL), a groundbreaking branch of Machine Learning (ML), has emerged as a driving force in both theoretical and applied Artificial Intelligence (AI). DL algorithms, rooted in complex and non-linear artificial neural systems, excel at extracting high-level features from data. DL has demonstrated human-level performance in real-world tasks, including clinical diagnostics, and has unlocked solutions to previously intractable problems in virtual agent design, robotics, genomics, neuroimaging, computer vision, and industrial automation. In this paper, the most relevant advances from the last few years in Artificial Intelligence (AI) and several applications to neuroscience, neuroimaging, computer vision, and robotics are presented, reviewed and discussed. In this way, we summarize the state-of-the-art in AI methods, models and applications within a collection of works presented at the 9th International Conference on the Interplay between Natural and Artificial Computation (IWINAC). The works presented in this paper are excellent examples of new scientific discoveries made in laboratories that have successfully transitioned to real-life applications.
SciData2022
K. Kutt, D. Drążyk, L. Żuchowska, M. Szelążek, S. Bobek, and G. J. Nalepa, “BIRAFFE2, a multimodal dataset for emotion-based personalization in rich affective game environments,” Sci. Data, vol. 9, no. 1, p. 274, 2022
Abstract Generic emotion prediction models based on physiological data developed in the field of affective computing apparently are not robust enough. To improve their effectiveness, one needs to personalize them to specific individuals and incorporate broader contextual information. To address the lack of relevant datasets, we propose the 2nd Study in Bio-Reactions and Faces for Emotion-based Personalization for AI Systems (BIRAFFE2) dataset. In addition to the classical procedure in the stimulus-appraisal paradigm, it also contains data from an affective gaming session in which a range of contextual data was collected from the game environment. This is complemented by accelerometer, ECG and EDA signals, participants’ facial expression data, together with personality and game engagement questionnaires. The dataset was collected on 102 participants. Its potential usefulness is presented by validating the correctness of the contextual data and indicating the relationships between personality and participants’ emotions and between personality and physiological signals.
AfCAI2022
K. Kutt, P. Sobczyk, and G. J. Nalepa, “Evaluation of Selected APIs for Emotion Recognition from Facial Expressions,” in Bio-inspired Systems and Applications: from Robotics to Ambient Intelligence. IWINAC 2022 Proceedings, Part II, 2022, pp. 65–74.
Abstract Facial expressions convey the vast majority of the emotional information contained in social utterances. From the point of view of affective intelligent systems, it is therefore important to develop appropriate emotion recognition models based on facial images. As a result of the high interest of the research and industrial community in this problem, many ready-to-use tools are being developed, which can be used via suitable web APIs. In this paper, two of the most popular APIs were tested: Microsoft Face API and Kairos Emotion Analysis API. The evaluation was performed on images representing 8 emotions—anger, contempt, disgust, fear, joy, sadness, surprise and neutral—distributed in 4 benchmark datasets: Cohn-Kanade (CK), Extended Cohn-Kanade (CK+), Amsterdam Dynamic Facial Expression Set (ADFES) and Radboud Faces Database (RaFD). The results indicated a significant advantage of the Microsoft API in the accuracy of emotion recognition both in photos taken en face and at a 45∘ angle. Microsoft’s API also has an advantage in the larger number of recognised emotions: contempt and neutral are also included.
MRC2021b
L. Żuchowska, K. Kutt, and G. J. Nalepa, “Bartle Taxonomy-based Game for Affective and Personality Computing Research,” in MRC@IJCAI 2021, 2021, pp. 51–55.
Abstract The paper presents the design of a game that will serve as a research environment in the BIRAFFE series experiment planned for autumn 2021, which uses affective and personality computing methods to develop methods for interacting with intelligent assistants. A key aspect is grounding the game design on the taxonomy of player types designed by Bartle. This will allow for an investigation of hypotheses concerning the characteristics of particular types of players or their stability in response to emotionally-charged stimuli occurring during the game.
MRC2021a
K. Kutt, L. Żuchowska, S. Bobek, and G. J. Nalepa, “People in the Context – an Analysis of Game-based Experimental Protocol,” in MRC@IJCAI 2021, 2021, pp. 46–50.
Abstract The paper provides insights into two main threads of analysis of the BIRAFFE2 dataset concerning the associations between personality and physiological signals and concerning the game logs' generation and processing. Alongside the presentation of results, we propose the generation of event-marked maps as an important step in the exploratory analysis of game data. The paper concludes with a set of guidelines for using games as a context-rich experimental environment.
Sensors2021
K. Kutt, D. Drążyk, S. Bobek, and G. J. Nalepa, “Personality-Based Affective Adaptation Methods for Intelligent Systems,” Sensors, vol. 21, no. 1, p. 163, 2021.
Abstract In this article, we propose using personality assessment as a way to adapt affective intelligent systems. This psychologically-grounded mechanism will divide users into groups that differ in their reactions to affective stimuli for which the behaviour of the system can be adjusted. In order to verify the hypotheses, we conducted an experiment on 206 people, which consisted of two proof-of-concept demonstrations: a “classical” stimuli presentation part, and affective games that provide a rich and controllable environment for complex emotional stimuli. Several significant links between personality traits and the psychophysiological signals (electrocardiogram (ECG), galvanic skin response (GSR)), which were gathered while using the BITalino ®evolution kit platform, as well as between personality traits and reactions to complex stimulus environment, are promising results that indicate the potential of the proposed adaptation mechanism.
ICAISC2020
S. Bobek, M. M. Tragarz, M. Szelążek, and G. J. Nalepa, “Explaining Machine Learning Models of Emotion Using the BIRAFFE Dataset,” in ICAISC 2020, vol. 12416 LNAI, Springer, 2020, pp. 290–300.
Abstract Development of models for emotion detection is often based on the use of machine learning. However, it poses practical challenges, due to the limited understanding of modeling of emotions, as well as the problems regarding measurements of bodily signals. In this paper we report on our recent work on improving such models, by the use of explainable AI methods. We are using the BIRAFFE data set we created previously during our own experiment in affective computing.
HAIIW2020
K. Kutt, D. Drążyk, M. Szelążek, S. Bobek, and G. J. Nalepa, “The BIRAFFE2 Experiment – Study in Bio-Reactions and Faces for Emotion-based Personalization for AI Systems.” Paper was presented at HAII workshop during ECAI 2020 (online)
Abstract The paper describes BIRAFFE2 data set, which is a result of an affective computing experiment conducted between 2019 and 2020, that aimed to develop computer models for classification and recognition of emotion. Such work is important to develop new methods of natural Human-AI interaction. As we believe that models of emotion should be personalized by design, we present an unified paradigm allowing to capture emotional responses of different persons, taking individual personality differences into account. We combine classical psychological paradigms of emotional response collection with the newer approach, based on the observation of the computer game player. By capturing ones psycho-physiological reactions (ECG, EDA signal recording), mimic expressions (facial emotion recognition), subjective valence-arousal balance ratings (widget ratings) and gameplay progression (accelerometer and screencast recording), we provide a framework that can be easily used and developed for the purpose of the machine learning methods.
MRC2020
L. Żuchowska, K. Kutt, K. Geleta, S. Bobek, and G. J. Nalepa, “Affective Games Provide Controlable Context. Proposal of an Experimental Framework,” in MRC@ECAI, 2020, vol. 2787, pp. 45–50.
Abstract We propose an experimental framework for Affective Computing based of video games. We developed a set of specially designed mini-games, based of carefully selected game mechanics, to evoke emotions of participants of a larger experiment. We believe, that games provide a controllable yet overall ecological environment for studying emotions. We discuss how we used our mini-games as an important counterpart of classical visual and auditory stimuli. Furthermore, we present a software tool supporting the execution and evaluation of experiments of this kind.
AfCAI2019
K. Kutt, D. Drążyk, P. Jemioło, S. Bobek, B. Giżycka, V. Rodriguez-Fernandez, and G. J. Nalepa, “BIRAFFE: Bio-Reactions and Faces for Emotion-based Personalization,” in Proceedings of the 3rd Workshop on Affective Computing and Context Awareness in Ambient Intelligence (AfCAI 2019), 2020, vol. 2609
Abstract In this paper we introduce the BIRAFFE data set which is the result of the experiment in affective computing we conducted in early 2019. The experiment is part of the work aimed at the development of computer models for emotion classification and recognition. We strongly believe that such models should be personalized by design as emotional responses of different persons are subject to individual differences due to their personality. In the experiment we assumed data fusion from both visual and audio stimuli both taken from standard public data bases (IADS and IAPS respectively). Moreover, we combined two paradigms. In the first one, subjects were exposed to stimuli, and later their bodily reactions (ECG, GSR, and face expression) were recorded. In the second one the subjects played basic computer games, with the same reactions constantly recorded. We decided to make the data set publicly available to the research community using the Zenodo platform. As such, the data set contributes to the development and replication of experiments in AfC.
SEMANTiCS2019
B. Giżycka, K. Kutt, and G. J. Nalepa, “Knowledge-based Development of Games Using Design Patterns Ontology,” in SEMANTiCS 2019 Posters&Demos, 2019, vol. 2451.
Abstract Tools for automatization of knowledge on game mechanics and their interrelationships are still lacking. Game design patterns, as proposed by Björk and Holopainen, seem promising in this area, as they can be represented formally as an ontology. This paper presents our proposal of such a representation, developed using OWL2. We discuss the design of the ontology, and demonstrate how it can be used to conceptualize the design of a classic video game. In the future, the ontology will provide a knowledge base for a new tool for game developers, in order to enable more complex, interesting and emergent game design.
Sensors2019
G. J. Nalepa, K. Kutt, B. Giżycka, P. Jemioło, and S. Bobek, “Analysis and Use of the Emotional Context with Wearable Devices for Games and Intelligent Assistants,” Sensors, vol. 19, no. 11, p. 2509, 2019.
Abstract In this paper, we consider the use of wearable sensors for providing affect-based adaptation in Ambient Intelligence (AmI) systems. We begin with discussion of selected issues regarding the applications of affective computing techniques. We describe our experiments for affect change detection with a range of wearable devices, such as wristbands and the BITalino platform, and discuss an original software solution, which we developed for this purpose. Furthermore, as a test-bed application for our work, we selected computer games. We discuss the state-of-the-art in affect-based adaptation in games, described in terms of the so-called affective loop. We present our original proposal of a conceptual design framework for games, called the affective game design patterns. As a proof-of-concept realization of this approach, we discuss some original game prototypes, which we have developed, involving emotion-based control and adaptation. Finally, we comment on a software framework, that we have previously developed, for context-aware systems which uses human emotional contexts. This framework provides means for implementing adaptive systems using mobile devices with wearable sensors.
ICAISC2019b
M. Z. Łępicki and S. Bobek, “Affective Context-Aware Systems: Architecture of a Dynamic Framework”
Abstract Affective computing gained a lot of attention from researchers and business over the last decade. However, most of the attempts for building systems that try to predict, or provoke affective state of users were done for specific and narrow domains. This complicates reusing such systems in other, even similar domains. In this paper we present such a solution, that aims at solving such problem by providing a general framework architecture for building affective-aware systems. It supports designing and development of affective-aware solutions, in a holistic and domain independent way.
ICAISC2019a
P. Jemioło, B. Giżycka, and G. J. Nalepa, “Prototypes of Arcade Games Enabling Affective Interaction”
Abstract The use of emotions in the process of creating video games is still a challenge for the developers from the fields of Human-Computer Interaction and Affective Computing. In our work, we aim at demonstrating architectures of two operating game prototypes, implemented with the use of affective design patterns. We ground our account in biological signals, i.e. heart rate, galvanic skin response and muscle electrical activity. Using these modalities and the game context, we reason about emotional states of the player. For this purpose, we focus on defining rules with linguistic terms. What is more, we address the need for explainablity of biological mechanics and individual differences in terms of reactions to different stimuli. We provide a benchmark, in the form of a survey, to verify our approach.
CoSECiVi2018
G. J. Nalepa and B. Giżycka, “How a mobile platform for emotion identification supports designing affective games”
Abstract Affective computing is a multidisciplinary area of research regarding modeling, identification, and synthesis of emotions using computer-based methods. Affective gaming is dedicated specifically to developing games that use the information regarding player’s emotional condition. Such games focus on the emotional dimension of gaming experience, to provide greater player engagement. In this short paper we give an overview of our recent works aimed at developing a mobile software platform for emotion identification using wearable devices. Furthermore, we have been working on the integration of this approach with the design and development of affective games
GEM2018
B. Giżycka and G. J. Nalepa, “Emotion in models meets emotion in design: building true affective games”
Abstract A relatively new field of research on affective gaming suggests applying affective computing solutions to develop games that can interact with the player on the emotional level. To bring together selected models of affect and affect-driven frameworks developed to date, we propose an approach based on affective design patterns. We build on the assumption that player’s emotional reactions to in-game events can be evoked by patterns used early in the design phase. We provide description of experiments conducted to test our hypothesis so far, along with some tentative observations, and opportunities for further studies.
HAI2018
B. Giżycka, G. J. Nalepa, and P. Jemioło, “AIded with emotions – a new design approach towards affective computer systems”
Abstract As technologies become more and more pervasive, there is a need for considering the affective dimension of interaction with computer systems to make them more human-like. Current demands for this matter include accurate emotion recognition, reliable emotion modeling, and use of unobtrusive, easily accessible and preferably wearable measurement devices. While AI methods provide many possibilities for better affective information processing, it is not a common scenario for both emotion recognition and modeling to be integrated in the design phase. To address this concern, we propose a new approach based on affective design patterns in the context of video games, together with summary of experiments conducted to test the preliminary hypotheses.
CCSC2018
B. Giżycka, “Using Affective Loop as Auxilliary Design Tool for Video Games”
Presented at the: 10th Cracow Cognitive Science Conference
Abstract As modern technologies become more apparent and persistent, human-computer interaction becomes an important research topic. With birth of affective computing, which aims at developing systems capable of detecting and processing emotionally significant data from the environment, new possibilities for applications unfold, and video games can benefit from them as well. Bringing innovative solutions to this area involves new modes of affective data collection and affect modelling of various aspects of the game experience. My research, focusing on affective game design patterns, is located on the intersection of modelling player affect and affective game design framework. In this paper, an outline of how affective computing ideas (especially affective loop) are introduced to video game design is presented. A new approach to designing video games in the form of affective game design patterns is proposed, together with research method description and summary of studies conducted so far.
HSI2018
K. Kutt, G. J. Nalepa, B. Giżycka, P. Jemioło, and M. Adamczyk, “BandReader – A Mobile Application for Data Acquisition from Wearable Devices in Affective Computing Experiments,” in 2018 11th International Conference on Human System Interaction (HSI), 2018, pp. 42–48.
Abstract In the paper we describe a new software solution for mobile devices that allows for data acquisition from wristbands. The application reads physiological data from wristbands and supports multiple recent hardware. In our work we focus on the Heart Rate (HR) and Galvanic Skin Response (GSR) readings. This data is used in the affective computing experiments for human emotion recognition.
FGCS2018b
J. K. Argasiński and P. Węgrzyn, “Affective patterns in serious games”
Abstract We discuss affective serious games that combine learning, gaming and emotions. We describe a novel framework for the creation and evaluation of serious affective games. Our approach is based on merging pertinent design patterns in order to recognize educational claims, educational assessment, best game design practices, as well as models and solutions of affective computing. Björk’s and Holopainen’s game design patterns have been enhanced by Evidence Centered Design components and affective components. A serious game has been designed and created to demonstrate how to outline a complex game system in a communicative way, and show methods to trace how theoretically-driven design decisions influence learning outcomes and impacts. We emphasize the importance of patterns in game design. Design patterns are an advantageous and convenient way of outlining complex game systems. Design patterns also provide favorable language of communication between multidisciplinary teams working on serious games.
ICAISC2018
K. Kutt, W. Binek, P. Misiak, G. J. Nalepa, and S. Bobek, “Towards the Development of Sensor Platform for Processing Physiological Data from Wearable Sensors,” in ICAISC 2018, vol. 10842 LNAI, Springer, 2018, pp. 168–178.
Abstract The paper outlines a mobile sensor platform aimed at processing physiological data from wearable sensors. We discuss the requirements related to the use of low-cost portable devices in this scenario. Experimental analysis of four such devices, namely Microsoft Band 2, Empatica E4, eHealth Sensor Platform and BITalino ®evolution is provided. Critical comparison of quality of HR and GSR signals leads to the conclusion that future works should focus on the BITalino, possibly combined with the MS Band 2 in some cases. This work is a foundation for possible applications in affective computing and telemedicine.
AfCAI2018
G. J. Nalepa, K. Kutt, S. Bobek, and B. Giżycka, “Development of Mobile Platform for Affect Interpretation. Current Progress,” in Proceedings of the Workshop on Affective Computing and Context Awareness in Ambient Intelligence (AfCAI 2018), 2018.
Abstract In this overview paper we focus on our recent progress in the work on the mobile platform for AfC. We provide the main assumptions about the platform, as well as describe affective data acquisition and interpretation. We discuss our most recent experiments and provide an outlook of our future works.
FGCS2018a
G. J. Nalepa, K. Kutt, and S. Bobek, “Mobile platform for affective context-aware systems,” Futur. Gener. Comput. Syst., vol. 92, pp. 490–503, Mar. 2019.
Abstract In our work, we focus on detection of affective states, their proper identification and interpretation with use of wearable and mobile devices. We propose a data acquisition layer based on wearable devices able to gather physiological data, and we integrate it with mobile context-aware framework. Furthermore, we formulate a method for personalization of emotion detection. This solution offers a non-intrusive measurement thanks to the use of wearable devices, such as wristbands. As means of validation of our concepts we describe a series of experiments that we conducted.
FedCSIS2017
G. J. Nalepa, B. Giżycka, K. Kutt, and J. K. Argasiński, “Affective Design Patterns in Computer Games. Scrollrunner Case Study,” in Communication Papers of the 2017 Federated Conference on Computer Science and Information Systems, 2017, vol. 13, pp. 345–352.
Abstract The emotional state of the user is a new dimension in human-computer interaction, that can be used to improve the user experience. This is the domain of affective computing. In our work we focus on the applications of affective techniques in the design of video games. We assume that a change in the affective condition of a player can be detected based on the monitoring of physiological signals following the James-Lange theory of emotions. We propose the use of game design patterns introduced by Björk and Holopainen to build games. We identify a set of patterns that can be considered affective. Then we demonstrate how these patterns can be used in a design of a scroll-runner game. We address the problem of the calibration of measurements in order to reflect responses of individual users. We also provide results of practical experiments to verify our approach.
AfCAI2016b
G. J. Nalepa, J. K. Argasiński, K. Kutt, P. Węgrzyn, S. Bobek, and M. Z. Łępicki, “Affective Computing Experiments in Virtual Reality with Wearable Sensors. Methodological considerations and preliminary results,” in Proceedings of the Workshop on Affective Computing and Context Awareness in Ambient Intelligence (AfCAI 2016), 2016, vol. 1794.
Abstract In this paper we discuss selected important challenges in designing experiments that lead to data and information collection on affective states of participants. We aim at acquiring data that would be basis to formulate and evaluate computer methods for detection, identification and interpretation of such affective states, and ultimately human emotions.
AfCAI2016a
G. J. Nalepa, K. Kutt, S. Bobek, and M. Z. Łępicki, “AfCAI systems: Affective Computing with Context Awareness for Ambient Intelligence. Research proposal,” in Proceedings of the Workshop on Affective Computing and Context Awareness in Ambient Intelligence (AfCAI 2016), 2016, vol. 1794.
Abstract We are aiming at developing a technology to detect, identify and interpret human emotional states. We believe, that it can be provided based on the integration of context-aware systems and affective computing paradigms. We are planning to identify and characterize affective context data, and provide knowledge-based models to identify and interpret affects based on this data. A working name for this technology is simply AfCAI: Affective Computing with Context Awareness for Ambient Intelligence.