Published on 23.04.19 in Vol 6, No 4 (2019): April
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/13485, first published Jan 24, 2019.
The Potential of Game-Based Digital Biomarkers for Modeling Mental Health
Background: Assessment for mental health is performed by experts using interview techniques, questionnaires, and test batteries and following standardized manuals; however, there would be myriad benefits if behavioral correlates could predict mental health and be used for population screening or prevalence estimations. A variety of digital sources of data (eg, online search data and social media posts) have been previously proposed as candidates for digital biomarkers in the context of mental health. Playing games on computers, gaming consoles, or mobile devices (ie, digital gaming) has become a leading leisure activity of choice and yields rich data from a variety of sources.
Objective: In this paper, we argue that game-based data from commercial off-the-shelf games have the potential to be used as a digital biomarker to assess and model mental health and health decline. Although there is great potential in games developed specifically for mental health assessment (eg, Sea Hero Quest), we focus on data gathered “in-the-wild” from playing commercial off-the-shelf games designed primarily for entertainment.
Methods: We argue that the activity traces left behind by natural interactions with digital games can be modeled using computational approaches for big data. To support our argument, we present an investigation of existing data sources, a categorization of observable traits from game data, and examples of potentially useful game-based digital biomarkers derived from activity traces.
Results: Our investigation reveals different types of data that are generated from play and the sources from which these data can be accessed. Based on these insights, we describe five categories of digital biomarkers that can be derived from game-based data, including behavior, cognitive performance, motor performance, social behavior, and affect. For each type of biomarker, we describe the data type, the game-based sources from which it can be derived, its importance for mental health modeling, and any existing statistical associations with mental health that have been demonstrated in prior work. We end with a discussion on the limitations and potential of data from commercial off-the-shelf games for use as a digital biomarker of mental health.
Conclusions: When people play commercial digital games, they produce significant volumes of high-resolution data that are not only related to play frequency, but also include performance data reflecting low-level cognitive and motor processing; text-based data that are indicative of the affective state; social data that reveal networks of relationships; content choice data that imply preferred genres; and contextual data that divulge where, when, and with whom the players are playing. These data provide a source for digital biomarkers that may indicate mental health. Produced by engaged human behavior, game data have the potential to be leveraged for population screening or prevalence estimations, leading to at-scale, nonintrusive assessment of mental health.
JMIR Ment Health 2019;6(4):e13485
- digital games;
- digital phenotyping;
- mental health;
- computational modeling;
- big data;
- video games;
Playing games on computers, gaming consoles, or mobile devices (ie, playing digital games) has become a leading leisure activity of choice, with consumer spending on digital games exceeding US $134 billion  and outranking spending on music and movie tickets combined [ - ]. There are approximately 200 million gamers in North America, 354 million in Europe, 330 million in the Middle East and Africa, 234 million in Latin America, and 1.2 billion in Asia, which represents between 55% (Latin America) and 67% (North America) of the global online population [ ].
When people play commercial digital games, they produce a lot of data—data that are not only related to play frequency, but also include performance data reflecting low-level cognitive and motor processing; text-based data that are indicative of affective state; social data that reveal networks of relationships; content choice data that imply preferred genres; and contextual data that divulge where, when, and with whom the player is playing. The game-based data produced by players are a rich source of information with the potential to be used for the assessment and modelling of health and health decline. In this paper, we argue that game-based data from commercial off-the-shelf games can be used as a digital biomarker to assess and model mental and cognitive health and health decline.
Assessment for mental health is performed by experts using interview techniques, adjacent to questionnaires and test batteries, and following standardized manuals . However, there is interest in finding behavioral correlates that are predictive of mental and cognitive health and can be used for population screening or prevalence estimations [ ]. When behavioral correlates are already known, researchers can develop custom games that are intended to place a player in a situation and monitor their response, response time, or performance. There are several examples of custom assessment games that have been developed to assess aspects of physical [ ] and mental [ ] health. For example, Sea Hero Quest [ ], an internet game to track and assess dementia through navigational skills, can assess a large number of people very quickly, providing huge data sets with information on health decline over time for demographic groups and individuals.
Although there is great potential in custom games for assessment, we believe that the activity traces left behind by natural interactions with digital games can be used as a digital biomarker of health and health decline. As such, in this paper, we focus on the less-studied topic of data that can be gathered from “in-the-wild” gameplay of commercial off-the-shelf games and how natural gameplay data can yield insights into a person’s mental health.
Furthermore, although there are sources of data from games that may inform physical health or health decline (eg, identifying tremor development in patients with Parkinson disease from mouse kinematic data in a targeting-based game), we focus on the potential of game-based biomarkers in the context of mental and cognitive health. We consider the scope of mental health as defined by the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition , which includes assessment criteria for mental disorders including developmental disorders (eg, autism spectrum disorder), neurodevelopmental disorders (eg, Parkinson disease), anxiety-related disorders, and depressive and personality disorders.
We first present a primer on games, the types of data, and the sources of data that are generated from natural digital gameplay. We then provide a description of five categories of digital biomarkers that can be derived from game-based data, including behavior, cognitive performance, motor performance, social behavior, and affect. We end with a discussion on the ethics, limitations, and potential of data from commercial off-the-shelf games for use as a digital biomarker of mental health.
Primer on Games and Game Data
Digital game play yields a variety of data that can be used to create novel digital biomarkers for health. Commercial games, as well as those created for research purposes, embed logging software that tracks interaction with the game. Game analytics—the tracking, analysis, and visualization of game-related data—is an important tool used by game designers, developers, and studios to inform about player experience [- ]. Used both when games are under development as well as after release, game analytics leverage a variety of data types, the most common of which are player-generated data from interaction with the game [ ].
To understand how data gathered from gameplay can be used as a digital biomarker, we describe a standard machine learning pipeline —starting with observations (eg, game log data), progressing to feature extraction based on the observed signals, and ending with predictions (ie, digital biomarkers)—in the context of gameplay. demonstrates this pipeline in the context of gameplay data and biomarkers for mental health. In the measurement layer, raw signals are gathered from various sources. In the inference layer, features are extracted and computational models are applied in an iterative process [ ]. This computational modeling process results in final predictions, which are shown in the indicator layer. In this section, we briefly describe the measurement and inference layer; the next section focuses on the indicator layer and the five categories of game-based digital biomarkers that we propose.
The first stage of generating digital biomarkers from game data is measurement of the data. Game data are available from a variety of sources, including the player, the player’s interaction with the game, and the player’s interactions with others within and around the game. Although the biggest source is derived from the player interacting with the game, data from the players themselves and from the players’ interactions with others can also provide value by building on research advances in digital biomarkers from nongame sources, for example, within the fields of affective computing  and natural language processing [ ]. However, data from the interaction with the game is a novel source of signal with great potential for inference, but one that requires future research to interpret and understand.
Player data are gathered from the players themselves. Signals such as heart rate, electrical activity of the brain, and galvanic skin response have all been explored in game contexts to infer player experience [, ]. Although physiological data may not be easily accessible at the resolution gathered by medical-grade devices, novel methods of gathering physiological data in situ continue to emerge; sensors are embedded into game controllers for use in biofeedback games [ ], to engage spectators [ ], or simply as an innovative game input [ ]. Further, physiological signals such as heart rate [ ], blink rate, and facial expressions [ ] can be derived from webcam-based signals along with facial expressions.
Player Interaction Data
Data from a player’s interaction with the game is primarily gathered from the game logs and includes high-level data (eg, number of active daily players) and low-level data (eg, each bullet fired in a shooting game, as was done in Gears of War 3 [Microsoft Studios, Redmond, WA; 2011]. Game developers insert logging software that tracks interaction with the game, and common metrics that derive from player interaction data include usage data (eg, login frequency, play time, and role choices), performance data (eg, win/loss data, death rate, kill rate, and score), and social data (eg, teammate records). Researchers can also use low-level input device data to infer information about the player. For example, the motion kinematics (eg, characteristics of the velocity or acceleration of a movement ) of the mouse or thumbstick movements could be used to infer information, or the pressure exerted on a screen in a touch-based game [ ] or on the buttons of a game controller [ ] could provide information on the experience state of the player. Finally, researchers could leverage the audio signals used in many online games to infer information; for example, environmental noise sensed through the microphone and coughing or respiration patterns of the player [ ] could provide a rich source of data.
Data from Players’ Interactions with Others
Data from a player’s interaction with other players, spectators, or fans yield signals that are social in context, but can be used in the development of individual digital biomarkers. For example, text-based data from in-game chats could be analyzed using a keystroke dynamics approach, which leverages the timing and variability in typing rhythms (analyzing both time spent dwelling on keys and moving between keys, known as flight time) of common two-letter combinations (digraphs) and three-letter combinations (trigraphs) to infer the emotional state . Further, sentiment analysis of social media posts for detecting suicidal ideation [ ] could be modified and applied to text data generated in and around games. Text data are available from various sources, including in-game chat logs, game reviews, and texting about games on platforms such as Discord (Discord Inc, San Francisco, CA; 2015). Audio chat between players also generates a valuable signal; building on previous work that determines the emotional tone of the speech signal [ ] might prove useful in the context of mental health modeling.
Gathering Game Data
Game data can be gathered from a single game; the majority of mobile games, as well as console games and computer games, record login and performance data as part of their game analytic engines. Much of these data are accessible to researchers through game application interfaces (APIs), including for the multiplayer online battle arena League of Legends (Riot Games, West Los Angeles, CA; 2009) , which boasts of 100 million active players monthly, or for the massively multiplayer role-playing games EVE online (CCP Games, Reykjavik, Iceland; 2003) [ ] and Guild Wars 2 (ArenaNet, Bellevue, WA; 2012) [ ]. However, in the context of modeling mental health, the aggregate play across game genres and titles may be more relevant.
Although aggregate usage data can be difficult for health researchers to gather (eg, PlayStation account data are not publicly accessible), there are methods by which overall play data can be gathered. Specifically, some game publishers provide an API to their suite of games; for example, battle.net features an API for a collection of games developed by Blizzard Entertainment  that includes the massively multiplayer role-playing game World of Warcraft, the role-playing game Diablo III, and the real-time strategy game Starcraft II, which is a popular title in the competitive electronic sports (e-sports) domain. Steam is the largest online portal for game play on computers (as opposed to dedicated gaming consoles), which features thousands of games and 100 million members; Steam has an API [ ] that gives access to usage data and much more. Further, some data are available through other game portals such as Google Play [ ] or Facebook Game Services [ ], or through APIs that interface with games servers, such as the Sponge API [ ] for the popular construction game Minecraft (Microsoft Studios, 2014). Further game streaming platforms (eg, twitch.tv) have recently been investigated as a potential source of data for research [ ]. APIs generally provide limited access to the vast data available to the game publishers themselves, and collaborating with game companies to access richer datasets would be a valuable approach.
Computational approaches used to make predictions in data science derive inferences from features that are extracted from signals, which are observed in the previously described measurement layer. The quality of predictions derived from machine learning approaches (including, for example, deep learning  and clustering [ ]) depends greatly on the quality of the features that are extracted from the signal. Selecting features and creating new ones [ ] require expertise in the signals, their meaning, and the mathematical and computational approaches that are used in data science. Selecting and creating meaningful features (known as feature engineering [ ]) are challenging and are the point in the pipeline where the expertise of the researcher makes a difference between a black box machine learning model (which blindly applies prediction algorithms to extracted data) and an informed expert-driven model that is built on theory, hypotheses, and iterative testing [ ].
Although feature selection for some types of data is already well established in the literature (eg, sentiment analysis of text excerpts using natural language processing ), there are still open questions on how to best characterize signals derived from novel data sources such as video games. Game-based signal data require specific considerations, because players interact with the games on several levels (eg, explicit interaction with the game and implicit interaction around the game on third-party channels). A full description of the inference layer (or the machine learning approaches that are used within it) is beyond the scope of this paper; however, there are many standard resources to guide researchers who wish to leverage game-based signals to derive digital biomarkers of mental and cognitive health in the process of feature creation, feature selection, and machine learning [ , - ].
Game-Based Digital Biomarkers
The data generated by digital games for player experience understanding can be used to extract features, from which researchers can predict characteristics of a player that may inform models of mental health. This prediction is part of the Inference Layer in. In this section, we describe five categories of digital biomarkers for health that we feel can be extracted from game-based data—behavior, cognitive performance, motor performance, social behavior, and affect. In each section, we describe why this source of data is important for mental health modeling, the game-based sources from which it can be derived, and relevant literature that associates it with mental health.
Behavioral biomarkers refer to metrics that are derived from the act of playing (eg, time spent playing and play frequency), high-level behaviors of players within a game (eg, role choices and play styles), and lower-level behaviors that reflect interaction with the game (eg, gaze fixation and low-level interface interactions).
Dosage as Behavior
In the context of health, frequency is often referred to as dosage and reflects both the overall engagement of the player with the game system and more complex patterns of engagement. For example, researchers may wish to know simply how much time is spent daily playing digital games (ie, quantity), but may also be interested in when gaming sessions occur, how long gaming sessions last, how much variability (ie, predictability) there is in the overall patterns of play—referred to as entropy in research on the computational analysis of human behavior —or the context in which play is occurring (eg, where, with whom, and on which device).
Usage data, in terms of logins and play frequency, are available through in-game logging. There has been some research on the relationship between game play dosage and mental health from the perspective of addiction research. For example, research on pathological gaming has used definitions of addictive behavior to describe problematic or obsessive video game playing ; however, associations between excessive play and psychosocial health have not been consistently evident, even among gamers classified as “addicted” [ ].
Although there may be meaning inferred from the overall dosage of play, there is likely even more richness in the patterns of play. For example, Lemola et al  showed that habitual computer game playing at night (between 10 pm and 6 am) is associated with increased depression scores, even after controlling for the total time played, suggesting that the timing of play could be predictive of mental health (eg, sleep-wake disorders). Further, research suggests that the context of play can moderate the associations of gaming behavior and well-being [ ]. In-depth analysis of the contextualized patterns of play may indicate mental health; by considering variability in the timing, device, or location of play, there is likely predictive potential in behavioral data beyond dosage estimation.
Content and Preference Choices as Behavior
What we play and how we play tell us about who we are. Do we prefer games with complex, visually stunning, narrative heavy, open-world play (eg, Skyrim); games that gain their depth from complex interdependent choices (eg, SimCity); or games built around challenging puzzles (eg, The Witness)? When we play a specific genre, how do we play: Do we prefer to investigate every aspect of an environment or move through in-game quests quickly? When playing with others, do we play a supportive role (eg, healer) or do we prefer to play characters that drive progress (eg, damage dealer)? Do we play multiplayer games at all or do we prefer single-player games? Do we prefer to play games that are easy to progress through or do we choose ultra-hard modes to challenge our own abilities? There are several ways in which we can differentiate between different types of players , and the consideration of individual preferences that are observed through behavior may prove useful for identifying patterns related to mental health.
Research on predominantly male players who display problematic online play behavior (for example, play for long hours, aggressive behavior when forced to stop playing, and financial struggles as a result of play) shows that social anxiety is most prevalent among massively multiplayer role-playing game players and lowest for first-person shooter players , suggesting that genre preference could be an indicator of mental health. Further, self-discrepancy (ie, how well who we are aligns with who we want to be) research has shown that people with low mental well-being ascribe more desirable attributes (eg, kindness and creativeness) to a self-created character than to themselves [ ]. Finally, content or preference choices that deviate from predictable patterns may indicate changes in life circumstances (eg, change in relationships or work status) that could contextualize other observed differences, which may help better predict and assess mental health.
Low-Level Game Interaction as Behavior
Play behavior is the composite of myriad low-level interactions, such as selecting a menu item or checking the health bar . Low-level interactions that create events, such as firing a bullet, can be measured by including the logging code in the game software. Although low-level interactions such as clicking a button to fire a shot carry a signal, there is additional information in the interactions with the system itself (eg, moving the mouse) that precede the final event (ie, shooting). For example, we can learn about the visual cues a person responds to by analyzing eye gaze patterns or investigating mouse movement patterns during inventory search behavior. There is reason to believe that gaze patterns might help indicate mental health; for example, saccadic gaze behavior has been used extensively in the study of schizophrenia and bipolar affective disorder [ - ].
Measuring low-level interactions such as mouse movements or eye gaze requires either special equipment (ie, eye tracker) or needs to be implemented during development (ie, cursor logging software). Assessing low-level interactions during play of off-the-shelf games is challenging, because the game code cannot be modified to measure input (eg, click behavior), and bringing third-party equipment (eg, gaze tracking) to the user is logistically challenging and would influence play behavior. To overcome these challenges, researchers must determine ways to measure the signals unobtrusively. For example, research on older adults has leveraged image processing to investigate measuring cognitive performance through the card game Klondike Solitaire . Although measuring low-level interactions is challenging outside of the laboratory context, there are many open questions about which player characteristics can be inferred. Does frequent non-goal–directed click behavior indicate nervousness? Can looking up the same in-game hint (eg, a password on a note in the inventory) multiple times indicate forgetfulness? Are quick changes in gaze fixation indicative of cognitive performance and information processing? The behavior that we show unconsciously might be hardest to access, but because implicit behaviors are difficult for players to influence, they have huge potential for inferring mental health.
Cognitive Performance Biomarkers
Many games incorporate cognitive challenges such as memorizing sequences that rely on short-term memory (eg, Shadowrun), making decisions under certain conditions (eg, first-person shooter games), recognizing patterns (eg, Bejeweled), or analyzing complex information (eg, SimCity). Game difficulty can generally be adjusted (or adjusted dynamically), so that the cognitive performance required to succeed is well matched to the player’s abilities . When players tackle cognitive challenges in games, they generate performance data such as the number of attempts, time spent to overcome a challenge, reaction times to stimuli, or the highest level they were able to complete without failure. There are a variety of game sources from which one can identify and measure cognitive performance in games. Leaderboards and achievements are aggregated indicators of game performance [ ]: Leaderboards provide a high score in comparison with other players or personal bests, whereas achievements represent specific challenges a player has overcome, such as a challenging attack sequence, collecting a specific set of items, or taking out a certain number of enemies. Other performance indicators are ranks or titles of a player (eg, Platinum or Diamond rank in League of Legends), the rarity of owned items (eg, Stunted Direhorn in World of Warcraft), or a player’s player-vs-player statistic; however, these performance indicators may be more related to time investment than to the underlying cognitive ability.
To access cognitive performance biomarkers through games, we can look to specific game mechanics that map well to cognitive abilities. For example, the number of dodged hits in a fighting game is indicative of a player’s ability to anticipate and react, a difficult skill prevalent in elite athletes  and reduced in people with depression [ ]. The turn-based role-playing game Shadowrun: Hong Kong (Harebrained Schemes, Seattle, WA; 2015) features a two-staged hacking challenge that requires players first to repeat an increasingly difficult sequence of numbers—a common task for assessing short-term memory [ ]. The puzzle game The Witness (Johnathan Blow, 2016) presents challenges that require spatial rotation and abstract thinking (eg, one puzzle requires players to first identify the path through a three-dimensional maze and to then redraw the pathway on a two-dimensional input device).
Cognitive performance is an indicator of many mental health issues such as depression, anxiety, Alzheimer disease, Parkinson disease, or attention-deficit disorders . People experiencing symptoms of depression, for example, show differences in executive functioning, sustained attention, and memory [ , ], which debilitates a player’s ability to play games that rely on these cognitive systems. Digital assessment software such as Cambridge Neuropsychological Test Automated Battery [ ] have shown that many of the resulting divergences in cognitive performance related to mental health can be detected through simple tasks that are likely mirrored in digital games. Further, performance in games can be used to naturally assess cognitive abilities such as attentional [ ] or visuospatial [ ] biases.
Motor Performance Biomarkers
Most games require motor input to interact with them. The input device used in games varies, but is generally comprised of touch input (ie, in mobile games); mouse and keystroke input (ie, in desktop games); and controller input, which consists of buttons to press and thumbsticks or mini joysticks to control (ie, in console games). Gaming consoles additionally sometimes have cameras that capture the user’s movements (eg, Microsoft Kinect and Sony PlayStation Camera). The level of motor interaction needed to play a game varies widely: Many games require complex sequences of input (eg, Street Fighter), whereas others take a very simple motor action, but require it to be completed quickly and repeatedly (eg, clicking in Cookie Clicker) or in combination with cognitive choices (eg, keystrokes in Starcraft).
Motor data in games can be gathered through the types of sources listed in the Measurement Layer section; however, researchers interested in millisecond accuracy of motor input can also write third-party logging software that captures interaction with the device . Leveraging research in human-computer interaction, the kinematics of mouse movements (eg, velocity, acceleration, and percentage of time spent decelerating [ ]) or the variability in mouse kinematics could be indicative of mental or cognitive health. Further, keystroke dynamics (ie, the low-level timing of typing actions) have been used to predict the stress of the typist [ , ], which could potentially inform stress-related mental health disorders. Game console cameras could be used to detect players’ movements, which has been associated with various emotional states [ ] that might be indicative of mental health or health decline. In addition, the pressure exerted on gaming controls has been linked to frustration for both button presses [ ] and touch input [ ], which provides relevance in terms of players’ resilience to stressful stimuli. Leveraging an entire touch gesture can also provide interesting signals; for example, in Fruit Ninja gestures, timing and pressure features were used to discriminate low and high arousal and valence, both of which are relevant in the context of mental health [ ].
Motor performance data have particular potential as a digital biomarker for neurodegenerative diseases such as Alzheimer, Parkinson, and Huntington diseases, which are characterized, in part, by psychomotor decline ; monitoring patients’ motor performance in games over time could reveal valuable information about the rate of disease progression.
Social rather than individual play is quickly becoming the dominant form of digital game play: Gamers spend an average of 6 hours/week playing with others online and 5 hours/week playing with others in person . When people play digital games with others, they generate social behavioral data (for example, with whom are they playing and what role did they take in that interaction) as well as social communication data (for example, what words were exchanged and in which channels).
In terms of social behavior, some multiplayer game APIs (eg, League of Legends) provide information on who people were playing with; more subtle cues of social interaction, such as being part of a guild (eg, in World of Warcraft), the number of social contacts as compared to population averages, choices of predefined texts (eg, “good game”) sent (eg, Hearthstone), or the ratio of the type of games played (eg, single-player games vs team-based games) could be indicative of mental health. Communication-based game data can be drawn from primary sources such as in-game chats, forum posts, or voice-over-internet protocol applications (eg, Discord or Team Speak) and secondary channels such as social media (eg, Reddit, Twitter, and Facebook) or video commentary (eg, Twitch and YouTube). Natural language processing approaches allow researchers to identify word categories used and gives insights into the way we use language, for example, the ratio of self-references to social references in written texts . Further, sentiment analysis allows us to gauge the valence of an attitude (ie, positive, neutral, or negative) toward a specific topic. Applied to written text or spoken words from game-based sources, these techniques provide insights into how users present themselves when interacting with others.
Social behavior is an important indicator in many mental health disorders and therefore important for health modelling. People suffering from depression, for example, show higher use of negatively valenced phrasing , a tendency to use more self-referencing words such as “I” or “me” [ , ], and use of fewer emoticons [ ]. It also matters with whom they have contact; research on game play suggests that playing socially may be linked to well-being [ - ]. In game play, this might be reflected in the consistency with which we play with other specific players.
Although online communication has been found to negatively affect one’s well-being despite its social nature , the issue is more complex and requires nuanced consideration of the usage purpose, context, and individual differences [ , ]. A large proportion of research on digital biomarkers to predict mental well-being has focused on depression and anxiety; however, psychiatric research [ ] suggests that the range of mental illnesses reflected in digital traces also applies to mental illnesses such as borderline disorder or bipolar disorder. Bipolar disorders might, for example, be reflected in increased parallel conversations, high levels of text output, and fluctuation of social relationships during up swings, while down swings would be characterized by disengagement and negatively valenced communication patterns.
Affective and Emotional Biomarkers
Quantifying the emotional state of the player has been of interest to affective computing researchers and game designers who wish to understand or improve player experience . Researchers detect affect using physiological signals [ ], behavioral signals (eg, posture [ ]), speech signals [ ], eye gaze and fixation data [ ], and sentiment analysis of text data [ ].
Within game play, sources of measurement for affective biomarkers include in-game chat (text data), in-game audio for communication during online play, audio during streaming (eg, on twitch.tv), and the input data discussed in relation to motor performance biomarkers. In the game, there are also game-based text data that can be gathered from forums and chat platforms (eg, Discord) as well as game-related posts on social media. Physiological data are not necessarily straightforward to gather in the context of commercial off-the-shelf computer game play; however, recent advances in sensing over a distance, such as heart rate  or facial expression [ ], gathered via webcam and increased prevalence of physiological input devices in commercial games [ ] suggest that third-party logging could be used to link physiological sources with game APIs.
Understanding the emotional experience of players is of particular interest in the context of mental health, as many prevalent mental health issues (eg, depression and generalized anxiety) are closely tied to emotional wellness or can be predicted by aberrant responses to specific stimuli (eg, posttraumatic stress disorder) or in-game self-representations (eg, anorexia nervosa). For example, eye gaze patterns have been used to characterize individuals with various depression and anxiety disorders . Impulse control and conduct disorders are characterized by difficulty in the self-regulation of emotions and behaviors [ ], whereas emotional self-regulation difficulties are also characteristic in some developmental disorders (eg, autism spectrum disorder [ ]).
Digital biomarkers are increasingly being used to indicate potential mental health issues [- ]. Computational phenotyping—the digital quantification of disease phenotypes—extracts the observable traits (eg, morphology, development, and behavior) of an entity from data sources that can be complex and heterogenous. In this paper, we propose using data generated from natural play of off-the-shelf digital games (ie, digital biomarkers) as one such complex data source. Five categories of game-based digital biomarkers—behavior, cognitive performance, motor performance, social behavior, and affect—were argued to include observable traits that could be indicative of mental health or health decline.
Multiple Data Sources and Sensor Fusion
Each of the five described digital biomarkers mentioned has the potential to indicate mental health or illness; however, it may be in their combination that the true power of game data as a digital biomarker of mental health can be seen. Further, combining game data with other observable traits derived from smartphone data (eg, geolocation, accelerometer, and Bluetooth devices), sentiment from social media (eg, Google search, Reddit, Twitter, and Facebook), or physiology from integrated trackers (eg, Fitbit) may allow for rich and predictive models of mental health that leverage sensor fusion (ie, the use of multiple sources of data in combination ) for accurate modeling. We do not suggest that game-based data can be the sole biomarker for mental health in isolation, but that considering its inclusion in a suite of behavioral indicators may improve modeling in the context of mental and cognitive health.
To interpret data generated by different sources, data collected from a large sample with the intention to create a norm has several advantages. Sea Hero Quest , for example, generates insights into humans’ general capability to navigate and then identify behavior that deviates from norms as an indicator for early onset dementia. Establishing norms in game-based biomarkers is a complex undertaking but may have value when looking for deviations from norms as an indicator of mental health decline.
There are various factors that may compromise the predictive potential of game-based digital biomarkers. For example, pharmaceutical treatment or remission through therapy can interfere with accurate prediction: In terms of impulsivity, depressed people in remission are often grouped with control subjects rather than depressed participants , whereas in terms of visual acuity, depressed participants, regardless of the pharmaceutical treatment, are differentiated from the control group, but not each other [ ]. As use of game-based biomarkers is a novel approach, researchers must carefully establish whether and how treatment or remission affects the behavior underlying the biomarker. Further, as with all digital biomarkers derived from in-the-wild data sets, events unrelated to the characteristic being predicted can greatly interfere; for example, a player who is on holiday, has a cold, has an upcoming deadline, or is experiencing harsh weather can exhibit behaviors that appear to be erratic, but which are driven by circumstance and not mental health.
Another limitation is that game-based biomarkers will only be relevant for assessment of populations who play games. People of all ages play games; however, preferences for various genres and platforms change with age . A greater proportion of people of color in America play games, identify as gamers [ ], own a gaming system, and form a faster-growing market than their white non-Hispanic counterparts [ ]. In addition, almost half of all gamers self-identify as female; however, social gaming and mobile gaming are more important to female gamers than male gamers [ ]. Demographic factors will need to be accounted for in any model of mental health built on game-based biomarkers. Finally, research on game-based biomarkers is in its early infancy: In this paper, we hypothesize that the potential biomarkers are based on existing scientific literature; however, significant research is needed to demonstrate the efficacy of game-based biomarkers and to identify specific game-based biomarkers for specific groups of mental health disorders.
Ethics, Privacy, and Legal Use of Game Data
The use of data derived from digital sources is part of a larger discussion [- ]; as with all digital data, game-based biomarkers require consideration for topics such as inferring identity, communicating mental health assessment, privacy of in-game conversations, and legality of gathering data unobtrusively.
Rare events in game-based data (eg, a difficult-to-unlock achievement) allows for the identification of individual players. As a result, researchers need to consider information reported on their players. When working with public players such as e-sports athletes or streamers, the in-game identity can be publicly linked to the actual identity of a player, which therefore requires considerations for privacy.
Technologies such as Mindstrong Health Services  or Facebook’s Suicide detection [ ] show how data gathered from digital sources can be used to infer mental health and reach out to individuals who are at risk. Game-based biomarkers could result in similar services. However, communicating detected potential mental health issues might have negative consequences for users; for example, it is unclear what the consequences are of false positives (ie, a player being urged to seek help when there is actually no issue at all). Researchers need to consider the potential consequences of reporting mental health to players and the methods by which they communicate with players about the detected mental health issues.
Forums, in-game chats, and game streams require extra attention when analyzed, because users do not intend to have their information stored and may use phrasing and communication patterns that misrepresent conversations out of context. Consent to allow researchers to monitor conversations needs to be given explicitly, and player-initiated removal of unwanted data from further storage (eg, the use of a derogative term) requires consideration.
In addition to standard ethical considerations of data-driven inference, game-based data have characteristics that require special attention. First, the expectation that mental health could be inferred from a player’s pressure profile of a button press during game play is lower than that from explicitly posted texts about suicidal ideation on social media platforms [, ]. Players should be clearly notified of the implications of implicit assessment from natural in-game behaviors that are, in the player’s view, unrelated to mental health. Further, extensive play for the purposes of assessment should not be a “slippery slope” into pathological play behaviors. Second, players often behave differently in games by enacting fantasies [ ], trying out different personalities [ ], or behaving in ways congruent with gameplay, which are incongruent with out-of-game expectations, such as acting violently or ultra competitively. Applying value judgements from models of behavior generated out of games may not apply within games and could increase the risks of false positives if not properly considered.
Mental illness has become a major disease burden globally: Depression is currently the leading cause of disability worldwide . Untreated mental illness has serious consequences; the estimated US $2.5-8.5 trillion globally in lost output attributed to mental, neurological, and substance use disorders is expected to almost double by 2030 [ ]. In addition to these financial costs, people experience costs to their well-being that range from a lower quality of life [ ] to a loss of life [ ]. The emergence of smartphone and wearable devices has begun to show promise for the assessment of mental health, for example, for real-time assessment of suicidal thoughts [ ] or acute phases of psychosis among people with schizophrenia [ ]; however, there is additional work to be done before researchers can reliably use game-based biomarkers to predict a decline in a person’s mental health, such as the onset of a depressive episode, the progression of dementia, or behavioral changes that are related to social anxiety.
In this paper, we argued that owing to the prevalence of digital game play, there are several untapped sources of data, including behavior, cognitive performance, motor performance, social behavior, and affect. Further, we proposed that due to existing statistical associations between these five game-based digital biomarkers and mental health, there is untapped potential in game data for computational modeling that predicts mental health and mental health decline.
We thank the Natural Sciences and Engineering Research Council of Canada, the EWR Steacie Memorial Fellowship, and the Saskatchewan-Waterloo Gamers User Research (SWaGUR) training program for the funding, and the Interaction Lab at the University of Saskatchewan and the Eindhoven University of Technology for support.
Conflicts of Interest
- Newzoo. 2018. 2018 Global Games Market Report URL: https://resources.newzoo.com/hubfs/Reports/Newzoo_2018_Global_Games_Market_Report_Light.pdf [accessed 2019-03-18] [WebCite Cache]
- IFPI. 2017. Global music report 2017: Annual State of the Industry URL: https://www.ifpi.org/downloads/GMR2017.pdf [accessed 2019-03-18] [WebCite Cache]
- Motion Picture Association of America. 2017. 2016 Theatrical Market Statistics Report URL: https://www.mpaa.org/wp-content/uploads/2017/03/MPAA-Theatrical-Market-Statistics-2016_Final-1.pdf [accessed 2019-03-18] [WebCite Cache]
- Newzoo. 2018. Mobile Revenues Account for More Than 50% of the Global Games Market as It Reaches $137.9 Billion in 2018 URL: https://newzoo.com/insights/articles/global-games-market-reaches-137-9-billion-in-2018-mobile-games-take-half/ [accessed 2019-03-18] [WebCite Cache]
- American Psychiatric Association. Diagnostic and statistical manual of mental disorders. 5th ed. Arlington, VA: American Psychiatric Association Publishing; 2013.
- Gillan CM, Daw ND. Taking psychiatry research online. Neuron 2016 Dec 06;91(1):19-23 [FREE Full text] [CrossRef] [Medline]
- Vanden Abeele V, Wouters J, Ghesquière P, Goeleven A, Geurts L. Game-based Assessment of Psycho-acoustic Thresholds: Not All Games Are Equal!. New York, USA: ACM Press; 2015 Presented at: Annual Symposium of Computer-Human Interaction in Play - CHI Play 2015; October 2015; London, England p. 331-341. [CrossRef]
- Gielis K, Brito F, Tournoy J, Vanden Abeele V. Can Card Games Be Used to Assess Mild Cognitive Impairment?: A Study of Klondike Solitaire and Cognitive Functions. New York, USA: ACM Press; 2017 Presented at: Extended Abstracts of the Annual Symposium of Computer Human Interaction in Play - CHI Play ’17; October 2017; Amsterdam, Netherlands p. 269-276. [CrossRef]
- Sea Hero Quest. URL: http://www.seaheroquest.com/site/en/ [accessed 2019-03-18] [WebCite Cache]
- Medler B, John M, Lane J. Data cracker: Developing a visual game analytic tool for analyzing online gameplay. New York, USA: ACM Press; 2011 Presented at: Annual Conference on Human Factors in Computing Systems- CHI 2011; May 2011; Vancouver, Canada p. 2365-2374. [CrossRef]
- Nacke L, Drachen A. Towards a Framework of Player Experience Research. 2011 Presented at: Second international workshop on evaluating player experience in games at FDG; June 2011; Bordeaux, France.
- Wallner G, Kriglstein S. Visualization-based analysis of gameplay data – A review of literature. Entertainment Computing 2013 Aug;4(3):143-155. [CrossRef]
- Drachen A, Canossa A, Seif El-Nasr M. Gamasutra. 2013. Intro to User Analytics URL: https://www.gamasutra.com/view/feature/193241/intro_to_user_analytics.php [accessed 2019-03-18] [WebCite Cache]
- O'Neil C, Schutt R. Doing Data Science: Straight Talk From The Frontline. Sebastopol, CA: O'Reilly Media; 2013.
- Picard R. Affective Computing. Cambridge, MA: The MIT Press; 2000.
- Indurkhya N, Damerau FJ, editors. Handbook Of Natural Language Processing (Machine Learning & Pattern Recognition). Boca Raton, FL: Chapman And Hall/CRC; 2010.
- Mandryk R. Physiological Measures for Game Evalution. In: Isbister K, Schaffer N, editors. Game usability: advancing the player experience. Boca Raton, FL: CRC Press; 2008.
- Mandryk R, Nacke L. Biometrics in gaming and entertainment technologies. In: Mitra S, Gofman M, editors. Biometrics In A Data Driven World: Trends, Technologies, And Challenges. Boca Raton, FL: Chapman And Hall/CRC; 2016.
- Mandryk R, Dielschneider S, Kalyn M, Bertram C, Gaetz M, Doucette A, et al. Games as neurofeedback training for children with FASD. New York, USA: ACM Press; 2013 Presented at: 12th International Conference on Designing for Children- IDC 2013; June, 2013; New York, USA p. 165-172. [CrossRef]
- Robinson R, Rubin Z, Segura E, Isbister K. All the feels: designing a tool that reveals streamers' biometrics to spectators. New York, USA: ACM Press; 2017 Presented at: Foundations of Digital Games - FDG 2018; August, 2017; Cape Cod, MA. [CrossRef]
- Nacke L, Kalyn M, Lough C, Mandryk R. Biofeedback game design: using direct and indirect physiological control to enhance game interaction. New York, USA: ACM Press; 2011 Presented at: Annual Conference on Human Factors in Computing Systems - CHI 2011; May 2011; Vancouver, Canada p. 103-112. [CrossRef]
- Poh M, McDuff DJ, Picard RW. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt Express 2010 May 10;18(10):10762-10774. [CrossRef] [Medline]
- McDuff D, Mahmoud A, Mavadati M, Amr M, Turcot J, Kaliouby R. AFFDEX SDK: a cross-platform real-time multi-face expression recognition toolkit. New York, USA: ACM Press; 2016 Presented at: Extended Abstracts of the Annual Conference on Human Factors in Computing Systems - CHI 2016; May, 2016; San Jose, USA p. 3723-3726. [CrossRef]
- Walker N, Meyer DE, Smelcer JB. Spatial and temporal characteristics of rapid cursor-positioning movements with electromechanical mice in human-computer interaction. Hum Factors 1993 Sep;35(3):431-458. [CrossRef] [Medline]
- Miller M, Mandryk R. Differentiating in-Game Frustration from at-Game Frustration using Touch Pressure. New York, USA: ACM Press; 2016 Presented at: ACM Interact Surfaces Spaces - ISS ’16; November, 2016; Niagara Falls, Canada p. 225-234. [CrossRef]
- Sykes J, Brown S. Measuring emotion through the gamepad. New York, USA: ACM Press; 2003 Presented at: Annual Conference on Human Factors in Computing Systems - CHI 2003; April 2003; Fort Lauderdale, USA p. 732-733. [CrossRef]
- Mandryk R, Inkpen K, Calvert T. Using psychophysiological techniques to measure user experience with entertainment technologies. Behaviour and Information Technology 2006;25(2):141-158. [Medline]
- Epp C, Lippold M, Mandryk R. Identifying emotional states using keystroke dynamics. New York, USA: ACM Press; 2011 Presented at: Annual Conference on Human Factors in Computing Systems - CHI 2011; May 2011; Vancouver, Canada p. 715-724. [CrossRef]
- De Choudhury M, Kiciman E, Dredze M, Coppersmith G, Kumar M. Discovering Shifts to Suicidal Ideation from Mental Health Content in Social Media. 2016 May Presented at: Annual Conference on Human Factors in Computing Systems - CHI 2016; May 2016; San Jose, USA p. 2098-2110 URL: http://europepmc.org/abstract/MED/29082385 [CrossRef]
- Bachorowski J, Owren M. Vocal expressions of emotion. In: Lewis M, Haviland-Jones JM, Barrett L, editors. Handbook of Emotions. New York, USA: The Guilford Press; 2008:196-210.
- Riot Games. URL: https://developer.riotgames.com/ [accessed 2019-03-18] [WebCite Cache]
- Eve Developers. URL: https://developers.eveonline.com/ [accessed 2019-03-18] [WebCite Cache]
- Guildwars Official Wiki. URL: https://wiki.guildwars2.com/wiki/API:Main [accessed 2019-03-18] [WebCite Cache]
- Blizzard. URL: https://develop.battle.net/ [accessed 2019-03-18] [WebCite Cache]
- Steam Web API Documentation. URL: https://steamcommunity.com/dev [accessed 2019-03-18] [WebCite Cache]
- Make Games with Google. URL: https://developers.google.com/games/ [accessed 2019-03-18] [WebCite Cache]
- Facebook for Game Developers. URL: https://www.facebook.com/unsupportedbrowser [accessed 2019-03-18] [WebCite Cache]
- Sponge. URL: https://www.spongepowered.org/ [accessed 2019-03-18] [WebCite Cache]
- Roebuck AJ, Dubnyk AJB, Cochran D, Mandryk RL, Howland JG, Harms V. Competitive action video game players display rightward error bias during on-line video game play. Laterality 2018 Sep;23(5):505-516. [CrossRef] [Medline]
- LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015 May 28;521(7553):436-444. [CrossRef] [Medline]
- Zhang M, Zhou Z. ML-KNN: A lazy learning approach to multi-label learning. Pattern Recognition 2007;40(7):2038-2048. [CrossRef]
- Murphy KP. Machine Learning: A Probabilistic Perspective. Cambridge, MA: The MIT Press; 2012.
- Segaran T. Programming Collective Intelligence: Building Smart Web 2.0 Applications. Sebastopol, CA: O'Reilly Media; 2007.
- Bishop C. Pattern recognition and machine learning. New York, USA: Springer-Verlag; 2006.
- Lemmens JS, Valkenburg PM, Peter J. Development and Validation of a Game Addiction Scale for Adolescents. Media Psychology 2009;12(1):77-95. [CrossRef]
- Van Rooij AJ, Schoenmakers T, Vermulst A, Van den Eijnden RJJM, Van de Mheen D. Online video game addiction: identification of addicted adolescent gamers. Addiction 2011;106(1):205-212. [CrossRef] [Medline]
- Lemola S, Brand S, Vogler N, Perkinson-Gloor N, Allemand M, Grob A. Habitual computer game playing at night is related to depressive symptoms. Personality and Individual Differences 2011 Jul;51(2):117-122. [CrossRef]
- Colder Carras M, Van Rooij AJ, Van de Mheen D, Musci R, Xue Q, Mendelson T. Video gaming in a hyperconnected world: A cross-sectional study of heavy gaming, problematic gaming symptoms, and online socializing in adolescents. Computers in Human Behavior 2017;68:472-479. [CrossRef]
- De Grove F, Cauberghe V, Van Looy J. Development and Validation of an Instrument for Measuring Individual Motives for Playing Digital Games. Media Psychology 2014;19(1):101-125. [CrossRef]
- Park J, Han D, Kim B, Cheong J, Lee YS. Correlations among Social Anxiety, Self-Esteem, Impulsivity, and Game Genre in Patients with Problematic Online Game Playing. Psychiatry Investigation 2016;13(3):297-304 [FREE Full text] [CrossRef] [Medline]
- Bessière K, Seay A, Kiesler S. The ideal elf: identity exploration in World of Warcraft. Cyberpsychology & Behavavior 2007 Aug;10(4):530-535. [CrossRef] [Medline]
- Wuertz J, Birk M, Bateman S. Healthy Lies: The Effects of Misrepresenting Player Health Data on Experience, Behavior, and Performance. New York, USA: ACM Press; 2019 Presented at: Annual Conference on Human Factors in Computing Systems- CHI 2019; May 2019; Glasgow, UK p. 12. [CrossRef]
- Hutton SB, Ettinger U. The antisaccade task as a research tool in psychopathology: a critical review. Psychophysiology 2006 May;43(3):302-313. [CrossRef] [Medline]
- Calkins ME, Iacono WG, Ones DS. Eye movement dysfunction in first-degree relatives of patients with schizophrenia: a meta-analytic evaluation of candidate endophenotypes. Brain Cogn 2008 Dec;68(3):436-461 [FREE Full text] [CrossRef] [Medline]
- Myles JB, Rossell SL, Phillipou A, Thomas E, Gurvich C. Insights to the schizophrenia continuum: A systematic review of saccadic eye movements in schizotypy and biological relatives of schizophrenia patients. Neurosci Biobehav Rev 2017 Jan;72:278-300. [CrossRef] [Medline]
- Smeddinck J, Mandryk R, Birk M, Gerling K, Barsilowski D, Malaka R. How to present game difficulty choices? Exploring the impact on player experience. New York, USA: ACM Press; 2016 Presented at: Annual Conference on Human Factors in Computing Systems - CHI 2016; May 2016; San Jose, USA p. 5595-5607. [CrossRef]
- Bowey J, Birk MV, Mandryk RL. Manipulating Leaderboards to Induce Player Experience. New York, USA: ACM Press; 2015 Presented at: Annual Symposium on Computer Human Interaction in Play - CHI Play; 2015; London, UK p. 115-120. [CrossRef]
- Aglioti SM, Cesari P, Romani M, Urgesi C. Action anticipation and motor resonance in elite basketball players. Nature Neuroscience 2008 Aug 10;11(9):1109-1116. [CrossRef]
- Winograd-Gurvich C, Georgiou-Karistianis N, Fitzgerald P, Millist L, White O. Ocular motor differences between melancholic and non-melancholic depression. J Affect Disord 2006 Jul;93(1-3):193-203. [CrossRef] [Medline]
- Baddeley A. The episodic buffer: a new component of working memory? Trends Cogn Sci 2000 Nov 01;4(11):417-423. [Medline]
- Rock PL, Roiser JP, Riedel WJ, Blackwell AD. Cognitive impairment in depression: a systematic review and meta-analysis. Psychological Medicine 2014 Jul;44(10):2029-2040. [CrossRef] [Medline]
- Ahern E, Semkovska M. Cognitive functioning in the first-episode of major depressive disorder: A systematic review and meta-analysis. Neuropsychology 2017 Dec;31(1):52-72. [CrossRef] [Medline]
- Cambridge Cognition. URL: https://www.cambridgecognition.com/ [accessed 2019-03-18] [WebCite Cache]
- Dennis TA, O'Toole L. Mental Health on the Go: Effects of a Gamified Attention Bias Modification Mobile Application in Trait Anxious Adults. Clin Psychol Sci 2014 Sep 01;2(5):576-590 [FREE Full text] [CrossRef] [Medline]
- Vizer LM, Zhou L, Sears A. Automated stress detection using keystroke and linguistic features: An exploratory study. International Journal of Human-Computer Studies 2009 Oct;67(10):870-886. [CrossRef]
- Kleinsmith A, Bianchi-Berthouze N. Affective Body Expression Perception and Recognition: A Survey. IEEE Trans Affective Comput 2013 Jan;4(1):15-33. [CrossRef]
- Gao Y, Bianchi-Berthouze N, Meng H. What Does Touch Tell Us about Emotions in Touchscreen-Based Gameplay? ACM Trans Comput Hum Interact 2012 Dec 01;19(4):1-30. [CrossRef]
- 2017 Essential Facts About the Computer and Video Game Industry.: Entertainment Software Association; 2017. URL: http://www.theesa.com/wp-content/uploads/2017/09/EF2017_Design_FinalDigital.pdf [accessed 2019-03-18] [WebCite Cache]
- De Choudhury M, Gamon M, Counts S, Horvitz E. Predicting Depression via Social Media. : AAAI Press; 2013 Presented at: Seventh international AAAI conference on weblogs and social media; June 2013; Bellevue, WA, USA p. 128.
- Rude S, Gortner E, Pennebaker J. Language use of depressed and depression-vulnerable college students. Cognition & Emotion 2004 Dec;18(8):1121-1133. [CrossRef]
- Sloan DM. It’s All About Me: Self-Focused Attention and Depressed Mood. Cogn Ther Res 2005 Jun;29(3):279-288. [CrossRef]
- Wang X, Zhang C, Ji Y, Sun L, Wu L, Bao Z. A Depression Detection Model Based on Sentiment Analysis in Micro-blog Social Network. New York, USA: Springer-Verlag; 2013 Presented at: Pacific-Asia Conference on Knowledge Discovery and Data Mining; 2013; Gold Coast, QLD, Australia p. 201-213. [CrossRef]
- Trepte S, Reinecke L, Juechems K. The social side of gaming: How playing online computer games creates online and offline social support. Computers in Human Behavior 2012 May;28(3):832-839 [FREE Full text] [CrossRef]
- Snodgrass JG, Bagwell A, Patry JM, Dengah HF, Smarr-Foster C, Van Oostenburg M, et al. The partial truths of compensatory and poor-get-poorer internet use theories: More highly involved videogame players experience greater psychosocial benefits. Computers in Human Behavior 2018 Jan;78:10-25. [CrossRef]
- Depping A, Johanson C, Mandryk R. Designing for Friendship: Modeling Properties of Play, In-Game Social Capital, and Psychological Well-being. New York, USA: ACM Press; 2018 Presented at: Annual Symposium on Computer-Human Interaction in Play- CHI Play 2018; October 2018; Melbourne, AU p. 87-100. [CrossRef]
- Kraut R, Kiesler S, Boneva B, Cummings J, Helgeson V, Crawford A. Internet Paradox Revisited. J Social Isssues 2002 Jan;58(1):49-74. [CrossRef]
- Bessière K, Kiesler S, Kraut R, Boneva BS. Effects of internet use and social resources on changes in depression. Information, Communication & Society 2008 Feb;11(1):47-70. [CrossRef]
- Shen C, Williams D. Unpacking Time Online: Connecting Internet and Massively Multiplayer Online Game Use With Psychosocial Well-Being. Communication Research 2010 Nov 30;38(1):123-149. [CrossRef]
- Mandryk R, Atkins M, Inkpen K. A continuous and objective evaluation of emotional experience with interactive play environments. New York, USA: ACM Press; 2006 Presented at: Annual Conference on Human Factors in Computing Systems - CHI 2006; April 2006; Montreal, Canada p. 1027-1036. [CrossRef]
- Mandryk RL, Atkins MS. A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. International Journal of Human-Computer Studies 2007 Apr;65(4):329-347. [CrossRef]
- Kleinsmith A, Bianchi-Berthouze N. Recognizing Affective Dimensions from Body Posture. 2007 Presented at: Affect Comput Intell Interact; September 2007; Lisbon, Portugal p. 48-58. [CrossRef]
- O'Dwyer J, Flynn R, Murray N. Continuous affect prediction using eye gaze. : IEEE; 2017 Presented at: 28th Irish Signals and Systems Conference (ISSC); 2017; Killarney, Ireland p. 2001-2007. [CrossRef]
- Poh M, McDuff DJ, Picard RW. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt Express 2010 May 10;18(10):10762-10774. [CrossRef] [Medline]
- De Silva LC, Hui SC. Real-time facial feature extraction and emotion recognition. : IEEE; 2003 Presented at: Fourth International Conference on Information, Communications and Signal Processing and the Fourth Pacific Rim Conference on Multimedia; December 2003; Singapore p. 1310-1314. [CrossRef]
- Armstrong T, Olatunji BO. Eye tracking of attention in the affective disorders: a meta-analytic review and synthesis. Clin Psychol Rev 2012 Dec;32(8):704-723 [FREE Full text] [CrossRef] [Medline]
- Insel TR. Digital Phenotyping: Technology for a New Science of Behavior. JAMA 2017 Oct 03;318(13):1215-1216. [CrossRef] [Medline]
- Manikonda L, De Choudhury M. Modeling and Understanding Visual Attributes of Mental Health Disclosures in Social Media. New York, USA: ACM Press; 2017 Presented at: Annual Conference on Human Factors in Computing Systems- CHI 2017; May 2017; Denver, USA p. 170-181. [CrossRef]
- Patrick K, Hekler EB, Estrin D, Mohr DC, Riper H, Crane D, et al. The Pace of Technologic Change: Implications for Digital Health Behavior Intervention Research. Am J Prev Med 2016 Dec;51(5):816-824. [CrossRef] [Medline]
- Glenn T, Monteith S. New measures of mental state and behavior based on data collected from sensors, smartphones, and the Internet. Curr Psychiatry Rep 2014 Dec;16(12):523. [CrossRef] [Medline]
- Coutrot A, Schmidt S, Coutrot L, Pittman J, Hong L, Wiener J, et al. Virtual navigation tested on a mobile app is predictive of real-world wayfinding navigation performance. PLoS ONE 2019 Mar 18;14(3):e0213272. [CrossRef]
- Maalouf FT, Klein C, Clark L, Sahakian BJ, Labarbara EJ, Versace A, et al. Impaired sustained attention and executive dysfunction: bipolar disorder versus depression-specific markers of affective disorders. Neuropsychologia 2010 May;48(6):1862-1868 [FREE Full text] [CrossRef] [Medline]
- Bubl E, Kern E, Ebert D, Bach M, Tebartz van Elst L. Seeing gray when feeling blue? Depression can be measured in the eye of the diseased. Biol Psychiatry 2010 Jul 15;68(2):205-208. [CrossRef] [Medline]
- Birk M, Friehs M, Mandryk RL. Age-based preferences and player experience: A crowdsourced cross-sectional study. New York, USA: ACM Press; 2017 Presented at: Annual Symposium on Computer-Human Interaction in Play - CHI PLAY 2017; 2017; Amsterdam, Netherlands p. 157-170. [CrossRef]
- Duggan M. Pew Research Center: Internet & Technology. 2015. Gaming and Gamers URL: http://www.pewinternet.org/2015/12/15/gaming-and-gamers/ [accessed 2019-03-18] [WebCite Cache]
- Passmore C, Birk M, Mandryk R. The privilege of immersion: Racial and ethnic experiences, perceptions, and beliefs in digital gaming. New York, USA: ACM Press; 2018 Presented at: Annual Conference on Human Factors in Computing Systems - CHI 2018; April 2018; Montreal, Canada p. 383. [CrossRef]
- Osborn G. Newzoo. 2017. Male and Female Gamers: How Their Similarities and Differences Shape the Games Market URL: https://newzoo.com/insights/articles/male-and-female-gamers-how-their-similarities-and-differences-shape-the-games-market/ [accessed 2019-03-18] [WebCite Cache]
- Mittelstadt BD, Allo P, Taddeo M, Wachter S, Floridi L. The ethics of algorithms: Mapping the debate. Big Data & Society 2016 Dec 19;3(2):205395171667967. [CrossRef]
- Kitchin R. The Data Revolution: Big Data, Open Data, Data Infrastructures And Their Consequences. Thousand Oaks, CA, USA: Sage Publications Ltd; 2019.
- Bruckman A. Research Ethics and HCI. In: Olson J, Kellogg W, editors. Ways Of Knowing In HCI. New York, USA: Springer; 2014:449-468.
- Mindstrong Health. URL: https://mindstronghealth.com/ [accessed 2019-03-18] [WebCite Cache]
- Vogel L. AI opens new frontier for suicide prevention. CMAJ 2018 Dec 29;190(4):E119 [FREE Full text] [CrossRef] [Medline]
- Jashinsky J, Burton SH, Hanson CL, West J, Giraud-Carrier C, Barnes MD, et al. Tracking suicide risk factors through Twitter in the US. Crisis 2014;35(1):51-59. [CrossRef] [Medline]
- Turkle S. Life on the Screen: Identity in the Age of the Internet. New York, USA: Simon & Schuster; 1997.
- Livingston I, Gutwin C, Mandryk R, Birk M. How players value their characters in world of warcraft. New York, USA: ACM Press; 2014 Presented at: ACM Conference on Computer Supported Cooperative Work and Social Computing - CSCW 2014; February 2014; Baltimore, USA p. 1333-1343. [CrossRef]
- Friedrich M. Depression Is the Leading Cause of Disability Around the World. JAMA 2017 Apr 18;317(15):1517. [CrossRef] [Medline]
- Chisholm D, Sweeny K, Sheehan P, Rasmussen B, Smit F, Cuijpers P, et al. Scaling-up treatment of depression and anxiety: a global return on investment analysis. Lancet Psychiatry 2016 May;3(5):415-424 [FREE Full text] [CrossRef] [Medline]
- Skevington SM, Lotfy M, O'Connell KA, WHOQOL Group. The World Health Organization's WHOQOL-BREF quality of life assessment: psychometric properties and results of the international field trial. A report from the WHOQOL group. Qual Life Res 2004 Mar;13(2):299-310. [CrossRef] [Medline]
- Nock M, Hwang I, Sampson N, Kessler R, Angermeyer M, Beautrais A, et al. Cross-national analysis of the associations among mental disorders and suicidal behavior: findings from the WHO World Mental Health Surveys. PLoS Med 2009 Aug;6(8):e1000123 [FREE Full text] [CrossRef] [Medline]
- Kleiman EM, Nock MK. Real-time assessment of suicidal thoughts and behaviors. Current Opinion in Psychology 2018 Aug;22:33-37. [CrossRef]
- Oorschot M, Lataster T, Thewissen V, Wichers M, Myin-Germeys I. Mobile assessment in schizophrenia: a data-driven momentary approach. Schizophr Bull 2012 May;38(3):405-413 [FREE Full text] [CrossRef] [Medline]
|API: application programming interface|
|e-sport: electronic sport|
Edited by J Torous; submitted 24.01.19; peer-reviewed by M Colder Carras, O Danilina, A Korchi, MS Aslam; comments to author 14.02.19; revised version received 11.03.19; accepted 11.03.19; published 23.04.19
©Regan Lee Mandryk, Max Valentin Birk. Originally published in JMIR Mental Health (http://mental.jmir.org), 23.04.2019.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on http://mental.jmir.org/, as well as this copyright and license information must be included.