Background: Conflicting data emerge from literature regarding the actual use of smartphone apps in medicine; some considered the introduction of smartphone apps in medicine to be a breakthrough, while others suggested that, in real-life, the use of smartphone apps in medicine is disappointingly low. Yet, digital tools become more present in medicine daily. To empower parents of a child with autism spectrum disorder, we developed the Smartautism smartphone app, which asks questions and provides feedback, using a screen with simple curves.
Objective: The purpose of this study was to evaluate usage of the app by caregivers of individuals with autism spectrum disorders.
Methods: We conducted a prospective longitudinal exploratory open study with families that have a child with autism spectrum disorder. Data were recorded over a period of 6 months, and the outcome criteria were (1) overall response rates for a feedback screen and qualitative questionnaires, and (2) response rates by degree of completion and by user interest, based on attrition.
Results: Participants (n=65) had a very high intent to use the app during the 6-month period (3698/3900 instances, 94.8%); however, secondary analysis showed that only 46% of participants (30/65) had constant response rates over 50%. Interestingly, these users were characterized by higher use and satisfaction with the feedback screen when compared to low (P<.001) and moderate (P=.007) users.
Conclusions: We found that real or perceived utility is an important incentive for parents who use empowerment smartphone apps.
International Registered Report Identifier (IRRID): RR2-10.1136/bmjopen-2016-012135
Autism spectrum disorder is a chronic disorder that affects daily life and can be a burden on parents, with consequences on their quality of life [, ]. Parents of children with autism spectrum disorders have greater anxiety levels than parents of children without developmental disorders [ ]. Some major autism symptoms, such as communication disorders and aggression [ ], along with behavioral symptoms (such as agitation, feeding difficulties, sleep disturbances, obsessive behavior, or refusal of authority), are well-known stressors.
Management of children’s inappropriate behaviors by professionals (or advice from professionals) is not accessible at all times of the day, and families often criticize the short duration or low frequency of their consultations [- ]. Digital tools might be of particular interest in such situations by filling the gap between psychiatric consultations.
Digital mental health interventions have been of interest to the medical community for the past decade. Patients, caregivers, and medical practitioners are surrounded by apps and digital tools in everyday life; therefore, it seems natural that the mental health community, despite their initial criticism and poor initial acceptance , should now attempt to understand and develop digital mental health interventions programs. The benefits of digital mental health interventions are known: (1) low cost at a time when there are not enough professionals to help all patients, (2) no commute, and thus a gain of time, (3) increased patient involvement, and (4) increased patient accessibility, when needed. However, recent studies [ - ] have shown a gap between the alleged, and sometimes scientifically validated, purposes of such digital interventions and their actual use by patients and professionals. The American Psychiatric Association proposed a 5-level hierarchical framework to evaluate an app: quality of information, basic medical decision making (and nonmaleficence), scientific evidence usability, and interoperability [ ].
We developed Smartautism—a smartphone app for parents with children with autism spectrum disorders—to meet the need for parental support between medical appointments. The app collects data through ecological momentary assessment  or the experience sampling method [ ] from parents, who answer simple questions about daily life [ ]. The app then provides a feedback screen with a graphical representation of their score, allowing parents to see the evolution of the scores during the weeks before to help parents be objective about the difficulties they are facing
But, even for tools with state-of-the-art development, validity, and usefulness, the rate of attrition may remain high. Eysenbach  defined this phenomenon as “nonusage attrition” and proposed influential factors such as the nature of the perceived advantage of an innovation. Likewise, usability, determined by its complexity, contributes to intention to use. It is fundamental that participants fit within the intended scope of users of the apps; therefore, appropriate information about the main purpose of the apps must be provided in order to avoid unrealistic user expectations. Finally, the nature of the feedback given by the apps may encourage users to continue and reinforce their engagement.
Moreover, the effect of the “law of attrition” can be a confounding factor in eHealth studies . Yet, thorough analysis of these factors can explain the difference between the optimism of the designers and the reality of the rate of use. These considerations should drive the development of a digital tool as well as the interpretation of study results.
Our main objectives were to evaluate the usefulness, usability, and reliability of our smartphone app during a 6-month period (level 4 of the American Psychiatric Association hierarchical framework) to determine the acceptability of the app and to qualitatively evaluate the factors that could explain differences in use (in accordance with the principles of specific attrition bias ).
Ethical approval was obtained from the Comité de Protection des Personnes, Ouest V (January 2017). All participants gave consent.
Parents of individuals diagnosed with autism spectrum disorders, using Autism Diagnostic Interview-Revised diagnoses based on National Health Authority recommendations , from 3 to 16 years of age (at the time of the study) in the Pays de la Loire region of France were recruited. All families at the Department of Child and Adolescent Psychiatry of Centre Hospitalier Universitaire de Nantes and from the Regional Center for Autism (Pays de la Loire) were eligible. Inclusion criteria were having a child with an autism spectrum disorder diagnosis based on International Classification of Diseases tenth revision criteria, having a smartphone (iOS Apple or Android), and signing the consent form for participation. Exclusion criteria were having several children with autism spectrum disorder diagnoses, having children living in more than 2 houses, having an old smartphone with which there would be a significant decrease in user experience, or having no personal smartphone. We included the first 100 families that agreed to participate and that met the criteria. Because the study was exploratory, we decide that it was not necessary to randomize patients.
To obtain approval from the National Center for Informatics and Liberty, we designed a secure data-handling pathway. Data were stored in the app on parents’ smartphones and were inaccessible by unauthorized people (an individual code was required when the app was opened). Families included in this study transmitted these data to the investigators using a strict, secure 5-step process: (1) Data in the smartphone of the participant (in-app coding) were encrypted with a 16-digit encryption key and a personal temporary code for access to data. (2) Encrypted data were transmitted to a secure server (Ivory Healthcare Inc). (3) Data were physically transferred by USB stick or disk from the server to Nantes University Hospital secure medical server at the end of the study period. (4) Once the data were uploaded on the intranet of the University Hospital, decryption was performed in situ using specific software after the principal investigator entered a short automatically generated validation code. (5) Data were organized in a data spreadsheet for analysis and calculation.
The idea for the Smartautism  app emerged during discussions with associations of parents of people with autism, who were informally involved in the development process. The acceptability assessment of Smartautism was one step of a large digital empowerment project. The app is a combination of ecological momentary assessment with feedback that may be used by parents to adapt their educational behavior. The parents must provide regular ratings (mandatory twice a week but additional on-demand ratings were possible if needed by the parents) for a long period, and the feedback screen provides a synthesis of the ratings.
Parents rated behavior in several day-to-day basic situations (meal, lunch, etc) and answered questions (and ) in the app. Parents had the opportunity to answer ecological questions about their children and their own psychological state [ ]. The feedback screen converted user scores to graphs in order to allow users to data visualize their responses to provide an overview of the information.
Smartautism is freely downloadable from the Apple Appstore and the Google Play Store but requires a key code for access.
We conducted a prospective longitudinal exploratory open study. We collected qualitative and quantitative data over a 6-month period to assess acceptability of the app. Acceptability has several aspects. In digital technology, it may be reduced to 4 dimensions: (1) usefulness, (2) usability, (3) reliability, and (4) risk . Usefulness reflects intent to use, which is a good predictor of usage behavior. Acceptability encompasses several concepts, for example, in ISO standards [ ], usability comprises 3 dimensions: effectiveness, efficiency, and satisfaction.
To assess usefulness and intent to use, our primary outcome was overall response rate, R = (number of dates with responses) / (possible instances with responses over the 6-month period). In this case, there were 60 possible instances. All data were directly extracted from the app.
Completion rate was categorized into 5 levels: 100%, 99% to 80%, 79% to 50%, 49% to 20%, and <20%. The choice of intervals was not based on a specific theory, but on facilitating ranking of user behavior; the size of the intervals was modeled on Cohen effect size intervals.
We were able to determine the total number of views of the feedback screen because the screen requires activation by the user. Raw attrition proportions at different steps in time can be illustrated as attrition curves, and the shape of these curves (logarithmic or sigmoid) allows formulation of hypotheses about the causes of attrition .
We assessed attrition criteria  in the form of a questionnaire ( ). This questionnaire was given to users at the end of the 6-month period. Responses were rated on a 5-point Likert scale (where 1 indicated the worst satisfaction and 5 indicated high satisfaction).
We categorized patients according to their completion rate (high: ≥80%, moderate: 20-80, and low: <20) and used the Kruskal-Wallis test to isolate questions for which there was at least1 difference between groups (high, moderate, low). To determine posthoc stochastic dominance, we used the Dwass-Steel-Critchlow-Fligner pairwise comparison test.
|Item||Question content||Impact on nonusage and dropout attrition rate|
|1||Quality and relevance of information given before the trial||If low, risk of unrealistic expectations which results in a disengagement|
|2||Ease of the inclusion process (consent, implementation)||Quality of recruitment affects attrition. if it is too easy to enroll then the dropout rate may be high|
|3||Ease of drop out/stop using it||This parameter can negatively influence the use of the app|
|4||Ease of use and reliability of the technical interface||Poor usability (complexity of the interaction between an object and its user) contributes to a high rate of attrition|
|5||“Incentive” or “push” factors (callbacks, reminders, research assistants chasing participants)||This parameter can positively influence the use of the app (staying more in the trial)|
|6||Personal contact (during registration and inclusion) via face-to-face or by phone, rather than virtual contacts||Human contact promotes the use of the app|
|7||General quality of the feedback information and of the information summary screen||Positive feedback and encouragement positively influence the use of the app|
|8||Perceived benefits of interest in completing the study||Motivational factor that decreases attrition|
|9||Free to use||Paying more commits the user and decreases attrition|
|10||Time and workload required by the apps||If the burden is too high, it may result in higher attrition|
|11||Existence of concurrent interventions (web, therapy)||Risk that the user no longer perceives the specific interest of the app|
|12||Major life events, or of society, which could have stopped using the app||Lead to distraction and nonuse by shifting priorities|
|13||Experience of the other user (or being able to obtain help)||Indirectly through to dropout and nonusage|
Based on initial results that showed different app use behaviors, we separated participants into 2 groups. Group A comprised participants who consistently had completion rates above 50%, and group B comprised participants who consistently had completion rates below 50%.
A total of 124 families were consecutively screened during an 18-month recruitment period (), of which, 65 families, with 46 boys and 19 girls, were included in our study ( ).
The overall response rate was high (3698/3900, 94.8%). Of the 3900 instances (for n=65 participants), only 1347 were completed in full, while 837 instances were more than 80% complete, 509 instances were between 50% and 79% complete, 897 instances were between 20% and 49% complete, and 310 instances were less than 20% complete (of which, 202 were 0% complete).
The number of responses completed by participants tended to decrease over time, mainly after the third month (). However, responses were consistently completed throughout the study by participants who completed over 90%. Only 13 of the 65 participants (20%) completed all responses. Of the 65 participants, only 1 participant had a 0% completion rate, while 17 participants (26%) had completion rates that remained above 50%, and 34 participants (52%) had progressively decreasing completion rates. Overall, 46% of participants (30/65) consistently had completion rates over 50%, and 54% (35/65) had completion rates under 50%.
|Age at diagnosisa (months), mean (SD)||20.3 (6.3)|
|Age of the father (years), mean (SD)||34.52 (6.52)|
|Age of the mother (years), mean (SD)||33.63 (3.32)|
|Age of the children (years), mean (SD)||7.56 (4.52)|
|Associated diseaseb, n (%)|
|Chromosomal abnormalities||9 (13.8)|
|Gender of the children, n (%)|
|Phone, n (%)|
|Apple iPhone||34 (52.3)|
aAutism Diagnostic Interview-Revised assessed diagnosis.
bSome patients may have >1 association; therefore, percentages do not add to 100%.
Grouped by Usage
The distribution of Android and Apple smartphones for participants with completion rates >50% (group A; Android: 48%, Apple: 52%) was similar to that for participants with completion rates <50% (group B; Android: 46%, Apple: 54%).
The ages of the fathers and mothers in group A (father: mean 24.3 years, SD 3.5; mother: mean 25.5 years, SD 4.0; P=.01) were lower than those in group B (father: mean 29.0 years, SD 6.0; mother: mean 32.0 years, SD 6.4; P=.03). We did not find any significant differences for age of the child (P=.31) or city type (more or less than 20,000 inhabitants: P=.117).
The individuals most likely to fully answer the questions were those who were most likely to display the feedback screen (). There were 39 attrition questionnaires with responses: 9 participants were low users, 3 participants were moderate users, and 27 participants were high users.
There were 7 questions (questions 1, 2, 4, 5, 7, 8, and 10) with between-group differences (). For question 7, which evaluated user satisfaction with the feedback screen, there were significant differences both between high and moderate (P=.007) and high and low (P<.001) users; however, the difference between low and moderate users was not significant (P=.14).
|W test statistic||P value||Chi-square (df)||P value||Effect size|
|10||29.06 (2)||< .001||0.765|
aPairwise comparisons are presented for significant items.
bMissing or unquantifiable data.
As expected from previous literature , in our study population, there were more male children than female children with autism spectrum disorder. We found that participants had a very high intent to use the app during the 6-month period (3698/3900, 94.8%); however, secondary analysis showed that only 46% of participants (30/65) consistently had completion rates over 50%. High users were characterized by having higher satisfaction (question 7) with the feedback screen when compared to low (P<.001) and moderate (P=.007) users.
These results are consistent with those in previous studies  on real-life use of smartphone apps in digital medicine and in psychiatry, which show disappointing use rates despite the high expectations of professionals. We did not take the high overall response rate (94.8%) into account in our interpretations and discussions because we recruited families and patients who were already very dedicated to our facility. There were many screened families (59/124 47.5%) that were not included in the study because they lacked a smartphone or had old devices, despite the fact that overall rate of 79% of people in France have smartphones [ ]. Moreover, the distribution of smartphone type in our study population was not representative of that of the French population—half of the participants used iPhones, whereas iPhones are used by only 15% of the population in France [ ].
Despite encouraging results, our study shows that half of the users (35/65, 54%) did not use the empowerment app regularly. We can assume that the specific parent population in our study (known from our facility) was intrinsically motivated; therefore, we can consider several reasons to explain the constant gap between real usage and expectation in digital health care : (1) the app did not offer sufficient gain for the families for regular use, which is suggested by the fact that only high users were the most interested by the feedback screen, (2) security concerns are always present even if we did explain (orally and with notice inside the app) the data protection measures that we used, and (3) design and ergonomics were tidy and elegant in our app, but we suspect it is very difficult to develop an app that can compete with those developed by large corporations and used in everyday life by our population.
Often, individuals use health apps only for a short period of time. This “law of attrition,” corresponding to the loss of participants during an experiment, raises some questions . Eysenbach [ ] highlighted that this rate decreases if a user perceives a relative advantage with the app (perception that this innovation is superior to the idea that it replaces). Our attrition questionnaire results (responses from partial study population: 39/65, 60%) also strongly suggest that reminders and feedback, when appreciated and accepted, are a strong motivation and may drive participants to be high users instead of low users. If engagement can be viewed as a product of experience or interaction, the presence of push factors (reminders) or positive feedback appears to be efficient for engagement.
Recently, the components of engagement in technology, in particular with respect to apps, were evaluated and a user scale based on 4 dimensions was proposed : focused attention, perceived usability, aesthetic appeal, and reward. The reward dimension corresponds to the hedonistic aspects of the experience; the feedback screen could enhance this dimension of the app when used; therefore, this component needs to be further developed.
Low users demonstrated low levels of satisfaction in various areas. They expressed reluctance to spend too much time using the app (question 10, high vs low: P<.001). They also were annoyed by technical issues (question 4, high vs low: P<.001). It appears that too much complexity in the interaction between an app and its users may contribute to a high rate of attrition. These parameters can negatively influence the use of the app. Interestingly, low users were also significantly more dissatisfied by the information and inclusion process of the study.
Perfect engagement with apps will never exist; O’Brien and Toms  argue that engagement is not a static, but multistage, process—with a point of engagement, then a period of engagement, a point of disengagement, and a period of reengagement. Moreover, when behavioral symptoms are not present, engagement may decrease due to lack of necessity. Our results show that motivation for long-term use is strongly associated with perceived benefits in completing the study (high vs moderate: P<.001; high vs low: P<.001), which emphasizes the importance of precise framing during the patient inclusion process (quality of information, explanation of consent, purpose, and benefit). There is a need for stronger collaboration between academics and digital specialists to provide accurate empowerment tools. Indeed, laws of attrition encompass classic motivational rules for study enrollment and new digital constraints related to technical aspects (design, acceptability, stability) [ ]. App development requires academic studies that take laws of attrition into account, mainly because, for the past decade, empowerment and enhancement of patients’ responsibilities in their own health care have been growing [ ]. Our app is only a first step, but it is consistent with the on-going shift toward the use of such technologies in psychiatry, especially for outpatients.
The Smartautism app is a first and encouraging step in digital empowerment for families of individual with autism spectrum disorders. Our results suggest that users need to perceive the utility of digital tools in order to use them. We plan to add an advice section, through the feedback screen, providing guidance and suggestions generated by Bayesian network algorithm  because more powerful algorithms (enhanced by artificial intelligence) might be useful in providing accurate and personalized advice for users.
This research was funded by Institut de Recherche en Santé Publique with additional funding provided by Agence Régionale de Santé des Pays de la Loire. Our study received funding from the Regional Health Agency and the Autism Resource Centre of the Loire Valley, as well as a grant from the National Center for Autonomy/Research Institute of Epidemiology and Public Health.
We thank Aude Doudard, MD, and Philippe Duverger, MD, for their help in the early stages of the project, and Jean-Christophe Guilbaud assistance with English editing.
OB, VA, DB, FGB, and SM were involved in the conception and design of the study. OB, VV, and FGB recruited patients. OB, VA, and SM drafted the manuscript, and DB, FGB, VV, and VA revised it critically for important intellectual content. The figures were prepared by SM, FGB, and OB. All authors gave final approval for publication.
Conflicts of Interest
- Baghdadli A, Pry R, Michelon C, Rattaz C. Impact of autism in adolescents on parental quality of life. Qual Life Res 2014 Aug;23(6):1859-1868. [CrossRef] [Medline]
- Farnik M, Brożek G, Pierzchała W, Zejda JE, Skrzypek M, Walczak. Development, evaluation and validation of a new instrument for measurement quality of life in the parents of children with chronic disease. Health Qual Life Outcomes 2010;8:151 [FREE Full text] [CrossRef] [Medline]
- Hayes SA, Watson SL. The impact of parenting stress: a meta-analysis of studies comparing the experience of parenting stress in parents of children with and without autism spectrum disorder. J Autism Dev Disord 2013 Mar;43(3):629-642. [CrossRef] [Medline]
- Davis NO, Carter AS. Parenting stress in mothers and fathers of toddlers with autism spectrum disorders: associations with child characteristics. J Autism Dev Disord 2008 Aug;38(7):1278-1291. [CrossRef] [Medline]
- Chamak B, Bonniau B. Trajectories, long-term outcomes and family experiences of 76 adults with autism spectrum disorder. J Autism Dev Disord 2016 Mar;46(3):1084-1095. [CrossRef] [Medline]
- Dickson KS, Suhrheinrich J, Rieth SR, Stahmer AC. Parent and teacher concordance of child outcomes for youth with autism spectrum disorder. J Autism Dev Disord 2018 May;48(5):1423-1435 [FREE Full text] [CrossRef] [Medline]
- Edwards AG, Brebner CM, McCormack PF, MacDougall CJ. From 'parent' to 'expert': how parents of children with autism spectrum disorder make decisions about which intervention approaches to access. J Autism Dev Disord 2018 Jun;48(6):2122-2138. [CrossRef] [Medline]
- Locke J, Ibanez LV, Posner E, Frederick L, Carpentier P, Stone WL. Parent perceptions about communicating with providers regarding early autism concerns. Pediatrics 2020 Apr;145(Suppl 1):S72-S80. [CrossRef] [Medline]
- Timberlake MT, Leutz WN, Warfield ME, Chiri G. "In the driver's seat": Parent perceptions of choice in a participant-directed medicaid waiver program for young children with autism. J Autism Dev Disord 2014 Apr;44(4):903-914. [CrossRef] [Medline]
- Bourla A, Ferreri F, Ogorzelec L, Peretti C, Guinchard C, Mouchabac S. Psychiatrists' attitudes toward disruptive new technologies: mixed-methods study. JMIR Ment Health 2018 Dec 14;5(4):e10240 [FREE Full text] [CrossRef] [Medline]
- Lipschitz J, Hogan TP, Bauer MS, Mohr DC. Closing the research-to-practice gap in digital psychiatry: the need to integrate implementation science. J Clin Psychiatry 2019 May 14;80(3):E1-E3 [FREE Full text] [CrossRef] [Medline]
- Torous J, Haim A. Dichotomies in the development and implementation of digital mental health tools. Psychiatr Serv 2018 Dec;69(12):1204-1206. [CrossRef]
- Torous J, Wisniewski H, Liu G, Keshavan M. Mental health mobile phone app usage, concerns, and benefits among psychiatric outpatients: comparative survey study. JMIR Ment Health 2018 Nov 16;5(4):e11715-eE1174 [FREE Full text] [CrossRef] [Medline]
- The app evaluation model. American Psychiatric Association. 2019. URL: https://www.psychiatry.org/psychiatrists/practice/mental-health-apps/app-evaluation-model [accessed 2021-08-11]
- Shiffman RN. Informatics and general pediatrics. Curr Opin Pediatr 1994 Oct;6(5):538-543. [Medline]
- Csikszentmihalyi M, Larson R. Validity and reliability of the experience-sampling method. J Nerv Ment Dis 1987 Sep;175(9):526-536. [Medline]
- Bonnot O, Bonneau D, Doudard A, Duverger P. Rationale and protocol for using a smartphone application to study autism spectrum disorders: smartautism. BMJ Open 2016 Nov 22;6(11):e012135 [FREE Full text] [CrossRef] [Medline]
- Eysenbach G. The law of attrition. J Med Internet Res 2005;7(1):e11 [FREE Full text] [CrossRef] [Medline]
- Autisme. Haute Autorité de Santé. URL: https://www.has-sante.fr/jcms/c_468812/fr/recommandations-pour-la-pratique-professionnelle-du-diagnostic-de-l-autisme [accessed 2021-09-03]
- ISO 9141-11. International Standards Organization. URL: http://www.iso.org [accessed 2021-09-03]
- Lord C, Elsabbagh M, Baird G, Veenstra-Vanderweele J. Autism spectrum disorder. Lancet 2018 Aug 11;392(10146):508-520 [FREE Full text] [CrossRef] [Medline]
- Autorité de Régulation des Communications Electroniques et des Postes (. Barometre du numérique. Gouvernement Français. 2019. URL: https://www.economie.gouv.fr/files/files/directions_services/cge/barometre-numerique-2019.pdf [accessed 2021-08-11]
- The Lancet Psychiatry. Digital health: the good, the bad, and the abandoned. Lancet Psychiatry 2019 Apr;6(4):273. [CrossRef] [Medline]
- Holdener M, Gut A, Angerer A. Applicability of the user engagement scale to mobile health: a survey-based quantitative study. JMIR Mhealth Uhealth 2020 Jan 3;8(1):e13244-eE1328. [CrossRef]
- O'Brien HL. Theoretical perspectives on user engagement. In: O’Brien HL, Cairns P, editors. Why Engagement Matters:Cross-Disciplinary Perspectives of User Engagement in Digital Media: Cham. Cham: Springer; 2016:1-26.
- Pulvirenti M, McMillan J, Lawn S. Empowerment, patient centred care and self-management. Health Expect 2014 Jun;17(3):303-310 [FREE Full text] [CrossRef] [Medline]
- Mouchabac S, Leray P, Adrien V, Gollier BF, Bonnot O. Beyond big data in behavioral psychiatry, the place of Bayesian Network. JMIR. 2021 (in Press) 2021 Sep 09:00-00 (forthcoming). [CrossRef]
Edited by J Torous; submitted 07.02.21; peer-reviewed by A Weiler, J Hamid, C Ochoa-Zezzatti; comments to author 13.03.21; revised version received 04.05.21; accepted 09.05.21; published 15.09.21Copyright
©Olivier Bonnot, Vladimir Adrien, Veronique Venelle, Dominique Bonneau, Fanny Gollier-Briant, Stephane Mouchabac. Originally published in JMIR Mental Health (https://mental.jmir.org), 15.09.2021.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.