This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.
Conflicting data emerge from literature regarding the actual use of smartphone apps in medicine; some considered the introduction of smartphone apps in medicine to be a breakthrough, while others suggested that, in real-life, the use of smartphone apps in medicine is disappointingly low. Yet, digital tools become more present in medicine daily. To empower parents of a child with autism spectrum disorder, we developed the Smartautism smartphone app, which asks questions and provides feedback, using a screen with simple curves.
The purpose of this study was to evaluate usage of the app by caregivers of individuals with autism spectrum disorders.
We conducted a prospective longitudinal exploratory open study with families that have a child with autism spectrum disorder. Data were recorded over a period of 6 months, and the outcome criteria were (1) overall response rates for a feedback screen and qualitative questionnaires, and (2) response rates by degree of completion and by user interest, based on attrition.
Participants (n=65) had a very high intent to use the app during the 6-month period (3698/3900 instances, 94.8%); however, secondary analysis showed that only 46% of participants (30/65) had constant response rates over 50%. Interestingly, these users were characterized by higher use and satisfaction with the feedback screen when compared to low (
We found that real or perceived utility is an important incentive for parents who use empowerment smartphone apps.
RR2-10.1136/bmjopen-2016-012135
Autism spectrum disorder is a chronic disorder that affects daily life and can be a burden on parents, with consequences on their quality of life [
Management of children’s inappropriate behaviors by professionals (or advice from professionals) is not accessible at all times of the day, and families often criticize the short duration or low frequency of their consultations [
Digital mental health interventions have been of interest to the medical community for the past decade. Patients, caregivers, and medical practitioners are surrounded by apps and digital tools in everyday life; therefore, it seems natural that the mental health community, despite their initial criticism and poor initial acceptance [
We developed Smartautism—a smartphone app for parents with children with autism spectrum disorders—to meet the need for parental support between medical appointments. The app collects data through ecological momentary assessment [
But, even for tools with state-of-the-art development, validity, and usefulness, the rate of attrition may remain high. Eysenbach [
Moreover, the effect of the “law of attrition” can be a confounding factor in eHealth studies [
Our main objectives were to evaluate the usefulness, usability, and reliability of our smartphone app during a 6-month period (level 4 of the American Psychiatric Association hierarchical framework) to determine the acceptability of the app and to qualitatively evaluate the factors that could explain differences in use (in accordance with the principles of specific attrition bias [
Ethical approval was obtained from the
Parents of individuals diagnosed with autism spectrum disorders, using Autism Diagnostic Interview-Revised diagnoses based on National Health Authority recommendations [
To obtain approval from the National Center for Informatics and Liberty, we designed a secure data-handling pathway. Data were stored in the app on parents’ smartphones and were inaccessible by unauthorized people (an individual code was required when the app was opened). Families included in this study transmitted these data to the investigators using a strict, secure 5-step process: (1) Data in the smartphone of the participant (in-app coding) were encrypted with a 16-digit encryption key and a personal temporary code for access to data. (2) Encrypted data were transmitted to a secure server (Ivory Healthcare Inc). (3) Data were physically transferred by USB stick or disk from the server to Nantes University Hospital secure medical server at the end of the study period. (4) Once the data were uploaded on the intranet of the University Hospital, decryption was performed
The idea for the Smartautism [
Parents rated behavior in several day-to-day basic situations (meal, lunch, etc) and answered questions (
Smartautism is freely downloadable from the Apple Appstore and the Google Play Store but requires a key code for access.
Smartautism app questions for parents (1b) about their children and themselves (1a) (image adapted from [
Smartautism app questions for parents about their children, two times every week (image adapted from [
We conducted a prospective longitudinal exploratory open study. We collected qualitative and quantitative data over a 6-month period to assess acceptability of the app. Acceptability has several aspects. In digital technology, it may be reduced to 4 dimensions: (1) usefulness, (2) usability, (3) reliability, and (4) risk [
To assess usefulness and intent to use, our primary outcome was overall response rate,
Completion rate was categorized into 5 levels: 100%, 99% to 80%, 79% to 50%, 49% to 20%, and <20%. The choice of intervals was not based on a specific theory, but on facilitating ranking of user behavior; the size of the intervals was modeled on Cohen effect size intervals.
We were able to determine the total number of views of the feedback screen because the screen requires activation by the user. Raw attrition proportions at different steps in time can be illustrated as attrition curves, and the shape of these curves (logarithmic or sigmoid) allows formulation of hypotheses about the causes of attrition [
We assessed attrition criteria [
We categorized patients according to their completion rate (high: ≥80%, moderate: 20-80, and low: <20) and used the Kruskal-Wallis test to isolate questions for which there was at least1 difference between groups (high, moderate, low). To determine posthoc stochastic dominance, we used the Dwass-Steel-Critchlow-Fligner pairwise comparison test.
Use and user experience question content (adapted from [
Item | Question content | Impact on nonusage and dropout attrition rate |
1 | Quality and relevance of information given before the trial | If low, risk of unrealistic expectations which results in a disengagement |
2 | Ease of the inclusion process (consent, implementation) | Quality of recruitment affects attrition. if it is too easy to enroll then the dropout rate may be high |
3 | Ease of drop out/stop using it | This parameter can negatively influence the use of the app |
4 | Ease of use and reliability of the technical interface | Poor usability (complexity of the interaction between an object and its user) contributes to a high rate of attrition |
5 | “Incentive” or “push” factors (callbacks, reminders, research assistants chasing participants) | This parameter can positively influence the use of the app (staying more in the trial) |
6 | Personal contact (during registration and inclusion) via face-to-face or by phone, rather than virtual contacts | Human contact promotes the use of the app |
7 | General quality of the feedback information and of the information summary screen | Positive feedback and encouragement positively influence the use of the app |
8 | Perceived benefits of interest in completing the study | Motivational factor that decreases attrition |
9 | Free to use | Paying more commits the user and decreases attrition |
10 | Time and workload required by the apps | If the burden is too high, it may result in higher attrition |
11 | Existence of concurrent interventions (web, therapy) | Risk that the user no longer perceives the specific interest of the app |
12 | Major life events, or of society, which could have stopped using the app | Lead to distraction and nonuse by shifting priorities |
13 | Experience of the other user (or being able to obtain help) | Indirectly through to dropout and nonusage |
Based on initial results that showed different app use behaviors, we separated participants into 2 groups. Group A comprised participants who consistently had completion rates above 50%, and group B comprised participants who consistently had completion rates below 50%.
A total of 124 families were consecutively screened during an 18-month recruitment period (
The overall response rate was high (3698/3900, 94.8%). Of the 3900 instances (for n=65 participants), only 1347 were completed in full, while 837 instances were more than 80% complete, 509 instances were between 50% and 79% complete, 897 instances were between 20% and 49% complete, and 310 instances were less than 20% complete (of which, 202 were 0% complete).
The number of responses completed by participants tended to decrease over time, mainly after the third month (
Flow diagram.
Study population of parents with children with autism spectrum disorders.
Characteristics | Value (n=65) | |
Age at diagnosisa (months), mean (SD) | 20.3 (6.3) | |
Age of the father (years), mean (SD) | 34.52 (6.52) | |
Age of the mother (years), mean (SD) | 33.63 (3.32) | |
Age of the children (years), mean (SD) | 7.56 (4.52) | |
|
|
|
|
None | 52(80) |
|
Epilepsy | 8 (12.3) |
|
Chromosomal abnormalities | 9 (13.8) |
|
Endocrine | 2 (3) |
|
|
|
|
Male | 46 (72) |
|
Female | 19 (28) |
|
|
|
|
Apple iPhone | 34 (52.3) |
|
Android | 31 (47.7) |
aAutism Diagnostic Interview-Revised assessed diagnosis.
bSome patients may have >1 association; therefore, percentages do not add to 100%.
Distribution of the answers by month of use and completion rate (each month 650 answers were expected).
The distribution of Android and Apple smartphones for participants with completion rates >50% (group A; Android: 48%, Apple: 52%) was similar to that for participants with completion rates <50% (group B; Android: 46%, Apple: 54%).
The ages of the fathers and mothers in group A (father: mean 24.3 years, SD 3.5; mother: mean 25.5 years, SD 4.0;
The individuals most likely to fully answer the questions were those who were most likely to display the feedback screen (
There were 7 questions (questions 1, 2, 4, 5, 7, 8, and 10) with between-group differences (
Participants who displayed the feedback screen by month, based on their completion rate.
Comparison between high, moderate, and low users of the Smartautism app for each item of the attrition questionnaire.
Item | Pairwisea | Between-group | ||||||||||
|
Chi-square ( |
Effect size | ||||||||||
|
|
|
19.70 (2) | <.001 | 0.519 | |||||||
|
High | Moderate | −3.88 | .02 |
|
|
|
|||||
|
High | Low | −5.72 | <.001 |
|
|
|
|||||
|
Moderate | Low | —b | — |
|
|
|
|||||
|
|
|
19.70 (2) | <.001 | 0.519 | |||||||
|
High | Moderate | −3.88 | .02 |
|
|
|
|||||
|
High | Low | −5.72 | <.001 |
|
|
|
|||||
|
Moderate | Low | — | — |
|
|
|
|||||
3 |
|
|
— | — | — | |||||||
|
|
|
30.10 (2) | <.001 | 0.792 | |||||||
|
High | Moderate | −4.92 | .001 |
|
|
|
|||||
|
High | Low | −7.10 | <.001 |
|
|
|
|||||
|
Moderate | Low | 4.69 | .003 |
|
|
|
|||||
|
|
|
8.76 (2) | .01 | 0.231 | |||||||
|
High | Moderate | −1.27 | .64 |
|
|
|
|||||
|
High | Low | −3.99 | .01 |
|
|
|
|||||
|
Moderate | Low | −1.56 | .51 |
|
|
|
|||||
6 |
|
|
— | — | — | |||||||
|
|
|
26.45 (2) | <.001 | 0.696 | |||||||
|
High | Moderate | −4.27 | .007 |
|
|
|
|||||
|
High | Low | −6.57 | <.001 |
|
|
|
|||||
|
Moderate | Low | −2.71 | .135 |
|
|
|
|||||
|
|
|
32.31 (2) | <.001 | 0.850 | |||||||
|
High | Moderate | −5.68 | <.001 |
|
|
|
|||||
|
High | Low | −7.51 | <.001 |
|
|
|
|||||
|
Moderate | Low | −1.56 | .51 |
|
|
|
|||||
9 |
|
|
— | — | — | |||||||
|
|
|
29.06 (2) | < .001 | 0.765 | |||||||
|
High | Moderate | −4.92 | .001 |
|
|
|
|||||
|
High | Low | −7.04 | <.001 |
|
|
|
|||||
|
Moderate | Low | 0 | >.999 |
|
|
|
|||||
11 |
|
|
— | — | — | |||||||
12 |
|
|
— | — | — | |||||||
13 |
|
|
— | — | — |
aPairwise comparisons are presented for significant items.
bMissing or unquantifiable data.
As expected from previous literature [
These results are consistent with those in previous studies [
Despite encouraging results, our study shows that half of the users (35/65, 54%) did not use the empowerment app regularly. We can assume that the specific parent population in our study (known from our facility) was intrinsically motivated; therefore, we can consider several reasons to explain the constant gap between real usage and expectation in digital health care [
Often, individuals use health apps only for a short period of time. This “law of attrition,” corresponding to the loss of participants during an experiment, raises some questions [
Recently, the components of engagement in technology, in particular with respect to apps, were evaluated and a user scale based on 4 dimensions was proposed [
Low users demonstrated low levels of satisfaction in various areas. They expressed reluctance to spend too much time using the app (question 10, high vs low:
Perfect engagement with apps will never exist; O’Brien and Toms [
The Smartautism app is a first and encouraging step in digital empowerment for families of individual with autism spectrum disorders. Our results suggest that users need to perceive the utility of digital tools in order to use them. We plan to add an advice section, through the feedback screen, providing guidance and suggestions generated by Bayesian network algorithm [
This research was funded by
We thank Aude Doudard, MD, and Philippe Duverger, MD, for their help in the early stages of the project, and Jean-Christophe Guilbaud assistance with English editing.
OB, VA, DB, FGB, and SM were involved in the conception and design of the study. OB, VV, and FGB recruited patients. OB, VA, and SM drafted the manuscript, and DB, FGB, VV, and VA revised it critically for important intellectual content. The figures were prepared by SM, FGB, and OB. All authors gave final approval for publication.
None declared.