Potentially eligible participants were recruited from a single large community mental health center in New England, United States, between May 2019 and February 2020 via flyers posted in waiting rooms and clinician invitations. Eligible participants were aged between 18 and 35 years, English speaking, stable in outpatient mental health treatment for SMI (ie, no hospitalization in the past 30 days per chart review), self-reported regular tobacco smokers (daily and nondaily) confirmed by breath carbon monoxide (CO)>7 parts per million (ppm) [28], and smartphone users (either Apple or Android). The desire to quit smoking was not required. We excluded patients who were pregnant or had a current, unstable substance use disorder per chart review or the patient’s mental health center clinician. We aimed to recruit 5 participants with psychotic disorders and 5 with other SMI diagnoses per app, as prior usability research has demonstrated that more than 80% of usability issues can be identified after the first 5 participants [29].

The QuitGuide and quitSTART apps are available free of charge on Smokefree.gov via the Apple Store or Google Play. Both apps encourage the user to set a quit date within 14 days, provide information about quitting, and allow users to enter personal data, such as how many cigarettes they smoked per day, what times of day they tend to smoke, and how much they spend per pack of cigarettes. They allow users to track cigarette cravings, slips (defined as cigarettes smoked for this study), and moods and provide information regarding coping with these experiences. They also provide information on users’ progress, such as cigarettes avoided and dollars saved by not smoking. In addition, users can connect to social media through the apps.

Although QuitGuide and quitSTART offer many similar features, their design and content differ in a number of ways (Figure 1). QuitGuide offers a relatively linear layout, utilizes darker colors that convey a serious tone, and provides information through text with minimal graphics. Users can type journal entries and can read a text-only guide on how to quit smoking. In contrast, quitSTART utilizes a more complex layout with bright colors and informal language that results in a cheery tone and prominently displays relatively large symbols and few words within the icons. Information in quitSTART is displayed on swipeable cards, each with a different color background and 5 or fewer sentences. quitSTART provides a selection of 7 games for distraction that can be played within the app. Both apps allow users to set notifications based on time or location, and quitSTART also automatically sends check-in notifications that ask users how many cigarettes they have smoked since the last check-in (QuitGuide does not have an analogous check-in feature).

Potentially eligible participants completed an informed consent process and proceeded with the study procedures once eligibility was confirmed. The first 12 participants received a US $30 gift card to a retail store after completing each of the 2 study visits (for a total of US $60). To improve recruitment, we increased compensation to US $60 per visit (for a total of US $120) for the last 5 participants. The New Hampshire State Institutional Review Board approved and monitored all study activities.

Participation in the study lasted for 2 weeks. A trained researcher obtained demographics and diagnoses from record reviews and conducted structured interview assessments using a standardized procedure in which measures were administered aloud and answers were recorded on paper forms and then entered immediately into a computerized database. To assist with answering multiple-choice questions, answer choices were provided as visual cues on paper. A component of the assessment included a semistructured, open-ended interview, which was audio-recorded. Usability tasks were video-recorded.

At the first study visit (visit 1), participants completed a structured interview assessing demographics, tobacco use, and technology use. Participants were then randomly assigned to 1 of the 2 apps (QuitGuide or quitSTART) in groups of 4 blocked by age group [30]. They were asked to download the app on their smartphones. Researchers oriented the participants to the think-aloud method [31] and video-recorded participants as they completed a set of 9 predefined tasks without researcher guidance. Researchers then assessed participants’ perceived acceptability and ease of use of their assigned app using the measures described below.

At the conclusion of the first study visit, we provided participants with a brief in-person tutorial on using their assigned app. The tutorial focused on helping users obtain skills for 3 tasks within the app: setting a quit date, logging and viewing logged moods, and logging and viewing logged slips; these tasks were chosen based on guideline-recommended interventions [32], the importance of mental health symptoms to smoking in this population [33], and our prior findings that young adults with SMI want to track cigarettes [12], respectively. We instructed participants to use the app independently over the following two-week period. We recommended that they try to skip cigarettes using their assigned app, but we did not advise participants to quit smoking for this study.

Following this period of independent use, participants returned for a second study visit (visit 2) and were assessed for smoking, perceptions of acceptability and usability, and task performance within the usability task protocol.

As shown in Table 1, participants were 17 daily smokers with a mean age of 29 (SD 4) years, 41% (7/17) were diagnosed with psychotic disorders, and 94% (16/17) were Medicare or Medicaid beneficiaries (Table 1). Participants smoked an average of 15 (SD 7) cigarettes per day and were moderately dependent on tobacco (mean Fagerström score 4.4, SD 1.8). More than 90% (16/17) of participants reported using smartphone apps on a daily basis, and more than 75% (13/17) had previously downloaded an app related to health and wellness. Of the 17 participants, only 4 (24%) endorsed previously trying a smartphone app to aid in a quit smoking attempt; of the remaining 13 participants, 11 (85%) were unaware that smartphone apps were available to help people quit smoking.

A majority of participants agreed or strongly agreed at both visits that most of the proposed app features were important to help someone quit smoking (Multimedia Appendix 3). The importance of location tracking increased from 35% (6/17) of participants at visit 1 to 59% (10/17) at visit 2, whereas the importance of tracking smoking triggers decreased from 71% (12/17) to 47% (8/17). Less than half of the participants indicated that sharing progress on social media was important: 41% (7/17) at both visits.

The mean SUS scores for QuitGuide were similar at visits 1 and 2 (64; range 30-77.5, SD 18, and 66; range 25-85, SD 18, respectively). In contrast, mean SUS scores for quitSTART numerically increased from visit 1 (55; range 25-82.5, SD 20) to visit 2 (64; range 35-85, SD 16). Responses to the ease of use questions followed a similar pattern (Table 2). By the second visit, at least three-quarters of both QuitGuide and quitSTART users reported feeling satisfied with their app’s ease of use. In general, QuitGuide’s ease of use question responses were similar at visits 1 and 2, whereas affirmative responses to most questions regarding quitSTART’s ease of use increased between visits 1 and 2.

The observed usability task completion rates for both apps were high at both visits (Figures 2 and 3). At the first study visit, all 9 tasks were successfully completed by at least 75% of participants assigned to QuitGuide (7/9 for the first 8 tasks and 6/8 for the final task); similarly, 8 of the 9 tasks were successfully completed at the first study visit by at least 75% (6/8) of participants assigned to quitSTART.

The most common usability challenges occurred during attempts at 4 tasks: entering the number of cigarettes smoked per day, setting a quit date, connecting to social media, and uploading a photo. The first 3 of these tasks required navigation through menus, whereas the remaining tasks we assessed could be reached with a single click from the home screen, suggesting that menu navigation was associated with lower usability. The photo feature in QuitGuide (but not quitSTART) could also be accessed from the home screen but was uniquely challenging in that it only intermittently opened when users clicked on it because of a bug in the app (which has since been fixed).

Some feature locations were not intuitive to the participants. Users could enter their quit date during the initial app setup and later in their Quit Plan (QuitGuide) or Profile (quitSTART). Additionally, the apps were not designed to track cigarettes smoked on a daily basis but asked users to enter the average cigarettes smoked per day in the same locations (ie, Quit Plan or Profile). Many participants tried to enter cigarettes smoked or change their quit dates on the progress pages, but these pages were designed only for viewing information and not entering information. This sometimes led to participant frustration after repeated attempts to click on the pages’ inactive icons. Although some participants struggled with this issue, others were able to complete these tasks by accessing their Quit Plan in QuitGuide or their Profile in quitSTART.

Notably, setting a quit date required the user to set a date within the next 14 days. Although most users were ultimately able to find where to set a quit date, they were not planning to quit during this time frame, and the apps did not allow them to enter a later date (although quitSTART did include a not ready option). Both apps recommended users to choose a quit date within the next 14 days; however, most participants did not see this explanation located above the date selection field, and many users became confused when the apps defaulted to 14 days from the current date. They allowed the app to choose a quit date for them, even though they verbalized that the selected date was an unrealistic goal that they could not achieve.

Common menu navigation challenges involved misunderstanding menu labels and expecting to find features in certain locations in the app based on prior experience with other apps. A commonly misunderstood menu label was My tags for both apps. Although the intended meaning was to identify (ie, tag) the times and locations when participants were at a higher risk of smoking, many participants interpreted this label to indicate a connection to social media. Other tasks, such as uploading a photo in QuitGuide, proved problematic when participants tried to use their experience with other apps to guide them. For example, many participants expected to find this option in the Settings feature of the app, though the Settings feature does not contain this option. Instead, the photo feature could be accessed within the Quit Plan under Reasons to Be Smokefree. In contrast, quitSTART users were able to upload a photo in the Profile section of the app (QuitGuide does not have a Profile feature).

Finally, participants often used personal information (data) entry pathways to complete tasks unrelated to logging or tracking personal information in the app, potentially leading to inaccurate feedback to the user if the app tailors feedback based on these features. For example, users can obtain information about quitting in both apps by touching the slip button on the home page, and both apps provide a tally of the user’s entered slips on the Progress page. Participants in both groups frequently used the slip feature to obtain information about quitting, often favoring it over alternate pathways that involved more complicated menu navigation (which did not involve data entry). Although they were ultimately able to reach the desired end point, using these pathways when not planning to log information influences feedback on the Progress page, which users perceive as being inaccurate as discussed below.

Responses to the acceptability questions are presented in Table 2. Notably, more than half (5/9, 56%) of QuitGuide users indicated that they liked using the app at both visits, whereas the proportion of users who liked using quitSTART increased from 25% (2/8) at visit 1 to 63% (5/8) at visit 2. About two-thirds of users thought each app would help them quit smoking at the second study visit, although fewer participants in each group felt that their assigned app had all the functions they expected it to have at visit 2 compared with visit 1.

In contrast to the SUS scores, app use patterns demonstrated dramatically greater engagement with quitSTART than QuitGuide (Figure 4). Compared with QuitGuide users, quitSTART users demonstrated greater mean days of use (10.8, SD 3.5, vs 4.6, SD 2.8), greater mean total app interactions (41, SD 26, vs 5.6, SD 3.8), and greater median responses to notifications (18.5, range 0-37, vs 1, range 0-8) during the 13-day period.

At visit 2, 78% (7/9) QuitGuide users reported that they had tried to quit or cut back during the 2-week trial period (mean CO for all QuitGuide users 25, SD 12); no participants quit. Similarly, 75% (6/8) of quitSTART users reported trying to quit or cut back during the study (mean CO for all quitSTART users 25, SD 16). In fact, 25% (2/8) of quitSTART users reported that they no longer smoked at visit 2 (both confirmed with breath CO <7 ppm), having instead switched completely to e-cigarettes.

Themes resulting from qualitative data analysis of QuitGuide interview data conveyed a mildly positive view of the app. In contrast, the themes among quitSTART users included stronger reactions that were both positive and negative. A major theme among QuitGuide users at both visits was that the app was easy to use; although a minor theme, participants found some aspects of navigation confusing. Among quitSTART users, difficult navigation was a theme during the first visit, but ease of use was a stronger theme during the second visit.

The most prominent theme at visit 1 was the same for both apps: participants liked that the apps used a positive and supportive tone and provided motivational quotes and feedback on money saved. They thought that the apps could track cigarettes smoked on a daily basis and liked the idea of that feature. Most quitSTART users also expressed interest in the games included within the quitSTART app at visit 1 (QuitGuide does not offer games).

At visit 2, many participants continued to perceive both apps as positive and supportive and noted that this was a strength of the apps. Many participants voiced a general concern that a negative tone or repeated reminders of a lack of progress would evoke feelings of guilt and failure, which could undermine their quit attempts. Although some participants worried that calling a smoked cigarette a slip in these apps could evoke negative emotions, most participants in both groups commented on the overall positive tone of the apps and how this was a necessary attribute to maintain their engagement over time. In addition, a strong theme was feeling cared for by the apps. A quitSTART user, who was initially very skeptical about using a smartphone app to quit smoking, commented on quitSTART’s check-ins at visit 2:

Similarly, a QuitGuide user noted:

Another strong theme for users of both apps was the importance of notifications at the second study visit. QuitGuide users wished that they had received more notifications, as they often forgot to open the app. quitSTART users often mentioned that the check-in notification feature was one of the most valued app features because it reminded them to use the app and increased their awareness of their smoking and because they appreciated the caring tone of these notifications. In fact, many quitSTART users (who received multiple notifications per day) indicated that they wished that they had received more.

In addition to noting the strengths of the apps at visit 2, a strong theme among participants in both groups was frustration at how the feedback features ultimately functioned for them. A quitSTART participant noted:

Similar sentiments were expressed by QuitGuide users:

Some of this app feedback relied on the users’ entered quit dates and could be updated by choosing a new quit date. However, only some user data reset when a new quit date is chosen, which further confuses some participants. In addition, some of the calculations were mathematically incorrect, possibly because of a bug in the app.

Many participants in both groups also expressed a desire for a cigarette tracking feature that enabled them to track cutting down, which they felt was important to frame their progress positively. Instead of tracking slips or smoke-free days, they expressed a desire for a daily cigarette tally, so they could track incremental progress toward quitting. A QuitGuide user stated:

Participants in both groups also wished that there were more sections in the apps where they could enter free-text responses to prompts (such as their moods or their triggers for smoking) instead of choosing from a prepopulated menu. As a participant noted:

Notably, most participants had little desire to connect with others on social media about their quit attempts. They worried that sharing information about quitting on social media could be detrimental if they were not successful in their quit attempts. In contrast, many suggested that the apps include a chat feature to connect with other app users. They felt that social support from others who were working on quitting, and therefore understood the challenge of quitting, could be helpful.

Participants provided opposing opinions regarding many of the remaining app features, highlighting the importance of a personalized experience for each user. Some participants planned to use only 1 or 2 app features, whereas others indicated their intention to explore all of the different features within their assigned app. Although many participants stated that they would not use certain app features, they also commonly recommended against removing these features from the apps because they thought other users might find them helpful. For example, in describing the games in quitSTART, a user stated:

These young adult users of quitSTART made no comments indicating that they thought it was designed for teens and not for them.

In this user-centered design study, we used a mixed methods approach and triangulated multiple measures to assess usability and appeal. We found that QuitGuide demonstrated greater initial ease of use and acceptability, which remained stable over time. In contrast, quitSTART demonstrated lower initial usability and acceptability, which improved over time to a level similar to that of QuitGuide. Although SUS scores indicated below average usability for both apps (average score between 68 and 70 [42,43]), the objective quantitative and qualitative usability measures provided positive indications of usability. First, objective task completion demonstrated that at least three-quarters of participants were able to complete all but one task upon first downloading the apps. Second, during the open-ended qualitative interviews, participants stated that the apps were easy to use (at both visits for QuitGuide, and primarily at the follow-up visit for quitSTART). Several other measures also suggested that quitSTART performed well and provided value to users. First, the backend administrative data analyses showed that although user engagement with QuitGuide remained low during the trial period, quitSTART users sustained a substantially higher level of engagement (discussed in detail below). Second, although none of the users were required to engage in a quit attempt during this study, 2 of the 8 quitSTART users had biologically confirmed abstinence from smoking at follow-up compared with none of the QuitGuide users. Notably, although quitSTART was designed for teens, by Visit 2 it was perceived positively based on the qualitative feedback and a high level of engagement among these young adults with SMI. Given the acceptable perceived usability at the second visit, the much higher level of engagement with quitSTART, and previous work demonstrating the importance of engagement to cessation outcomes [25-27], our findings suggest that quitSTART may be a reasonable choice for use among young adult smokers with SMI, particularly if support and coaching facilitate initial use of the app.

A strong behavioral indicator of usability and acceptability is engagement over time. One of the most striking differences we found between the apps was the participants’ engagement, or frequency of use, during the 2-week trial period. Engagement with quitSTART, 2 to 4 interactions per day that persisted steadily over the 2 weeks, was much more favorable than engagement with QuitGuide (less than 1 per day). On the basis of participant feedback, app notifications played an important role in the different use patterns of the apps. Although some participants initially voiced concern about receiving too many notifications, many ultimately felt that notifications were positive and important to their engagement with the apps. Because our participants were not required to wish to quit smoking and were not engaged in a cessation program, the level of engagement here is likely lower than what would be seen among smokers trying to quit.

Notably, although both apps contain content and features that participants deemed important and desirable during a quit attempt, the overall appeal of the apps was influenced by perceptions of the app’s tone and data tracking. Previous work [50-53] has documented the importance of positive message framing to engage tobacco users in considering a quit or reduction attempt, and our findings further support this. In addition, our participants noted the importance of a positive tone within the apps to support ongoing motivation and indicated that a negative tone could undermine their quit or reduction attempts. Users in both groups felt that the apps were overall positive and motivating and indicated that this was a key factor in their interest in using the apps.

Not surprisingly, the perceived inaccuracy of the feedback (such as the number of slips or money saved) had a significant impact on users’ overall perceptions of the apps. Some of the perceived inaccuracy was because of the apps’ reliance on the entered quit date to calculate money saved and cigarettes avoided. Our participants were not required to engage in a quit attempt for this study, and some of the perceived data inaccuracy was likely because of choosing a default date without intending to quit on that day. Although the apps may be intended for users who plan to quit abruptly, our participants expressed interest in using the apps within a reduction-to-quit framework [54]. Incorporation of features within the app to support initial smoking reduction followed by cessation may be beneficial for this population. In addition, our findings indicate that entering personal information to track progress should be unlinked from other app features such as viewing inspirational quotes; otherwise, accessing these other features could also affect the accuracy of the users’ feedback.

Compared with recently reported data among middle-aged adults with SMI who were trying to quit smoking, QuitGuide usability scores were lower among these young adult participants [41]. Among middle-aged adults with SMI, the mean SUS score for young adults was 78.4 (SD 16.5), compared with 64 (SD 18) in this study. We are not aware of other published usability studies of the QuitGuide app among young adults with SMI.

Comparison of user engagement data among studies can be challenging because of different durations of follow-up, varied measurements of app engagement (eg, app openings, days of use, and specific actions within the app), and previously demonstrated decay in the use of eHealth interventions over time [25]. Nevertheless, our QuitGuide findings appear similar to those from other studies. In the study of middle-aged adults with SMI by Vilardaga et al [41], participants used the app on 32 (SD 24.5) days during the 120-day trial period, whereas the young adults in this study used the app on 4.6 days during our 13-day study period. Bricker et al [55,56] have previously assessed QuitGuide as a comparator app for evaluating novel cessation apps among middle-aged general population smokers trying to quit. In one trial [55], QuitGuide users self-reported opening the app an average of 15 times during an 8-week study period (days of use not reported). In another study [56], backend app usage data demonstrated that QuitGuide users opened the app 9.9 times on 7.1 days during a 12-month trial period. Satisfaction with QuitGuide ranged from 45%-70% in these trials. Hebert et al [57] have also assessed QuitGuide as a comparator for a just-in-time adaptive intervention. In this pilot study, QuitGuide users opened the app an average of 9.9 times on an average of 10.6 days during a 5-week trial period. Satisfaction scores for QuitGuide were lower than for the just-in-time intervention or usual care (in-person and/or telephone counseling), with QuitGuide averaging 3.64/5 for the survey item “I believe that my treatment will help me quit smoking and stay quit.”

Although we were unable to identify other studies that assessed the acceptability, usability, or user engagement of quitSTART, our findings regarding user engagement and satisfaction with quitSTART are promising compared with those of other apps in the general population of adult tobacco users. This includes studies of SmartQuit [55], in which users opened the app an average of 37 times during the 8-week trial period (days of use not reported), and of whom 59% were satisfied with SmartQuit overall; iCanQuit [56], in which users opened the app an average of 37.5 times on an average of 24.3 days during the 12 month trial period, and more than 80% of iCanQuit users found their app useful for quitting; and Clickotine [58], in which users opened the app an average of 100.6 times during the 8-week study period (days of use not reported).

Our qualitative usability findings are similar to those of Ferron et al [14], which found that middle-aged adults with SMI noted text-heavy apps to be unappealing, had difficulty navigating more engaging apps because of abstract symbols and one-word menu labels, and had difficulty following subtle directions to use various app features. Notably, our young adult participants rapidly overcame most challenges with these design features in quitSTART after a brief coaching session and 2 weeks of independent use.

Our acceptability findings are similar to those of other evaluations of middle-aged adults with SMI [40,59]. The evaluation of an earlier version of QuitGuide (QuitPal) by Vilardaga et al [40] highlighted participants’ desire for finer-grained cigarette tracking and interactive and motivating features, as well as the importance of seeing incremental progress. Klein et al [59] found that middle-aged participants with SMI expressed the importance of social support within the app, the role of caring and positivity from the app, and concern for negative emotions related to relapse. Our findings also significantly overlap with the assessment of Struik et al [60] among general population young adult tobacco users who assessed the Crush the Crave app, including the importance of positive message framing, preference for lighter colors, and frustration that progress feedback based on the user’s quit date was not accurate.

Our study has several limitations. First, we included a small number of participants because prior research has demonstrated the adequacy of this number for identifying most usability issues [29], but our acceptability findings should be interpreted with caution. Second, participants were not required to be interested in quitting or engaging in a quit attempt for this study. App preferences may differ during planned quit attempts when engagement is likely to be higher. Although more than three-quarters of our participants reported attempting to quit or reduce their smoking during the trial period, they had not committed to cessation treatment. This may have contributed to the frustration with inaccurate feedback, which was based on the entered quit date. In addition, engagement may differ in the context of a study compared with the use outside of the study context. However, the consistency between our findings and those of previous research supports the validity of our usability and acceptability findings. Finally, our 2-week follow-up period was relatively short, and user engagement has been shown to decay with time. However, users of quitSTART sustained their use for these 2 weeks, indicating a promising level of initial usability and acceptability during that period.

Overall, we found that both NCI’s smoking cessation apps (QuitGuide and quitSTART) were usable and appealing among young adults with SMI. However, engagement with quitSTART was high, and ratings of its usability improved with time, indicating that quitSTART may be a more favorable tool than QuitGuide for young adult smokers with SMI. Our findings suggest a possible role for quitSTART during quit attempts in this group; however, clinical support or coaching may be needed to overcome initial usability issues. These findings may assist with the development and adaptation of interventions for young adults with SMI.