Smoking cessation therapists have long used the motivational interviewing (MI) talk therapy to guide clients toward positive behavioral change [1]. MI engages clients in a structured conversation that encourages them to contemplate their behavior more deeply and motivates them to change it. MI has been shown to be successful in helping clients reduce or quit their smoking habits [2], but the availability of MI-trained clinicians is limited to hospitals and medical centers, and MI therapy is usually only initiated after a smoking-related health issue occurs [3]. These restrictions make it difficult for smokers to access therapy outside of medical centers and occur too late to have a preventative effect.

Our research seeks to automate the therapist side of an MI conversation which, if successful, could broaden access to care at a population level. We have been developing a chatbot, called MIBot [4], whose purpose is to move ambivalent smokers toward the direction of quitting. MIBot is being developed by an interdisciplinary research collaboration among expert MI-trained clinicians, social scientists, and computer engineers. The initial version of the MIBot chatbot guides the client through a fairly simple MI conversation by combining scripted interactions with context-specific responses generated by natural language models, based on elements of the MI approach.

The focus of the initial version of the MIBot chatbot is on one core skill of MI: reflective listening [1], in which the chatbot provides reflections on what the client has most recently said. In general, reflections are meant to express the therapist’s current understanding of the client’s most recent response and invite the client to continue further contemplation of their behavior. Reflections can be simple or complex [1]. A simple reflection rephrases a client’s response, sending the message that the response was understood and inviting the client to continue. A complex reflection attempts to infer relevant information about the client from the client’s utterance by linking the client’s response to relevant facts or ideas. A good quality complex reflection may further infer something about the emotional state of the client through their utterance.

In a complex reflection, when these relevant facts come from a client’s earlier responses in the conversation, we call this a backward-looking complex reflection (BLCR). Preferably, a BLCR does not simply summarize all the past conversational information in order but is composed of the information that is sensible for the context. Textbox 1 shows an example of a conversation in which the final statement by the therapist is a BLCR.

The initial MIBot chatbot [4] only generates reflections using the client’s most recent utterance and does not make use of prior utterances. The ability to generate BLCRs can expand the chatbot’s options for generating context-appropriate complex reflections.

The goal of this work is to develop and evaluate a method to automatically generate BLCRs given a prior conversation. It has become possible to do this kind of generation through recent dramatically powerful advancements in natural language processing [5], and more specifically the most recent large language models (LLMs) from GPT-3.5 and later [6-8].

LLMs are language models which take text as input and generate textual output. GPT-4, an LLM introduced in March 2023, has significantly improved capability to generate text to satisfy particular requirements compared to previous LLMs [6-9]. One way to use GPT-4 is to write a prompt, which is a language-based instruction that literally tells the model the processing that is desired [9]. This processing is potentially anything that can be described in language, which is a truly remarkable, new capability that will have many applications. We describe a method for developing the prompts needed to “tell” the model to create BLCRs.

This paper is organized as follows: the Prior Work section introduces MI, GPT-4, and the relevant parts of the MIBot project that we build on. The Methods section describes the prompt developed to generate a BLCR, the specific structure of the input to GPT-4, the rating scale developed to assess when a BLCR is acceptable, the experimental procedure to test the acceptability of BLCRs generated by the prompt, and the data used to test this procedure. The Results section provides the evaluation, and the Discussion section interprets the results of the experiment and lists limitations. The Conclusions section suggests avenues for further work.

In this section, we describe the structure of the method used to generate BLCRs, the set of data we test our BLCR generation method on and how the resulting BLCRs are assessed.

Ethical standards and approval directly follow those of Brown et al [4] as per the use of the data in the experiments described in that paper. The research used to acquire that data was approved by the University of Toronto Research Ethics Board under protocol number 35567, amended June 29, 2022, and all participants provided consent before participating in the Brown et al [4] study.

In a chatbot conversation with a client, the client’s latest and previous responses, along with the questions that were asked to evoke those responses, are packaged into a text called the client message input. A set of instructions, called the BLCR prompt, tells an LLM how to generate a BLCR from the client message input. These 2 texts are used together to generate a BLCR.

The client message input (Textbox 3) consists of (1) conversation—the sequence of therapist questions and client responses up to the client’s response right before the therapist’s latest question—and (2) latest question-response—the therapist’s latest question and the client’s latest response.

The client message input is unique to each client response, and so changes on every client response. An LLM processes this input to generate a BLCR by first processing the instructions given in the BLCR prompt.

The BLCR prompt, shown in Textbox 4, consists of (1) a request to generate a BLCR meeting the standards of MI, using terms presented in the client message input (see Client Message Input section); (2) a description of a complex reflection, taken from Miller and Rollnick [1]; (3) constraints and criteria to ensure the generated text meets the criteria of a complex reflection; (4) constraints and criteria to ensure the generated text meets the criteria of a BLCR; and (5) repetition of the request to generate a BLCR, given the above constraints and criteria.

The BLCR prompt is the same regardless of the client input message used. The BLCR prompt draws upon an LLM’s implicit domain knowledge of MI [4,11], combined with a specific definition of a complex reflection, and constraints and criteria on what the output must follow to be an acceptable BLCR. For each client message input, an LLM can use the BLCR prompt’s guidelines to generate a BLCR.

The BLCR prompt was created through an iterative process. Starting with an initial description was set of rules describing a BLCR and the requirements to generate a BLCR. This initial prompt was used to generate reflections on preexisting conversational data from prior conversations. These reflections were evaluated using the scale described in the Evaluation of Quality of a BLCR section. The prompt was subsequently revised to improve the responses, and the method attempted again on another set of independent conversational data. The revisions consisted of additional constraints and guidance, written in English, to address the shortcomings of the generated reflections. This iterative process continued until a prompt of sufficiently high evaluation score of the generated reflections was was achieved. The following sections describe both the data and the scale used.

To test the BLCR prompt and client message inputs on real conversational data, 50 conversations were randomly selected from the MIBot version5.1 experiment data [4]. Each conversation consisted of the 5 MIBot core questions shown in (Textbox 2), along with their respective participant responses. As described in Brown et al [4], the participants were 50 anonymous volunteers from the Prolific platform who self-selected based on being current smokers. All 50 participants wrote their responses in text via the MIBot text-based chat interface. Multimedia Appendix 1 provides a sample conversation. Using the BLCR prompt and client message input, BLCRs would be generated for responses to Q3, Q4, and Q5 for each conversation, giving a total of 150 candidate BLCRs to assess.

A rating scale was developed to numerically evaluate the quality of a BLCR. This scale allows one to determine whether a BLCR is acceptable, that is, it meets the definition of a BLCR described in the Prior Work section.

The BLCR rating scale (Textbox 5) is an ordinal scale where higher number ratings successively include and build upon lower number ratings. If a BLCR achieves a rating of 3, this means it meets the criteria of 1 (referencing a client’s latest response), 2 (referencing previous information in the conversation), and 3 (makes an inference about the client using present and past information). Satisfying these 3 requirements meets the definition of a BLCR as defined in the Prior Work section; therefore, we call any BLCRs rated 3 or greater acceptable BLCRs. A further rating of 4 is included to meet the preference for a “good” BLCR, which does not summarize the previous contents of the conversation, an optional condition that was deemed useful for indicating an unambiguous BLCR that exceeds the minimum acceptability requirements.

This section reports the fraction of the BLCRs generated using the evaluation method described in the Methods section that were deemed acceptable by each of the 3 human raters. The first section reports the percentage of accepted BLCRs between the 3 raters and between the 3 questions, along with a breakdown of the frequency of ranking scores per question and rater. The second section reports the interrater reliability between 3 pairs of the 3 raters (rater 1 and rater 2, rater 1 and rater 3, and rater 2 and rater 3) using percent agreement, with a brief discussion on the κ results.

Table 2 displays the percentage of BLCRs meeting the BLCR rating criteria as acceptable (BLCR rating of 3 or greater) broken down by the rater and the question. Table 3 displays the frequency of rating ranks broken down by question and by rater.

Table 2 breaks down the percent of acceptable BLCRs by rater and question, and the total column indicates the percent of BLCRs scored acceptable across all 150 responses by a single rater. The percentages in parentheses indicate the SD of the acceptability percentage.

The combination of high acceptability (Table 2) and high frequency of “4” ratings (Table 3) indicates that the majority of BLCRs generated by this method were considered “good” among all 3 raters. This is an indication that the LLM GPT-4 is highly capable of generating a BLCR. Multimedia Appendices 2 and 3 graph the frequencies of rating by question and rater, with both indicating a large skew toward “4” ratings.

To assess the agreement of the results provided in Tables 1-3, Table 4 displays the percent agreement and Cohen κ for each rater pair. All 3 raters agreed on results at least 80% of the time.

Altogether, the combination of high “4” frequency and a rating agreement of 80% and above indicates that this BLCR generation method can be expected to produce “good” BLCRs in the large majority of cases. In comparison, the κ values (Table 3) indicated weak to fair agreement between all 3 pairs of raters, based on standard interpretation criteria of κ. The discrepancy between high percentage agreement and weak to fair κ may be due to the majority of BLCRs being rated “4” by all 3 raters. The lack of contrastive negative examples (very few generated BLCRs that were rated 0, 1, or 2) skews the calculation of κ toward treating the labeling of widespread agreement as random chance. Therefore, percent agreement is thought to be a more realistic assessment of effectiveness in this context.

Multimedia Appendix 1 contains an example of a real conversation from Brown et al [4], with Brown et al’s [4] reflections (labeled MIBot [data]) and BLCRs generated by this paper’s method (labeled MIBot [BLCR]) below those reflections. Overall, the BLCRs generated successfully iterate on Brown et al’s [4] provided reflections by better incorporating direct reflections on responses and linkages to previous responses to make inferences. A high-quality MI reflection would further infer about the emotional state of the client, and while the generated BLCRs are able to make rudimentary inferences about the mental state of the client (“it seems that…”), more work may be necessary to turn these inferences into those of emotional states. The high percentage of accepted BLCRs shows promise in prompt-based methods being an effective technique for MIBot to generate complex reflections that incorporate information from the past.

The prompt-based BLCR generation method is restricted to MI conversations for smoking cessation and has only been tested in the context of 5-question MIBot conversations. Beyond this scope, this work may not generalize to other MI smoking cessation therapeutic contexts without changes to the prompt. However, the structure of the prompt itself is not specific to the data or the situation. The prompt can in theory be modified to remove references to smoking cessation and replace these with references to other domains, potentially offering a degree of domain generalizability across different subjects of MI therapy beyond smoking cessation. GPT-4 was the LLM model used in this work, but this method is applicable to any LLM model in theory. Newer LLM models, including future GPT models, may provide more robust results.

This paper presented a method to use an LLM-based prompt to generate BLCRs for a version of MIBot’s MI smoking cessation conversation. It provided a definition of a BLCR, a prompt used to generate BLCRs, and a BLCR rating scale to assess whether a BLCR is acceptable. We found that 88% (n=150) of the generated BLCRs were deemed acceptable. This paper extends the work of Brown et al [4] by providing a method to generate complex reflections that incorporate information from earlier in the conversation, and uses GPT-4’s strong text-generation capability rather than GPT-2.

Future work may build upon the definitions and methods introduced by this paper in three ways. First, the definition of a BLCR and the BLCR rating scale may be further refined to provide an accurate conceptual model of what the BLCR is trying to capture in a MI conversation. Second, the BLCR’s prompt method can be adjusted to different MI therapeutic contexts beyond smoking cessation or refined to be more accurate for the smoking cessation context. Finally, the BLCR prompt method can be incorporated into MIBot, and its generated BLCRs can be assessed qualitatively and quantitatively in live experimental conversations.