JMIR Publications

JMIR Mental Health

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 03.10.17 in Vol 4, No 4 (2017): Oct-Dec

This paper is in the following e-collection/theme issue:

    Original Paper

    Development of a Questionnaire to Measure the Attitudes of Laypeople, Physicians, and Psychotherapists Toward Telemedicine in Mental Health

    Corresponding Author:

    Peter Tonn, Dr med

    Institute of Applied Research in Neuropsychiatry

    Neuropsychiatric Center Hamburg-Altona

    Stresemannstr. 23

    Hamburg, 22769

    Germany

    Phone: +49 40 533 0 738 0

    Fax:+49 40 533 0 738 20

    Email:


    ABSTRACT

    Background: In the field of psychiatry and psychotherapy, there are now a growing number of Web-based interventions, mobile phone apps, or treatments that are available via remote transmission screen worldwide. Many of these interventions have been shown to be effective in studies but still find little use in everyday therapeutic work. However, it is important that attitude and expectation toward this treatment are generally examined, because these factors have an important effect on the efficacy of the treatment. To measure the general attitude of the users and prescribers toward telemedicine, which may include, for instance, Web-based interventions or interventions through mobile phone apps, there are a small number of extensive tests. The results of studies based on small groups of patients have been published too, but there is no useful short screening tool to give an insight into the general population’s attitude. We have developed a screening instrument that examines such attitude through a few graded questions.

    Objective: This study aimed to explore the Attitude toward Telemedicine in Psychiatry and Psychotherapy (ATiPP) and to evaluate the results of general population and some subgroups.

    Methods: In a three-step process, the questionnaire, which is available in three versions (laypeople, physicians, and psychologists), was developed. Afterwards, it was evaluated by four groups: population-representative laypeople, outpatients in different faculties, physicians, and psychotherapists.

    Results: The results were evaluated from a total of 1554 questionnaires. The sample population included 1000 laypeople, 455 outpatients, 62 physicians, and 37 psychotherapists. The reliability of all three versions of the questionnaire seemed good, as indicated by the Cronbach alpha values of .849 (the laypeople group), .80 (the outpatients’ group), .827 (the physicians’ group), and .855 (the psychotherapists’ group).

    Conclusions: The ATiPP was found to be useful and reliable for measuring the attitudes toward the Web-based interventions in psychiatry and psychotherapy and should be used in different studies in this field in the future to evaluate and reflect the attitude of the participants.

    JMIR Ment Health 2017;4(4):e39

    doi:10.2196/mental.6802

    KEYWORDS



    Introduction

    There are many options currently available for telemedical contact or communication between the patients and the professionals. These include Web-based interventions, mobile phone apps, and remote screen calls. In psychiatry and psychotherapy, in addition to the conventional long-distance contact via phone calls or mail, the Web-based interventions (e-mental health) and mobile apps (m-mental health) are increasingly becoming important areas of care for the patients. Mental health, as a special part of the so-called telemedicine, covers two important areas. The first involves the information and education about the psychiatric disorders and psychosocial distress, and the second pertains to diagnostic and treatment tools. Many different Web-based services, in the form of email, social media, and other avenues, are accessible over the Internet almost worldwide. Also, a number of telemedical apps are available, especially via the Internet, for the somatic illnesses [1]. The everyday use (outside clinical trials) of Web-based tools for the diagnosis and treatment of mental illness, however, is not very popular yet [2,3]. For example, in the United Kingdom, there are only two computerized interventions recommended for the clinical treatment of psychiatric diseases (one against depression and another against panic and phobia) [4-6]. Phone calls or emails are regularly, though not frequently, used as the media of communication between the therapists and the patients in psychiatry or psychotherapy. Web-based interventions, however, are seldom used. Hence, it seems that there are barriers to the implementation of effective Web-based interventions that can help identify and overcome the issues related to mental health. The potential users recognize the rational advantages of the Web-based interventions such as anonymity, convenience regarding time, location, and the ability to do without a structured setting [7,8]. At the same time, the dropout rate among the users of computerized interventions of cognitive behavior therapy (CBT) is almost twice as high as compared with the dropout rate of face-to-face CBT [9]. The willingness to participate in a study on a new therapeutic concept seems to be good, but its application and use in the daily practice of therapy is very limited [10]. In a paper on the question of the meaningfulness of computerized interventions, only 22% of the participants indicated that the symptoms can be effectively improved [8]. Other studies show a negative attitude of the patients toward Web-based interventions, too. They have shown a low expectation of patients regarding whether their psychological complaints can be improved by computerized interventions, even to the conclusion that the patients do not want to use Web-based interventions in the future [11]. The current surveys on the attitudes toward and expectations from Web-based interventions are, in part, too comprehensive for a screening tool. Also, they do not reflect the attitude of the general population, focusing instead on selected populations (certain patient groups, Internet users with high affinity to the medium, etc). Another aspect is their outdated status, which renders them unable to represent the current possibilities that the Internet offers today.

    Nevertheless, a considerable number of interventions via the Internet are actually already available, for example, to support the detection and treatment of posttraumatic stress disorder [4,12,13], anxiety disorders [14], or depression. In e-mental health, the currently available Web-based tools differ, among other things, regarding their depth of information—some only provide information about the disorder (psychoeducative), whereas others give different instructions for self-help (self-guided) or give some guidance from a psychotherapist about helpful activity (guided help) [1,15-17]. A few Web-based interventions allow additional contact with a psychotherapist via video call (Web-based counseling) or face-to-face care (blended) [18]. It is foreseeable that the Web-based interventions will play a growing role in all areas of medicine, even in psychiatry and psychotherapy, in the future [19]. However, the everyday use of the Web-based interventions does not correspond to the optimum use of the available resources. Also, the efficacy of the Web-based interventions does not lead to their use on a larger scale [10]. Therefore, it is necessary to find out the background of the probable users (both patients and the physicians or psychotherapists) to integrate this information into the development and distribution of new apps [20-27]. Furthermore, it also appears necessary, especially in the studies on efficacy and effectiveness, to examine the expectations of the users from such tools as a positive complement to the existing therapeutic spectrum. This is also important because, since the 1970s, it has been possible to show repeatedly that the patients’ expectations from and belief in the credibility of a therapeutic medium or process can significantly influence its effectiveness and efficacy [28]. Therefore, in the studies on the effectiveness of the Web-based interventions, especially when the participants are recruited from a website or other Web-based media, the sample may have been biased. This can be quantified at least by using a simple screening tool, which can show the data of the general population as comparative values.

    To our knowledge, there is no short questionnaire that measures the concept of a favorable attitude toward telemedicine in psychiatry and psychotherapy, especially, but not exclusively, e-mental health. Only some difficult and complex measurements are published yet [7]. Therefore, we developed a questionnaire that can be used to evaluate the expectations and the attitudes of the users of such interventions, and we seek to publish the data of a large sample of the general population and some special groups (patients of different faculties, physicians, and psychotherapists) to present the comparative values.


    Methods

    To develop the questionnaire, Attitudes toward Telemedicine in Psychiatry and Psychotherapy (ATiPP), we undertook a three-step process. We started by generating a set of items reflecting the attitude toward and the expectations from telemedicine in general and particularly, the use of telemedicine in mental health. We used the experience of a psychotherapeutic team and psychiatric consultants to find as many statements as possible, as well as depending on the available literature. For example, one statement was, “In general, telemedicine is a good addition to medical services.” The initial 15-item questionnaire was then used in a discussion process with the experts not otherwise involved in the development of the scale. The notes and comments that arose independently were included in the further development of the questionnaire. This was done in a modified Delphi process, which means that the experts discussed and developed the questions over several sessions [29]. In the modified Delphi technique, the panelists begin with a set of items to rate according to importance, rather than with an open-ended questionnaire. Here, these items were selected by the study team, drawing from various sources, including a literature review and interviews with content experts, patients, and the technical and medical specialists for Web-based interventions. The primary advantage of using this modification is that it typically improves the initial response rate [30]. The questions are posed in a series of rounds until a consensus is achieved or until it is obvious that the future rounds will not provide additional information. At each round after the first, the experts are provided with a feedback of anonymous comments from the panelists in the round before. The questions evaluated as substantive duplicates or evaluated as not relevant were excluded after the discussion when we developed the questionnaire. Then in a second step, we condensed the items to an 8-item questionnaire (from the initial 15-item questionnaire). A 5-point Likert scale was used with anchors ranging from 1=“I strongly agree” to 5=“I strongly disagree.” At this point, we developed three versions of the questionnaire so that the exact wording was matched to the individual function of the respondents. Consequently, the laypeople were asked what they prefer for themselves, whereas the practitioners were asked what they want to offer their patients.

    We explain the questionnaire with the following sentences (ie, questionnaire for the laypeople):

    The use of telemedicine as an online service, by phone call or, as a smartphone application, is lately being discussed more and more intensively. This is one of the consequences of improved technical opportunities on the one hand and scarcer resources on the other hand.
    While radiology or dermatology are already using telemedicine for the purpose of image transmission for X-rays, CT scan, or skin photos, the use of telemedicine in the area of psychiatry and psychotherapy—despite some pilot projects worldwide—is still, largely, an uncharted territory.
    In the context of a scientific study on psychiatric care research, we are investigating the attitudes of the users, the patients, and the referring physicians toward the telemedical services.
    With the following questions, we would like to determine how you assess psychiatric or psychotherapeutic care via a telemedical offer, ie, via the Internet, email, telephone, or via a smartphone application.
    We are also interested in whether you would use such an offer yourself.

    Table 1 shows the 8 items of the three different versions of the questionnaire. The differences of the question texts are essentially explained by the functions of the respondents. Thus, the psychotherapists are specifically asked whether they consider telemedicine to be helpful in somatic medicine because the Delphi process in developing the questionnaire showed that the psychotherapists did not reflect the somatic applications of telemedicine in the vast majority. Also in Question 6, the differences arise from the fact that the psychotherapists are in contact with the patients who are or have already been in a face-to-face therapy, that is, only those who have already been treated or who are waiting for treatment are affected, whereas in the case of the physicians, the patients were treated with many diagnoses but usually not in a psychotherapeutic setting there. In Question 3, we asked the patients for a successful treatment in colloquial speech, while we asked the same from the physicians and psychotherapists in professional wording.

    The resulting questionnaire was then used in a study with four groups of participants—the physicians, the psychotherapists, the outpatients in the waiting areas of various medical disciplines, and a representative population sample in a telephone interview.

    The population sample was based on the telephone number ranges provided by the German Federal Network Agency. This number range includes all telephone numbers in the Federal Republic of Germany, including mobile telephone numbers. We used the nationwide sample of telephone numbers. A nationwide list of the German Market Research Association (ADM) was used for the telephone survey. The connection data are presorted here according to various aspects (eg, regions) and can be appropriately used in a representative sample. As the share of households that do not publish their telephone numbers is increasing steadily, the ADM telephone sample contains both registered and generated numbers. The generation of numbers was done with a process similar to Brick [31]. In this study, we collected 50.00% (500/1000) of the general population with the generated numbers and 50.00% (500/1000) with the published numbers. This procedure ensured that both the regional distribution of the calls as well as the allocation to age groups, etc, corresponded to the population distribution in Germany; thus, a representative study sample could be collected.

    The group of patients was selected from four outpatient clinics—two general practitioners’ practices, a gynecology practice, and a center for neurology. The physicians and the psychotherapists were selected through a Germany-wide mailing action. We randomized 200 physicians from different disciplines and 200 psychotherapists from the nationwide database of the physicians and therapists by the means of a computer-generated randomization. We distributed the questionnaire as a printed version. The return was free and anonymous, and no reward was given.

    We analyzed the data from the measurement by testing the reliability. We had expected the items to be presented in a graduated form as one dimension—acceptance or attitude. In particular, we assumed that the questions are in a relationship with each other, and we examined whether the individual questions are replaceable or dispensable.

    Table 1. The 8 items of the Attitude toward Telemedicine in Psychiatry and Psychotherapy (ATiPP) questionnaire.
    View this table

    All statistical data were analyzed with the open source program R, version 2.2.4 (R-Core-Team).

    The chair of the Ethical Commission of the Board of Physicians (Institutional Review Board) in Hamburg, Germany, did not consider ethical approval to be necessary because of the anonymous type of survey of the participants.


    Results

    The participants of the first survey answering the questionnaire as a population-representative random sample were 51.40% (514/1000) female. The age was distributed according to the proportion in the population and divided into 10-year clusters. The smallest age group with 12.60% ranged from 16 to 25 years, followed by 14.30% (143/1000) from 26 to 35 years, 15.00% (150/1000) from 36 to 45 years, 17.70% (177/1000) from 46 to 55 years, 16.00% (160/1000) from 56 to 65 years, and 24.40% (244/1000) for the age group of 65 years and older.

    The Cronbach alpha value for the 8 items reached the conventional standards for scale reliability (alpha=.849), and no item reduction seemed meaningful. We used the questionnaire version for the laypeople in this particular study.

    In the next part of the study, we included a sample of outpatients in the waiting rooms of different physicians—the general practitioners, the gynecologists, and the neurologists. Here, 324 participants (of 624 asked) completed the questionnaire; 65.7% (213/324) of them were female. The age showed a distribution of 13.7% (48/324) in the age group of 16 to 25 years, 26.9% (94/324) for 26 to 35 years, 17.4% (61/324) for 36 to 45 years, 17.4% (61/324) for 46 to 55 years, 11.1% (39/324) for 56 to 65 years, and 13.4% (47/324) for those aged 65 years and older.

    The Cronbach alpha value for the 8 items reached the conventional standards for scale reliability (alpha=.80), and no item reduction seemed meaningful. We also used the questionnaire version for laypeople. Table 2 shows the results of the study with the mean (M) and standard deviations (SD). Due to the low sample size of the psychotherapists, we added the physicians and the psychotherapists here to the category of professionals.

    Table 2. The mean and the SD of the items and the overall measures. The questions are coded from “1=strongly agree” to “5=strongly disagree,” and so, the lower values are in better agreement with the question whereas the higher values indicate more disagreement. The total scale is the mean of the 8 single-item values.
    View this table
    Table 3. The scale reliability with single-item deficient (Cronbach alpha deleted for one item) and selectivity coefficient (corrected part-whole-correlation).
    View this table

    The third part of our study was directed toward the physicians. The group size was 92, with 47% (43/92) of the physicians being female (and the distribution of age showing 1% for those under 36 years (1/92), 11% (10/92) for those in the age range of 36 to 45 years, 43% (39/92) for 46 to 55 years, and 44% (41/92) for those aged above 55 years. We used the questionnaire version for the physicians in this part of the study. The Cronbach alpha value was .827, and so, no correction of the questionnaire was necessary at this point.

    In the last part of the study, we surveyed the psychotherapists. This group consisted of 36 participants, with 67% female and the distribution of age revealing 5% (2/36) to be less than 36 years, 5% between 36 and 45 years (2/36), 36% (13/36) in the 46 to 55 years age range, and 53% (19/36) being older than 55 years. Reliability was good; we found a Cronbach alpha value of .855. Here, we used the questionnaire version for the psychotherapists.

    The comparison of the 3 versions of the questionnaire shows no relevant differences; the reliability is nearly similar. Only in the questionnaire for the psychotherapists, item 8 (Therapy through the Internet services/telemedicine for mental illness can only work effectively with live contacts with a therapist through video calling and email/chat.) shows a selectivity under 0.1 and must, therefore, be questioned.

    The analysis of the Cronbach alpha values by deleting the singular items and selectivity are shown in Table 3.

    Question 4 shows the clearest significance (P=0.001) among the laity in the positive assessment; also in the Cronbach alpha a good consistency is confirmed here.

    A presentation and comparison of the results of all aspects of the study will be done and published.


    Discussion

    We developed an 8-item questionnaire with three versions that cover the important aspects of the attitudes and expectations of the laypeople, the physicians, and the psychotherapists toward the telemedical interventions in psychiatry and psychotherapy (eg, e-mental health, Web-based interventions, and phone interventions). This is the first time that a really short and clear questionnaire has been constructed to look at the attitudes toward Web-based interventions and telemedicine in mental health. This was important because of the influence that the patients’ expectations and attitudes toward a diagnostic and therapeutic process or tool have on the efficacy and the results of these processes or tools.

    Earlier studies have shown that a variety of telemedical interventions, especially Web-based interventions, are available in many countries. And it seems that the number, especially of Web-based interventions or mobile phone apps, is growing daily. However, despite the high demand for psychological support, these resources are only used to a limited extent in the everyday therapeutic work. It is unclear whether this is related to the attitude toward Web-based interventions and other telemedical services. However, previous work has already shown, for example, with the help of the credibility and expectancy questionnaire, that therapeutic success is closely related to the attitude toward therapy. So far, such investigations with regard to the Internet-based interventions have been carried out with some limitations. Schröder et al [32] had examined the patients and the psychotherapists in the context of telemedical care, but there has not been any recruitment of a sample of the general population. Gun et al [33] have also interrogated the disorder-specific groups, even in a large sample. Musiat et al [7] and Wangberg et al [34] have attempted to investigate a non-disorder-specific group, but as a population is only accessible by social media, those people questioned can be estimated as being open-minded toward the Internet. There is no doubt that a similar bias is to be feared in the evaluation of the therapeutic approaches on the Internet—especially if those who are positively opposed to this medium will take part in these studies. To achieve a broad effect and to use the Web-based interventions not only as niche products, the patients who are not “early adopters” must also be reached. Also, the (particularly) positive expectation of the participants in the studies—which may differ significantly from the general population—must be considered. Checking this possible bias will be helpful for examining the attitudes of the participants and the patients in a study and for a comparison with the general population. This can be done with a questionnaire such as the ATiPP—because it is short and quick, and there are growing numbers of different participants and groups we had checked (and will publish the data in the near future).

    Actually, we had a sufficient number of participants among the general population and the patients, but the physicians and the psychotherapists were also included in the first evaluation. The comparison of the three versions indicates that the same contents are measured, reflecting the positions of the participants. The collection and analysis of more data from the participants using the ATiPP is currently underway; we are testing the results in other groups and with other subjects. In addition, the questionnaires published so far, such as Credibility/Expectancy Questionnaire (CEQ) [28], will be checked against the ATiPP in smaller groups. In principle, we assume that the attitudes toward the use of Web-based interventions and other telemedical services can be adequately examined and recorded with the ATiPP.

    However, we are conducting a study to create pre- and postdata using the questionnaire in a sample of laypeople and professionals before and after participating in a three-lesson e-learning session with a focus on e-mental health interventions. We expect a further confirmation that the questionnaire may also reflect the developments in the attitudes toward and expectations regarding e-mental health. There are currently some limitations. Thus, there is a lack of a factor analysis, which we have prepared but have not yet concluded with regard to the evaluation of the subgroups. It is, thus, not clear whether the questionnaire actually measures a dimension, as the reliability values suggest, or whether the two dimensions (credibility and expectancy) are also collectively taken as an attitude in ATiPP, which the CEQ as somewhat a comparable instrument might at least suggest.

    In conclusion, we recommend the use of the questionnaire to measure the attitude toward and expectations regarding e-mental health. The growing number of Web-based interventions needs a growing knowledge of the potential users—the laypeople as well as the physicians and the psychotherapists. They must be informed about the bugs and hints as also the positive effects in order for them to accept the Web-based interventions and to understand the strengths and limitations of e-mental health. Only if the attitudes toward the Web-based interventions become more positive and the expectations of the users correspond to the actual good results of the interventions, a therapeutic use of telemedicine, especially e-mental health and m-mental health, will occupy a wider space in everyday work and life. This could be achieved through training programs, and the effect of such programs can be evaluated by the use of our questionnaire.

    Conflicts of Interest

    None declared.

    References

    1. Eysenbach G, Wyatt J. Using the internet for surveys and health research. J Med Internet Res 2002 Nov;4(2):E13 [FREE Full text] [CrossRef] [Medline]
    2. Kurki M, Koivunen M, Anttila M, Hätönen H, Välimäki M. Usefulness of internet in adolescent mental health outpatient care. J Psychiatr Ment Health Nurs 2011 Apr;18(3):265-273. [CrossRef] [Medline]
    3. Moritz S, Van QN, Lincoln TM, Köther U, Andreou C. Can we trust the internet to measure psychotic symptoms? Schizophr Res Treatment 2013;2013:457010 [FREE Full text] [CrossRef] [Medline]
    4. Lange A, Rietdijk D, Hudcovicova M, van de Ven J, Schrieken B, Emmelkamp PMG. Interapy: a controlled randomized trial of the standardized treatment of posttraumatic stress through the internet. J Consult Clin Psychol 2003 Oct;71(5):901-909. [CrossRef] [Medline]
    5. Depression: the treatment and management of depression in adults. In: Depression: The NICE Guideline on the Treatment and Management of Depression in Adults (Updated Edition). London: RCPsych Publications; 2010.
    6. General anxiety disorder and panic disorder in adults - NICE GC 113. London: The British Psychological Society and The Royal College of Psychiatrists; Oct 11, 2011.
    7. Musiat P, Goldstone P, Tarrier N. Understanding the acceptability of e-mental health - attitudes and expectations towards computerized self-help treatments for mental health problems. BMC Psychiatry 2014 Apr 11:109. [CrossRef]
    8. Wootton BM, Titov N, Dear BF, Spence J, Kemp A. The acceptability of internet-based treatment and characteristics of an adult sample with obsessive compulsive disorder: an Internet survey. PLoS One 2011;6(6):e20548 [FREE Full text] [CrossRef] [Medline]
    9. Melville KM, Casey LM, Kavanagh DJ. Dropout from internet-based treatment for psychological disorders. Br J Clin Psychol 2010 Nov;49(Pt 4):455-471. [CrossRef] [Medline]
    10. Waller R, Gilbody S. Barriers to the uptake of computerized cognitive behavioural therapy: a systematic review of the quantitative and qualitative evidence. Psychol Med 2009 May;39(5):705-712. [CrossRef] [Medline]
    11. Carper MM, McHugh RK, Barlow DH. The dissemination of computer-based psychological treatment: a preliminary analysis of patient and clinician perceptions. Adm Policy Ment Health 2013 Mar;40(2):87-95. [CrossRef] [Medline]
    12. Ruwaard J, Lange A, Schrieken B, Dolan CV, Emmelkamp P. The effectiveness of online cognitive behavioral treatment in routine clinical practice. PLoS One 2012;7(7):e40089 [FREE Full text] [CrossRef] [Medline]
    13. Van Voorhees BW, Gollan J, Fogel J. Pilot study of Internet-based early intervention for combat-related mental distress. J Rehabil Res Dev 2012;49(8):1175-1190 [FREE Full text] [Medline]
    14. Craske MG, Rose RD, Lang A, Welch SS, Campbell-Sills L, Sullivan G, et al. Computer-assisted delivery of cognitive behavioral therapy for anxiety disorders in primary-care settings. Depress Anxiety 2009 Mar;26(3):235-242 [FREE Full text] [CrossRef] [Medline]
    15. Brijnath B, Protheroe J, Mahtani KR, Antoniades J. Do web-based mental health literacy interventions improve the mental health literacy of adult consumers? results from a systematic review. J Med Internet Res 2016 Jun 20;18(6):e165 [FREE Full text] [CrossRef] [Medline]
    16. Morrison L, Moss-Morris R, Michie S, Yardley L. Optimizing engagement with internet-based health behaviour change interventions: comparison of self-assessment with and without tailored feedback using a mixed methods approach. Br J Health Psychol 2014 Nov;19(4):839-855 [FREE Full text] [CrossRef] [Medline]
    17. Zabinski MF, Wilfley DE, Calfas KJ, Winzelberg AJ, Taylor CB. An interactive psychoeducational intervention for women at risk of developing an eating disorder. J Consult Clin Psychol 2004 Oct;72(5):914-919. [CrossRef] [Medline]
    18. Månsson KN, Skagius RE, Gervind E, Dahlin M, Andersson G. Development and initial evaluation of an internet-based support system for face-to-face cognitive behavior therapy: a proof of concept study. J Med Internet Res 2013;15(12):e280 [FREE Full text] [CrossRef] [Medline]
    19. Ben-Zeev D, Scherer EA, Wang R, Xie H, Campbell AT. Next-generation psychiatric assessment: using smartphone sensors to monitor behavior and mental health. Psychiatr Rehabil J 2015 Sep;38(3):218-226 [FREE Full text] [CrossRef] [Medline]
    20. Andersson G. The promise and pitfalls of the internet for cognitive behavioral therapy. BMC Med 2010;8:82 [FREE Full text] [CrossRef] [Medline]
    21. Ebert DD, Berking M, Cuijpers P, Lehr D, Pörtner M, Baumeister H. Increasing the acceptance of internet-based mental health interventions in primary care patients with depressive symptoms. A randomized controlled trial. J Affect Disord 2015 May 1;176:9-17. [CrossRef] [Medline]
    22. Edwards L, Thomas C, Gregory A, Yardley L, O'Cathain A, Montgomery AA, et al. Are people with chronic diseases interested in using telehealth? a cross-sectional postal survey. J Med Internet Res 2014;16(5):e123 [FREE Full text] [CrossRef] [Medline]
    23. Handley T, Perkins D, Kay-Lambkin F, Lewin T, Kelly B. Familiarity with and intentions to use internet-delivered mental health treatments among older rural adults. Aging Ment Health 2015;19(11):989-996. [CrossRef] [Medline]
    24. Johnston L, Dear BF, Gandy M, Fogliati VJ, Kayrouz R, Sheehan J, et al. Exploring the efficacy and acceptability of internet-delivered cognitive behavioural therapy for young adults with anxiety and depression: an open trial. Aust N Z J Psychiatry 2014 Sep;48(9):819-827. [CrossRef] [Medline]
    25. Mewton L, Sachdev PS, Andrews G. A naturalistic study of the acceptability and effectiveness of internet-delivered cognitive behavioural therapy for psychiatric disorders in older australians. PLoS One 2013 Aug;8(8):e71825 [FREE Full text] [CrossRef] [Medline]
    26. Oh E, Jorm AF, Wright A. Perceived helpfulness of websites for mental health information: a national survey of young Australians. Soc Psychiatry Psychiatr Epidemiol 2009 Apr;44(4):293-299. [CrossRef] [Medline]
    27. Schneider J, Sarrami FP, Grime P, Thornicroft G. Acceptability of online self-help to people with depression: users' views of MoodGYM versus informational websites. J Med Internet Res 2014 Mar;16(3):e90 [FREE Full text] [CrossRef] [Medline]
    28. Devilly GJ, Borkovec TD. Psychometric properties of the credibility/expectancy questionnaire. J Behav Ther Exp Psychiatry 2000 Jun;31(2):73-86. [Medline]
    29. Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CF, Askham J, et al. Consensus development methods, and their use in clinical guideline development. Health Technol Assessment 1998 Mar 01;2(3):1-88. [CrossRef] [Medline]
    30. Custer R, Scarcella J, Stewart B. The modified delphi technique - a rotational modification. JCTE 1999 May 01;15(2):8. [CrossRef]
    31. Brick JM. Random digit-dialing. In: Lavrakas P, editor. Encyclopedia of Survey Research Methods. Thousand Oaks, CA: Sage Publications, Inc; 2008.
    32. Schröder J, Sautier L, Kriston L, Berger T, Meyer B, Späth C, et al. Development of a questionnaire measuring attitudes towards psychological online interventions-the APOI. J Affect Disord 2015 Nov 15;187:136-141. [CrossRef] [Medline]
    33. Gun SY, Titov N, Andrews G. Acceptability of Internet treatment of anxiety and depression. Australas Psychiatry 2011 Jun;19(3):259-264. [CrossRef] [Medline]
    34. Wangberg SC, Gammon D, Spitznogle K. In the eyes of the beholder: exploring psychologists' attitudes towards and use of e-therapy in Norway. Cyberpsychol Behav 2007 Jun;10(3):418-423. [CrossRef] [Medline]


    Abbreviations

    ATiPP: Attitude toward Telemedicine in Psychiatry and Psychotherapy
    CBT: cognitive behavior therapy
    CEQ: Credibility/Expectancy Questionnaire
    SD: standard deviation


    Edited by J Torous; submitted 13.10.16; peer-reviewed by S Chan, S Moritz, D Bakker, A Baumel; comments to author 19.11.16; revised version received 11.04.17; accepted 18.08.17; published 03.10.17

    ©Peter Tonn, Silja Christin Reuter, Isabelle Kuchler, Britta Reinke, Lena Hinkelmann, Saskia Stöckigt, Hanna Siemoneit, Nina Schulze. Originally published in JMIR Mental Health (http://mental.jmir.org), 03.10.2017.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on http://mental.jmir.org/, as well as this copyright and license information must be included.