Using standardized patients to train telephone counselors for a clinical trial

Background Standardized Patients (SPs) are actors trained to portray health care patients during the training and assessment of health care providers. This paper describes the methods and costs associated with using SPs to evaluate the skills of telephone counselors working on a clinical trial that evaluated a telephone smoking cessation program tailored for smokers using Department of Veterans Affairs mental health clinics. Findings Conducting the SP exercises required five main steps: (1) Write a SP case description detailing patient demographics, demeanor, clinical symptoms and history, and instructions on how to respond to counseling, (2) Identify, select and train actors to portray the SP cases; (3) Conduct audio-taped counseling encounters between the SPs and counselors, (4) Rate the counselors on their core counseling competencies, (5) Provide feedback to counselors. The SPs and study supervisors reported that the checklist was easy to use when rating the counselors. Counselors reported that the SP encounters were realistic and helpful for practicing their clinical work and for building self-efficacy for working with real patients. The labor costs of developing two SP cases and training two SP actors was approximately $1,475. The per-session labor cost of conducting a 1-hour counseling session between one SP and one counselor was approximately $314. Conclusions Using SPs to train telephone counselors working on a clinical trial was feasible and offered training benefits beyond those provided by didactic instruction and role plays. Our research group is now routinely using SPs for the training of incoming telephone counselors.


Background
Standardized Patients (SPs) are actors trained to consistently portray health care patients based on well-developed clinical scenarios [1,2]. SPs are often used in the training and assessment of clinicians and have been used during the implementation of clinical trials to improve study procedures prior to working with real patients [3][4][5]. This paper describes the methods and costs associated with using SPs to evaluate the skills of telephone counselors working on a study that evaluated a telephone smoking cessation program for smokers using Department of Veterans Affairs (VA) mental health clinics. The goal of the parent study was to evaluate the effectiveness of an intensive telephone smoking cessation counseling program for VA mental health patients that included behavioral change counseling and problem solving therapy approaches [6][7][8].

Methods
The parent study received approval from the Institutional Review Boards and Research and Development Committees at participating VA facilities. The study employed two counselors who underwent 15 hours of initial protocol training using didactic instruction and role-plays. After completing lectures and role-plays, the counselors completed two SP encounters. Developing and conducting the SP encounters required five main steps (Table 1).
Step 1 -Develop the SP cases Each counselor was to complete two SP encounters, so we developed two SP cases using our affiliation with New York University School of Medicine (NYUSoM). NYUSoM has a bank of SP cases, about a third of which focus on promoting behavior change. From NYUSoM's case bank we selected two cases to adapt for our study. The first was a smoking cessation case, to which we incorporated a history of depression. The second was a post-traumatic stress disorder (PTSD) case, to which we incorporated a history of smoking. To further adapt the cases for a VA population, we made one case a Caucasian male in his 20s who served in Iraq and the other case an African American male in his 60s who served in Vietnam. Table 2 provides an overview of case #2. The study's clinical supervisor, a VA clinical psychologist, wrote an extensive smoking, military, and mental health history for each case, and a careful description of how to respond to counseling questions, counselor tone and demeanor, and the counselor's proposed cessation treatment plans.
Step 2 -Identify, select and train standardized patient actors Because our study population was Veterans with mental health disorders, playing a study participant would be emotionally demanding and challenging [9]. Therefore, we employed professional SP actors to accurately convey emotions and behavioral symptoms common in mental health patients. To locate SPs, we used NYUSoM's pool of experienced SPs. Following recommendations in the literature for SP recruitment [2], we contacted actors who had experience playing the two cases adapted for the study and who matched the gender, age and racial demographics of the cases. After hiring two SPs, we emailed the assigned cases to each SP. The SPs reviewed their case prior to an in-person group training session with the study's project director (ESR) and one of the study's co-investigators (CG) who has SP training experience. The in-person training lasted 3 hours and provided the SPs with background information about the study and its counseling protocols (including a discussion of how to respond when the study counselors do and do not use appropriate counseling techniques), a detailed overview of each case, the opportunity to ask questions, and training in how to assess the study's counselors using a checklist developed for the study (see Step 4 for checklist details).

Step 3 -Conduct the SP encounters
The study's project director scheduled telephone appointments for the counselors and SPs, who were located in separate rooms in the hospital. The counselors called the SPs and completed a first telephone counseling session using the methods they would use with study participants. We audio-taped the sessions using a digital recorder connected to each counselor's phone, and the study co-investigator leading the SP encounters listed-in "live" on each call.
Step 4 -Rate trainee performance SPs and trainers can consistently and accurately rate trainee performance with well-developed assessment tools [10,11]. Therefore, immediately after the counseling sessions, the SPs and the co-investigator present at the SP encounters rated the counselors using a 35-item checklist created for the study (Figure 1). To create the checklist we combined the validated Behavioral Change Counseling Index [12] with items specific to our study's counseling protocol. Using Likert-type scales, the SPs and coinvestigator scored the presence and quality of smoking cessation counseling behaviors outlined in the study's counseling manual and counseling behaviors in accordance with behavioral change counseling techniques [7]. At a later date the study's clinical supervisor listened to the session audiotapes and completed the checklist for each session.
Step 5 -Provide feedback to trainee Immediately after the SP encounters, the co-investigator present at the encounters, the SPs, and the counselors met for a 30-minute debriefing session. The co-investigator and SPs provided counselors with feedback on their performance. The counselors discussed how they felt during and after the encounters, including their opinion on whether the encounters felt real to them and how the cases could be improved.
At a later date the clinical supervisor provided oneon-one feedback to each counselor using the checklists Table 1 Steps required to conduct an SP exercise Step Resources required completed by the supervisor and SPs. The clinical supervisor also reviewed the checklists to certify each counselor as ready to work with real patients. To be ready to work with real patients counselors needed the following score patterns: 1) On the Counseling Protocol scale counselors needed a rating of "Did Well" on the majority of items and no ratings of "Did Not Do;" 2) On the BECCI counselors needed a rating of "A Good Deal" or better on the majority of items and no ratings of "Minimally" or lower unless an item was not applicable; and 3) On the Overall Ratings scale counselors needed a rating of "Very Much" on the majority of items and no ratings of "Not at all."

Assessing costs
We calculated the labor costs associated with developing and conducting the SP exercises. We structured these costs into two categories: development costs (steps 1 and 2) and per-session costs (steps 3-5). To calculate the development costs we estimated the number of hours it took for personnel to develop two SP cases and identify and train two SP actors. We multiplied these hours by personnel hourly rates in 2009 dollars (the year in which we conducted the exercises) and added NYUSoM's fringe benefit cost. To these study personnel costs we added the costs of the two SPs, which were calculated as the SPs' hourly rate multiplied the number of hours each SP spent in training. For per-session costs we calculated labor costs associated with a 1-hour counseling session between one counselor and one SP. We provide estimates for one hour of counseling, because this was the recommended length of time it would take to complete an intensive first counseling session with our difficult study population. As with the development costs, we multiplied the numbers of hours each study staff member worked on an SP session by his/her hourly rate and added NYUSoM's fringe benefit cost. SP costs were calculated as the SP's hourly rate multiplied the number of hours an SP spent in counseling and performing trainee rating and feedback.

Findings
During the debriefing session the SPs and supervisors reported that the checklist was easy to use when rating the counselors' competencies, and the counselors reported that they found the ratings to be helpful in understanding how they performed during the SP exercises. Both counselors reported that the SP exercises were helpful for practicing their clinical work and for practicing the logistics of Has never tried counseling to quit smoking. If asked on a scale of 0-10 his motivation to quit smoking, patient will report a 7. Motivated to quit smoking because it is too expensive and because he knows it is bad for his health. If asked on a scale of 0-10 his confidence in being able to quit, patient reports a 5. He is not very confident because of his struggles with stress and his previous relapses. He is unsure if he will be able to cope with his PTSD and stress without cigarettes.
Mental health: PTSD linked to military service during Vietnam war. Suffers from insomnia, flashbacks of his best friend being killed in front of him, and heightened startle reflex to loud noises. History of alcohol abuse but sober for 3 years. Recovered from alcohol abuse using AA (which he still attends) and religion (Catholic). Very proud of his sobriety.
Other SP psychosocial history Wife left him 7 years ago due to his drinking. Does not have great relationship with his adult children and would like to be closer to them. They live in NYC but he rarely sees them. Lost his last job (NYC bus driver) due to his drinking. He has many friends through AA and VA programs. Most of his friends smoke and may not support his quitting.
Instructions on how to respond to intervention competencies demonstrated by trainee during the SP encounter Begins call in "contemplation" stage of change: thinking of quitting but not yet ready to set a quit date or commit to quitting. If the counselor uses appropriate counseling techniques (see protocol overview), the patient can commit to quitting. If the counselor does not use appropriate counseling techniques, the patient will remain unsure of his motivation to quitting.
working within a research setting. Both counselors also reported that working with the SPs allowed them to release some nervousness and build self-confidence in working with real patients. The counselors reported that they found the SP encounters to be realistic but reported two major weaknesses in the SPs' abilities to portray VA mental health patients. One SP was not familiar with common military terminology of his combat era, and a second SP was not able to let the counselor know if he was taking any psychiatric medications. These lapses in knowledge were salient to the counselors and affected their ability to perceive the encounters as real.
Based on the success of the SP exercises with our first two counselors, we used SPs to train three subsequent study counselors after making some changes to our methods. We modified the cases to include more information about each SP's military history and ongoing mental health treatment. In addition, to more closely resemble the procedures that would be used with real patients, the counselors called the SPs at their homes, we did not have a study team member listening-in on each call, and we did not have the counselors meet their SPs for an in-person debriefing session. Instead, the SPs provided feedback over the phone immediately after their counseling call. Table 3 provides a summary of the labor costs associated with developing and executing the SP exercises using our modified session methods described in the previous paragraph. The cost of developing the SP cases and training SP actors was approximately $1,475. The per-session cost of conducting a 1-hour counseling session between one SP and one counselor was approximately $314.

Discussion
The literature supports the use of SPs to train and assess clinicians [1,5]. Our study supervisors found the use of SPs to be effective in assessing counselor readiness to work with real patients, and the counselors reported that they found the SP exercises to be realistic and helpful for their training. Labor costs associated with the use of SPs for counselor training were modest and comparable to costs found in prior research piloting the use of SPs to train substance abuse counselors [9].
Our methods have several limitations. First, we told the counselors that they would be treating SPs, which could have affected the way in which they provided treatment [13]. Second, we only tested the counselors on a single first call with each SP rather than have them complete a full course of counseling with an SP. Third, because we did not assess the counselors' competencies with our checklist before the SP encounters, we are not able to determine if the counselors' skills improved as a result of the SP exercise. Fourth, the BECCI portion of our checklist has been validated in prior research, but the two sections of the checklist created for this study were not validated. Lastly, generalization of our findings may be limited. Our study took place in a city with a relatively high cost of living. Therefore, labor costs may be significantly higher than if the study were conducted in a different city. In addition, the per-session costs necessarily depend on the length of a counseling session. Costs would be lower to train counselors for briefer interventions.
In summary, the purpose of this paper was to describe the application of the standardized patient training methodology to the assessment of counselors working on a clinical trial. The application of the methodology was feasible, inexpensive and offered training benefits beyond those provided by didactic instruction and role plays. Our research group is now routinely using SPs for the training of incoming telephone counselors. Step 1 Total $515 Step Step 3 total $87 Step Step 4 total $90 Step