Lisa Doyle Howley, PhD^*, James Martindale, PhD^†

     The use of the standardized or simulated patient (SP) as a clinical teaching and assessment tool has a long history and strong evidence of validity.¹ The most recent LCME Annual Medical School Questionnaire (2002-03) revealed that ninety-seven schools (approximately 75% of all medical schools) reported using one or more SP examinations in the introductory skills courses. ² This methodology has proven effective for various reasons, including the accessibility of patients, control over external factors, and the standardization of patient problems. Standardized patients (SPs) allow faculty to gain a better understanding of the students’ skills in interpersonal communication, data gathering, problem solving, and the management and synthesis of clinical information.¹

     An additional advantage of this methodology, which has received much less attention, is the standardized patients’ unique position to offer the medical student feedback from the patient’s perspective. Patient feedback is an invaluable method for directing attention to the strengths and weaknesses of students’ interpersonal skills.³ While the reliability of written ratings or checklist scores provided by SPs has been well documented; the utility of the oral feedback delivered by the SP has not been actively investigated.⁴

     The purpose of the current study was to investigate the efficacy of oral feedback provided by SPs. Specifically, we sought to answer the following research questions:

      The Nature of Feedback - Medical students, as adult learners, seek frequent feedback and reinforcement regarding their performance. They are driven by internal motivators, such as the successful completion of a task, self esteem, and recognition, as opposed to external motivators such as letter grades, awards, and promotions.⁵ Unfortunately, students complain that they simply do not receive enough feedback and are rarely observed interacting with patients.^6-8 Dr. Beverly Wood postulated two reasons for the lack of feedback in medical education: 1) the infrequency with which trainees are closely observed performing a skill, and 2) the unease felt by faculty preceptors in delivering feedback.⁹

     Often students are forced to rely on summative examinations and reviews for reinforcement. Programs that attempt to provide formative feedback to their students typically do not do so in a standard format and may wait several hours or days to discuss student progress. In his seminal article, Ende discussed several consequences of withholding feedback from students: Mistakes go uncorrected, good performance is not reinforced, and the acquisition of clinical skills are threatended.¹⁰ In contrast, the provision of frequent, explicit feedback allows students to become more active and engaged in their educational process.

     For purposes of this study, we defined feedback as an informed, non-evaluative, objective oral appraisal of performance intended to improve clinical skills. This definition is based on the work of Ende.¹⁰ In addition to defining and raising awareness of the importance of feedback in medical education, Ende described how feedback was often omitted or handled improperly in clinical training. He proposed standard guidelines or principles for delivering constructive feedback including 1) work as an ally with the study, 2) base feedback on observed incidents and on modifiable behaviors, 3) give feedback in small digestible quantities, and 4) use language that is non-evaluative and nonjudgmental.¹⁰

     The current study incorporated these feedback principals into the SP training and addressed what Branch and Paranjape defined as “formal” feedback.¹¹ They defined three general categories of feedback: Brief (<5 minutes), Formal (5-20 minutes), and Major (15-30 minutes). Formal feedback is provided when the feedback provider and the learner set aside a specific amount of time to discuss performance. Unlike formal feedback, major feedback is more conducive to mid-point corrections and addressing extreme problematic or unprofessional behaviors.

      Research on Feedback - The empirical research on oral feedback in medical education has primarily focused on the professional instructor (resident/faculty) - student process. Several studies have shown that students who receive feedback regarding their clinical performances report more positive attitudes toward the experience.^12-14 Several studies have also shown that feedback can have a significant influence on student performance.^14-16 For example, Hollingsworth, Richards, and Frye found a significant correlation between scores on a clinical skills examination (OSCE) and the amount of feedback statements given by faculty observers.¹⁵ Hodder, Rivington, Calcutt, and Hart found that even very brief feedback encounters (2 minutes) can have a significant influence on clinical competency.¹⁶

     In contrast to professional instructor-student feedback, there is little research, descriptive or experimental, regarding the simulated patient-student feedback process. At the University of Virginia School of Medicine, we have found that standardized patients, when carefully trained, can offer another means for providing students with immediate, constructive, and focused feedback. Levenkron, Greenland, and Bowley found that direct oral feedback from a simulated patient trained in behavioral counseling skills was preferred to faculty feedback of a videotaped SP-student encounter.¹⁷ In preparation to provide feedback, the SPs in the Levenkron et.al. study underwent approximately 20 hours of training. The students in the first group who rated the encounter more favorable were not videotaped and the authors questioned whether these results were due to the students’ anxiety about being videotaped or due to the effects of the SP feedback. Leeper-Majors, Veal, Westbrook, and Reed conducted a small pilot study investigating the effects of SP feedback on eight surgical residents’ abilities to obtain informed consent.¹⁸ They found, on repeated occasions, that those residents in the feedback group performed significantly better than those in the non-feedback group. Although these results are tentative due to the small sample size, they provide promising early evidence to support the efficacy of SP feedback.

     We sought to gather empirical evidence to determine whether students who received oral feedback from SPs would evaluate the educational encounter more positively than those who did not receive this feedback. We also sought to determine whether SPs trained in a relatively short amount of time would be able to deliver constructive feedback to first year medical students regarding their general interviewing skills.

Methods

     For purposes of this study, an entire class of 136 first year medical students was randomly assigned to treatment (N=70) and control (N=66) groups. Consent was obtained from all participants prior to their participation. These students completed a regularly scheduled video-taped simulated patient interview of a single SP. The formative instructional activity was meant to provide students with the opportunity to practice basic medical interviewing techniques with a standardized patient complaining of a relatively common outpatient problem. The interview was conducted five months into the students’ first year of medical school and was the first formal encounter with an SP. The session was videotaped and later reviewed by the student. Select sections were also reviewed by a small group of his/her peers, and two faculty members. The only prior experience with an SP occurred four months earlier during an informal small group interview setting led by faculty tutors.

      Prior to participating in the study, all standardized patients (N=8) completed six hours of training: 3.5 hours were devoted to training the SPs to portray a single ambulatory focused case and 2.5 hours were devoted to feedback training. Individual SPs were recruited from a cadre of participants, ranging in prior SP experience, based on case characteristics and availability. The training materials and methods consisted of a previously developed and empirically-based standard workshop titled “Focusing Feedback on Interpersonal Skills: A Workshop for Standardized Patients.”¹⁹ The primary goal of this 2.5 hour training workshop, based on the work of Ende and others, was to foster the SPs ability to give clear, non-evaluative, descriptive feedback regarding interpersonal skills demonstrated by the medical student. During this training workshop, standardized patients were also instructed to follow a standard format for conducting the formal feedback session. Further details regarding this workshop are beyond the scope of this study and are reported elsewhere.²⁰

      Students were oriented to the activity and the research project immediately before participation. The study took place over four evenings for a total of 12 hours (6 encounters were run simultaneously). Treatment and control groups were scheduled to complete the activity on alternate dates to reduce diffusion of treatment and increase the validity of the research. Due to space limitations, the activity was conducted during the evening, in actual examination rooms, of the family medicine clinic at the hospital. Both groups of students were given 20 minutes to interview one SP while being video-taped. In addition, the treatment group was offered the opportunity to receive an additional five minutes of immediate feedback from the patient regarding his/her interviewing skills: One hundred percent of the treatment group elected to receive SP feedback. The activity was consistent for both groups with the exception of the feedback session or intervention for the treatment group.

     At the completion of the session, both groups completed an anonymous evaluation instrument regarding their attitudes towards the educational activity. Two forms of the instrument were developed by the Primary Investigator for purposes of this research. Both forms A (control group) and B (treatment group) included seven common items regarding various aspects of the activity and two open-ended questions regarding overall strengths and weaknesses. Form B included seven additional items regarding the quality of oral feedback provided by the SP. The students were asked to respond to each item using a 5-point Likert scale. Specific items are provided in Table 1 (Forms A & B) and Figure 1 (Form B). Reliability coefficients, Cronbach’s Alpha, were computed for each form (.73 and .78, respectively).

      Both qualitative and quantitative methods were used to investigate the effect of SP feedback. In order to address the first question, a basic randomized design comparing treatment (or those receiving SP feedback) to control (those not receiving SP feedback) was conducted. Specifically, responses to seven common Likert-item statements were compared across treatment and control groups. To address the second question, students in the treatment group were surveyed about their impressions of the quality of the feedback provided to them by their SP. Specifically, additional Likert-item responses from the treatment group were summarized with descriptive statistics. In addition, student open-ended responses to two questions were reviewed and categorized according to prominent theme.

Results

     Question 1 - In order to address the first question and analyze the effect of treatment on the attitudes of the medical students, we compared the mean scores on the seven evaluation items for both groups. Specifically, we tested the statistical hypothesis that mean differences among treatment and control groups on a combination of evaluation variables were due to chance.

      A total of 135 evaluations were fully completed and returned (99%). Prior to conducting any statistical analysis, the data were evaluated to determine whether the appropriate assumptions for performing the analysis were met: The Box’s Test was not significant and group sample sizes were similar.

      The results of the statistical analysis revealed a statistically significant difference in student attitudes between the feedback and no feedback groups. Specifically a one-way multivariate analysis of variance (MANOVA) revealed significant differences among control and treatment groups on the seven combined dependent variables, Wilks’ ?=.890, F(7, 127)=2.25, p<.034, canonical correlation of .332, ?2=.11. A discriminant function analysis was conducted to assess the relative contribution of each dependent variable to the discrimination between treatment and control groups. Table 1 presents means and standard deviations for evaluation items by each group. Standardized function coefficients and correlation coefficients (see Table 2) revealed that the variables of Insight and Safe were most associated with the function. The feedback group was classified correctly in 75.4% of the cases and 57.6% of the control group was classified appropriately, resulting in 66.7% correct overall classification rate. These results are consistent with the means of the discriminant functions. The feedback group had a function mean of .342, while the non-feedback group had a function mean of -.357. These results suggest that those students who responded favorably to Insight and Safe were more likely to have received feedback from the SP.

      Question 2 - In order to address the second research question, regarding the quality of the feedback provided by the SPs, we analyzed treatment group responses to seven Likert scale items. Figure 1 includes these items and their corresponding mean ratings (with confidence intervals). One hundred percent agreed (or strongly agreed) that oral feedback from the SP was an important component of the activity and that the SP provided reinforcement of a positive behavior. The vast majority agreed that the feedback provided was clear (97%), nonjudgmental (99%), and specific (80%). The vast majority also agreed that the feedback provided the student with an understanding of the patient’s perspective of the clinical encounter (97%).

     The post-activity evaluation instrument (Forms A & B) also included two standard open-ended questions regarding strengths and weaknesses of the general activity (viz., “What were the strengths of this activity?” and “What were the weaknesses of this activity?”). These statements were transcribed, reviewed, and categorized according to prominent theme. As Table 3 (see appendix) reflects, a total of 124 comments were coded into 7 strength categories ranging from “general” (N=40) to “organization” (N=5). A total of 18 unsolicited open-ended positive statements were provided regarding the quality of SP feedback. A total of 69 comments were coded into 6 weakness categories ranging from “lack of feedback” (N=12) to “timing” (N=7). No students in the treatment group reported quality of SP feedback as a weakness. Examples of open-ended statements that fell within each category are provided in Table 3.

Conclusions

     We believe that these qualitative and quantitative findings lend support to the inclusion of a brief formal feedback session immediately following a formative SP activity during pre-clinical medical education. Although we found statistically significant differences between feedback and non-feedback groups on satisfaction with the SP exercise, the overall effect size was small (?2=.110). The discriminant function analysis revealed that the evaluation items Insight (“This exercise provided me with further insight into specific interviewing techniques”) and Safe (“This was a safe method for me to work through some of my weaknesses”) were the most important variables for discriminating between treatment and control groups. Those students in the feedback group tended to report stronger agreement with these statements than those in the control group who did not receive feedback. Although further research is needed to understand why these two variables were able to discriminate better than the other five variables, we suggest that this evidence supports the validity of the feedback delivered by the SPs. The SPs are specifically trained to deliver feedback that is non-evaluative and descriptive and we believe that students’ ratings of Safe and Insight, respectively, reflect these aspects of the feedback delivery.

      According to the participants, the quality of the feedback provided by the SP was very strong. This is particularly encouraging considering the relatively brief feedback training session (2.5 hours). As Figure 1 displays, all items were negatively skewed towards strongly agree. Item four was rated below the other six quality indicator items. This item was intended to measure the students’ perceptions of whether SP feedback was behaviorally descriptive in nature. Although we do not have specific conclusions regarding this finding, we speculate that this reflects the relative difficulty in providing descriptive behavior-oriented feedback. A finding we have anecdotally experienced in practice and training. Despite this difference in mean ratings, it should be noted that only 6 (9%) students disagreed or strongly disagreed that SPs provided feedback about a specific behavior in need of change.

      This study has several limitations. First, student attitudes towards an educational activity are only one measure of effectiveness. Additional measures, such as performance outcomes would provide much stronger evidence for the efficacy of SP feedback. However, we believe that attitudes are an important factor, particularly to the adult learner. Another related weakness is our reliance on student self-report data to judge the quality of the SP feedback. Self report data supplemented with actual observational data would provide much stronger evidence.

      Further research is needed to explore the relationship between oral feedback delivery on students’ perceptions of safety in educational settings and how this feedback provides students with insight into their performance. Future studies that capture and critically evaluate the feedback provided by the SPs would be very beneficial. Additionally, we suggest the following research questions for future analysis: How does the quality of feedback delivered from formally trained and un-trained SPs differ? What SP characteristics, if any, contribute to the delivery of constructive feedback? How does the quality and nature of the feedback delivered by SPs differ from that delivered by medical professionals? Further research is certainly warranted to explore the seemingly important, unique, and valuable role of SP feedback in medical education.

      For the adult learner in medicine, feedback from a clinical faculty member or resident is most valuable. We do not suggest that this critical educational tool be replaced by SP feedback. Instead, we recommend that SPs be used as a supplemental feedback delivery resource. Given the increasing demands being placed on faculty and residents and the minimal time spent observing and providing feedback to students, it is a resource that is sorely needed. If our goal is to foster active, lifelong, and internally motivated learners, we must expand our avenues of instructional delivery. Standardized patient feedback is one mechanism for doing so.