Kathryn L. Lovell, PhD*, Brian E. Mavis, PhD^†, Jane L. Turner, MD^‡§, Karen S. Ogle, MD^¶, Marilee Griffith^§, *Department of Pathology and Neuroscience Program, ^†Office of Medical Education Research and Development, ^‡Department of Pediatrics & Human Development, ^§Office of Academic Programs, ^¶Department of Family Practice, College of Human Medicine, Michigan State University

Abstract: Performance-based assessments of medical students using standardized patients (SPs) have been conducted in a variety of formats during both pre-clinical and clinical years. At Michigan State University College of Human Medicine, a series of performance-based assessment (PBA) experiences have been established in the second year to evaluate students' abilities to integrate basic skills and medical knowledge. The format developed was based on classmates acting as SPs. The PBA experiences were perceived by students as being effective in reinforcing the goals of the curriculum and the format was viewed positively. The small-group learning experience provided by the patient training session was valued by students, and was an important part of their positive attitude toward the time required for acting as an SP. Thus using students as SPs can be an excellent cost-effective approach for performance-based assessments used as formative evaluation.

    Performance-based assessments of medical students using standardized patients (SPs) have been conducted in a variety of formats during both pre-clinical and clinical years. For example, standardized patients can be used for assessments during courses or clerkships, or for a comprehensive OSCE. Nationwide, the use of SPs for student assessment increased from 61% of medical schools in 1992-93 to 85% of schools in 1996-97¹. The use of SPs may be expensive, both in terms of payments to the individuals hired, and in terms of staff time for training SPs, but simulations of patient encounters are more representative of the clinical arena than are paper and pencil exams. We thought it was important for our second-year medical students to be challenged with clinical situations in which they had to integrate basic science and physical exam knowledge to solve a clinical problem. In order to conduct these assessments at several times during the second year curriculum, we explored options to hiring SPs, and, for both economic and educational reasons, initiated a format in which medical students served as SPs (student-SPs) for interview or examination by their class-mates (student-doctors). This paper describes our experiences and student subjective evaluation of the format.

    The Michigan State University College of Human Medicine has a traditional lecture-based format during the first year and a modified problem-based learning (PBL) format during the second year for the basic science curriculum. In the second year, the integrated PBL curriculum contains 10 domains (courses), each organized around a basic concept (Infectious Diseases, e.g.) or a system (Cardiovascular, Pulmonary, Neuromuscular, e.g.). The Clinical Skills course, which covers interviewing and physical exam skills, is taught throughout the two pre-clinical years. In the second year, students are taught one afternoon per week in small groups. Topics for physical exam in the Clinical Skills course are closely tied to the PBL curriculum where possible.

    Until 1995, a second-year OSCE was conducted as a formative evaluation. After reassessing its value to the curriculum and to the students², the decision was made to discontinue this OSCE. One reason for this decision is that faculty felt that the OSCE was freestanding outside of the curriculum. Without a close linkage to specific curricular components, its value for student and curricular feedback was severely limited. Since 1995, a series of performance-based assessments (PBAs) have been established as formative evaluation experiences in the second year to evaluate students' abilities to integrate basic skills and medical knowledge. Students are given feedback regarding performance in these exercises, but their performance does not count as part of their course grade. The PBAs were initiated to provide an assessment event reinforcing basic science knowledge and clinical skills within a clinical context, and in close proximity to instruction.

    The only other similar evaluation experiences for second-year medical students of which we are aware are a series of three-station mini-OSCEs at University of Alabama School of Medicine at Birmingham, associated with the Introduction to Clinical Medicine course and the organ system just completed in Correlative Pathology. Bonner et al. ³ reported greater student satisfaction using a mini-OSCE with non-student-SPs, as compared to their traditional second year twelve-station OSCE, and greatly simplified administrative logistics.

    The MSU PBAs challenge students to solve a clinical problem. Each PBA is comprised of a number of stations, which might include: gathering data from an SP using interviewing skills; performing a physical examination and interpreting physical examination findings to generate a differential diagnosis; answering computer-based or written questions related to diagnosis or management; and writing portions of a written record, including providing justification to support the differential diagnosis. A short essay in which students explain the pathophysiology underlying a sign or symptom of the case is included to emphasize the connection between the clinical problem and basic science content. Since these PBAs integrate clinical skills and basic science knowledge, basic science faculty work with clinical skills faculty to identify objectives, develop the cases, and define expectations for performance. The exercises occur the week following the completion of the relevant basic science PBL domain final exam, after students have had training in the relevant physical exam topic, but prior to the Clinical Skills course final exam. The PBA exercises have been successful in providing formative evaluation in an integrative experience requiring performance of skills and problem solving at several points in the curriculum.

    When the initial PBA was designed three years ago relating to the nervous system, the goals were to develop a performance-based assessment experience at minimal cost that would (a) evaluate students' abilities to perform a neurological exam including identification of abnormal signs and symptoms needed for localization of a lesion⁴, (b) determine a differential diagnosis based on basic science content material, and (c) complete portions of a written record. The format developed was based on classmates acting as SPs for three reasons. (1) Funding and staff time was not available for hiring and reliable training of external standardized patients. (2) Training students to act as patients was designed as a learning experience to enhance their abilities in localization of lesions and understanding of the neurological exam interpretation. (3) With their medical school background, students were easier to train than individuals with no knowledge of nervous system function. Since the initial neurological PBA, two additional exercises, based on cardiovascular and pulmonary systems, have been established, all using classmates as SPs. In spring 1998 the neurological PBA added the use of outside, paid standardized patients for one part of the experience, providing the opportunity to compare students' subjective opinions about classmates as 'patients' (student-SPs) with opinions about the externally recruited individuals as 'patients' (non-student-SPs).

Implementation of PBAs

    Each PBA was conducted on two afternoons during the week of the PBL content exam, with half of the students (total = 106) assigned as student-doctor and the other half assigned as student-SP each day; on the second afternoon, the roles were reversed and a different case was used. Based on analysis of performance data, there were no appreciable differences between student-doctors who participated as student-SPs two days before, compared with those who were assigned as student-doctors the first afternoon. Groups of 8 to 10 student-SPs and 8 to 10 student-doctors were scheduled to begin the PBA at six times during the afternoon. Although there were variations in sequence for each PBA, the following elements were included:

    The schedule for the 1998 Neurological PBA is shown below; the schedules for the Pulmonary and Cardiac PBAs had similar elements, but only three stations were involved and no non-student-SPs were used:

Table 1 - Activities during each station

    The training session for student-SPs was conducted by 1 or 2 faculty members meeting with the group of 8 to 10 students immediately prior to their encounter with the student-doctor. The format used most often and most successfully included one of the faculty simulating the 'patient', with the students asking questions to determine signs and symptoms, and then discussion to summarize the case and emphasize key clinical features. The students then were given time to practice with a classmate for simulation of the clinical signs. The student-SP training emphasized the importance of a valid simulation for the benefit of the student-doctor.

    In the three PBAs conducted during the year, the training of student-SPs was done differently; the format of training had a large impact on the attitude of students as student-SPs. In the cardiology PBA, in the first year that it was conducted (1996), student-SPs were given a written summary of the facts of the case and were instructed to commit the facts to memory and to play the role of the patient. The following year (1997), a faculty member went over the facts with each group of ten student-SPs, quizzed them about the facts of the case, instructed them to "role play" with each other, and challenged them to try to make a diagnosis based on the historical data. The educational value of being a student-SP was further enhanced in the neurology PBA, as two faculty members met with each group of students during the training period, and conducted an interactive session that served as an active learning experience.

    To determine if the case elements were remembered correctly by the student-SP, a quiz was given to the student-SPs immediately after the interview or physical exam in one PBA to test their memory of the correct symptoms/signs. With only a few exceptions, the key clinical features of the patient case were accurately remembered by students. Any inaccuracies revealed by more than one or two students on each afternoon were not important elements leading to the differential diagnosis, as determined by the faculty evaluating the PBA.

    The PBA was designed to be a formative evaluation, and was a required, non-graded experience. Each student was given individual written feedback on performance of knowledge and clinical skills elements of the PBA within 2 weeks of the experience. To obtain subjective feedback from students, the evaluation form at the end of the Clinical Skills course included questions on the value of the PBA and the students' attitudes toward being a student-SP. The questions asked about each PBA were not identical. Of the 106 students enrolled in the course, 85% returned the evaluation forms. In addition, a focus group was convened at the end of the semester to get more in-depth feedback. Students for the focus group were recruited by the PBL curriculum director, who had no connection to the PBA implementation, and the focus group was conducted by one of the authors (BEM) with experience in medical education research. The same person who conducted the group collected feedback.

Subjective Evaluation by Students

The PBA experiences were perceived by students as being effective in reinforcing the goals of the curriculum and the format was viewed positively. The evaluation form responses are shown in Table 2 (n=90).

Table 2 – Results from Course Evaluation Forms

SA=strongly agree; A=agree; D=disagree; SD=strongly disagree; NO=no opinion or no answer

    Over 70% of students responded 'agree' or 'strongly agree' that each of the three PBAs enhanced their learning in Clinical Skills. For the Pulmonary PBA only 3% said their classmate was not believable for the interview, and for the Neurological PBA, 10% said their classmate was not believable for the physical exam. The most negative aspect related to feedback about the non-student-SP; i.e. 29% of students said the interview was not worthwhile.

    On the fall semester 1997 Clinical Skills course evaluation form, students were asked for uncued written responses to "The most effective components of this course were:" The Cardiac PBA experience was listed by 9 students; the only components mentioned more frequently were the cardiac examination (13 students) and breast examination (12 students). On the spring semester 1998 Clinical Skills course evaluation form, students were asked for uncued written responses to "The most effective components of this course were:" The PBA experience was listed by 13 students; the only component mentioned more frequently was "practicing on, and exposure to, patients", which was listed by 15 students.

    There was a great deal of variability in the reaction to the externally recruited standardized patients used in the 1998 Neurological PBA. Comments written on the evaluation form included: "Some of the simulated patients were harsh; the patient I had did not accurately rate - she forgot questions I had asked her; need to standardize the patient more thoroughly; the interview could have just as well have been with a fellow student - simulated patient wasn't bad, but maybe not necessary; the actor patient did not correctly assess my interview."

    The focus group provided similar responses to those indicated on the evaluation forms. Everyone agreed that the training received for the role of a patient was very helpful in understanding the case and was itself a learning experience. One element mentioned as important was the time spent by faculty instructors in the patient training groups to provide a high-quality interactive educational experience in a small group setting. Students appreciated the opportunity to try their clinical skills on another student, which they acknowledged reduced the fidelity of the experience, but increased the comfort level.

Discussion

    A unique feature of the PBA format was the role of the student-SP. In the neurological PBA in 1997-98, 65% of the students agreed (answered 'SA' or 'A', Table 2) that playing the patient was worthwhile. Of those who gave an opinion, 15% disagreed with that statement. In 1996-97, many students complained that playing the patient was a waste of their time. In contrast, in 1997-98 students were generally quite positive about the experience of playing patient, often describing it as 'fun' and 'worthwhile'. Students were far more positive about playing the patient when a faculty member worked with them and challenged them to understand the case as they played the role. The use of students as SPs for an interview or focussed physical exam provided an opportunity for an additional small group educational experience for those students.

    We did not specifically investigate the accuracy of the portrayal by student-SPs or non-student-SPs. If this format is used in a graded experience, the accuracy would be a critical issue. The quiz of the history and clinical presentation given to student-SPs in some of the PBAs indicated that student-SPs remembered the most important features of the case. Consideration of the symptoms and signs listed in the written record also supported the general accuracy of protrayal by the student-SPs. In early trials of the PBA format we found that sometimes a student-SP gave hints to the student-doctor or was not serious in the portrayal of a patient. Thus in 1997-98, the patient training emphasized the importance of a valid, serious simulation for the benefit of the student-doctor. In addition, the interviews in 1997-98 were audiotaped for other purposes, but the taping may have had a positive impact on the validity of patient performance. Videotaping would be the best method to monitor the simulation, but we do not have the facilities to make that possible.

    Student-SPs can be compared to non-student-SPs in a number of ways. With respect to the interview with a non-student-SP, 29% of students did not feel the interview was worthwhile, although we have not fully explored the reasons for this reaction. Informal feedback suggested that students felt less comfortable interviewing a non-student SP. Also a few of the non-student-SPs apparently did not portray the patient in the manner expected – this issue could be addressed by altering the training session. There was an advantage of using student-SPs with respect to training, since medical students understood the technical language and understood the purposes of the simulation and the medical condition portrayed. For evaluation, student-SPs in general provide an advantage to the student-doctor and we found that student-SPs were more willing to give credit to student-doctors when the rating questions were ambiguous or when student-doctors used technical language⁵. These issues would need to be considered if the PBA were used as a summative rather than formative evaluation.

    In summary, students appreciated the opportunity to try their clinical skills on another student, which they acknowledged reduced the fidelity of the experience, but increased the comfort level. The learning experience provided by the patient training was valued, and was an important part of their positive attitude toward the time required for simulating a 'patient'. Students agreed that the experience was a reasonable simulation of a clinical problem-solving experience, and that their learning in Clinical Skills was facilitated. Thus we conclude that using students as standardized patients is an excellent cost-effective approach for performance-based assessments used as formative evaluation.

References

1. Emerging trends in the use of standardized patients. Contemporary Issues in Medical Education, 1998; AAMC.
2. Mavis BE, Henry RC, Ogle KS, and Hoppe RB. The emperor's new clothes: The OSCE reassessed. Acad. Med. 1996; 71:447-453.
3. Bonner JR, Yancey LW, Berner ES, and Jackson JR. ICM "mini-OSCEs": An alternative to an end-of-course multi-station examination. IME abstract, AAMC meeting, 1996.
4. Mavis BE, Lovell KL, and Ogle KS, Why Johnnie can't apply neuroscience: Testing alternative hypotheses using performance-based assessment. Adv. Health Sci. Educ. 1998, in press.
5. Mavis B, Ogle K, Lovell K, and Madden L. Performance-based assessment of interviewing skills for pain evaluation. RIME abstract, AAMC meeting, 1998.

Lovell KL, Mavis BE, Turner JL, Ogle KS, Griffith M. Medical students as standardized patients in a second-year performance-based assessment experience. Med Educ Online [serial online] 1998;4:6. Available from URL http://www.utmb.edu/meo/

Dr. Lovell is a Professor of Pathology at Michigan State University. She can be contacted via e-mail at lovell@pilot.msu.edu