Abstract: Although many medical schools have adopted a variety of methods to assess student competency, the extent to which these innovations have changed how decisions about student progress are made is not clear. This paper describes a survey of 126 accredited allopathic U.S. medical schools to determine which information sources are used for decisions related to medical student progress and graduation. Respondents were asked to indicate up to three information sources used for seven specific decisions about student progress. The results indicate that multiple choice questions (MCQs) and faculty ratings remain the most frequently used information sources. Clinical skills education in the pre-clinical curriculum is the area with the broadest use of assessments for progress decisions. Several explanations are suggested for the primacy of MCQs and faculty ratings in student decisions, including familiarity for faculty and students, ease of implementation and the resources required for the adoption of other assessment strategies.

     The assessment of professional competence is both difficult and frustrating. The difficulty lies in delineating the complex combinations of abilities required for professional practice, and the variability of performance across situations and conditions. Frustration arises when we try to translate our general sense of good and bad practice into a set of mutually agreeable criteria with which behavior can be judged.¹

     Multiple choice questions (MCQ) and faculty ratings have long been the dominant methods for assessing the competence of medical students. MCQ’s have the important advantage of allowing for broader sampling of competency domains per unit of testing time.² Their disadvantage is that during clinical encounters, patients don’t present with five choices from which the physician can choose.³ Faculty ratings have been used to assess students’ skills and problem-solving primarily within a clinical context. While this form of assessment clearly is relevant to the work of a physician, from a psychometric perspective faculty ratings frequently are biased and provide little discrimination among students.⁴

     With increasing awareness of the limitations of MCQs and faculty ratings, educators have become interested in exploring other forms of student assessment that more closely approximate the tasks of the physician.⁵ Also stimulating this interest in new assessment strategies has been recognition of the reciprocity between learning and assessment. There is increasing evidence that assessment strategies drive learning⁶ through the content, format, student feedback, and timing of assessment within the curriculum.⁷

     It is within this context that medical schools have expanded their student assessment strategies to include computer and live simulations, written projects, oral examinations and other situations in which the student must create rather than choose a response.⁷ Current efforts related to competency assessment are based on a general consensus that all assessment methods have limitations of one sort or another² so that a comprehensive approach to assessing medical student competency needs to include multiple assessment strategies.⁸ This has been codified in the Liaison Committee on Medical Education accreditation standards⁹:

"The medical school faculty must establish principles and methods for the evaluation of student achievement, and make decisions regarding promotion and graduation. The evaluation of student achievement must employ a variety of measures of knowledge, competence and performance, systematically and sequentially applied throughout medical school" (pp. 14-15).

     In the last decade medical educators have witnessed major changes in the assessment of medical students. A search of Medline from 1990 to 1999 found that over 600 articles have been published on educational measurement related to medical students. Two specific innovations in medical education have been the introduction of standardized patients¹⁰ and computer-mediated technologies¹¹ for instructional and assessment purposes.

     Many schools have adopted a variety of methods to assess the competency of their medical students. However it is not clear the extent to which innovations in student assessment have resulted in changes in how decisions about student progress are made. This paper describes a survey of U.S. medical schools to determine which information sources are used for decisions related to medical student progress and graduation. Of interest is the extent to which medical schools use information sources other than MCQ examinations and preceptor ratings.

Methods

     A survey was mailed to 126 accredited allopathic U.S. medical schools in November 1998, to the attention of the person overseeing the undergraduate education program. Two follow-up mailings were sent to non-respondents, at approximately 30-day intervals. The second follow-up mailing was addressed to the Dean, with the request that he or she forward the survey to the appropriate person within the medical school.

     The survey comprised two sections and required approximately 15 minutes to complete. The first section of the survey, which is the focus of this paper, asked respondents to indicate the three most important data sources used in determining students’ grades for basic science courses, clinical skills courses, clerkships, clinical electives and honors standing, as well as promotion to third year and graduation. The following list of student assessment modalities was provided:

• Multiple choice questions (MCQ);
• Oral examinations;
• Essay questions, including written exams, papers, essay projects;
• Laboratory practical examinations;
• Review of written records/chart notes;
• Faculty/preceptor ratings of students individually or in small groups;
• Objective Structured Clinical Examinations (OSCE);
• Other standardized patient assessments not included as part of an OSCE;
• Live observation of students;
• Logbooks;
• Rated or graded case presentations;
• CBX or computer-based testing (excluding MCQs);
• United States Medical Licensing Examination (USMLE) Step Examinations;
• NBME subject examinations; and
• Other.

     For each grade or progress decision, respondents were asked to indicate which one of the data sources was most influential. Space was provided to indicate the second and third most important information sources for each decision.

     The second part of the survey focused on the frequency with which assessment modalities were used in each year of undergraduate medical education. The same modalities listed above were presented to respondents and the results have been published separately. ¹²

Results

     Eighty-nine medical schools (70%) returned the survey of which 87 (68%) were returned complete. There were no significant differences between responding and non-responding medical schools on the basis of whether the medical school was public/private, or community-based. In addition, responding and non-responding schools were compared using data from the 1998 Institutional Goals Ranking Report.¹³ There was no difference between the group means based on dollars received for federal research grants and contracts, the percentage of graduates planning to practice in-state, the percentage of underrepresented minority graduates, and the proportion of graduates who assumed faculty positions. When compared on the basis of the proportion of graduates choosing primary care specialties, the difference between responding (32.0%) and non-responding (29.3%) schools was significant at the 10% level (t= 1.93, df=121, p=.057).

     A summary of the responses related to the most important and three most important data sources for each decision is presented in Table 1. Ninety-eight percent of the responding schools included MCQs as among the three most important data sources for grade determination for basic science courses. MCQs also were most frequently listed as the single most important source of information for basic science course grades. Laboratory practical examinations (74%) and essay questions (45%) were next in importance. A significant minority of respondents listed faculty/preceptor ratings. The other twelve information sources were listed by only a small number of respondents, if at all. Figure 1 compares the most important versus three most important data sources for basic science grade decisions.

     For grades related to clinical skills courses in the preclinical years, preceptor ratings were most frequently listed as the most important (67%) or among the three most important (78%) data sources (Figure 2). Other important data sources endorsed by respondents included MCQs (54%), direct observation (39%), standardized patients (35%) and OSCEs (32%). If 20% of respondents is used as an arbitrary minimum cut-off, the data suggest that clinical skills courses in Years 1 and 2 have the most variability in terms of the data sources used for grade determinations. Five different assessment methods were listed as among the most important by at least 20% of the respondents: faculty ratings, MCQs, live observation, SP assessments and OSCEs.

     Although a similar pattern was found among the data sources used to determine clerkship grades (Figure 3), schools tended to split on whether MCQs (30%) or faculty ratings (41%) were the most important data source. Live observation was also among the most important data sources (39%) listed by respondents. More standardized forms of assessment, such as OSCEs and standardized patients, were listed as important for decision making less often in the clinical clerkships than the clinical skills courses.

     For clinical electives (Figure 4), preceptor ratings were the most frequently endorsed source of information, either as most important (77%) or among the top three choices (87%). Live observation (47%), written records (24%) and case presentations (13%) were endorsed by a significant number of respondents. The remaining eleven sources were seldom, if at all, endorsed by respondents.

     Determinants for honors standing were based on faculty ratings (72%), MCQs (65%) and USMLE scores (26%), with MCQs listed most often (30%) as the most important data source (Figure 5). Class standing was written in the "Other" category as most important by 23% of the respondents, however, it is not clear which specific criteria were used to judge class standing. Few of the other information sources were endorsed by any of the respondents. Decisions related to promotion to Year 3 (Figure 6) and to graduation (Figure 7) were determined largely on the basis of USMLE scores (83% and 88% respectively). Information derived from OSCEs was factored into graduation decisions by 28% of the responding schools.

Discussion

     Despite recent curricular innovations in medical education such as small group instruction, standardized patients and computer-based testing, there is little evidence to indicate that new assessment strategies play an important role in promotion and graduation decisions. The survey data are limited in that respondents were asked to indicate the three most important sources of information as opposed to all of the information sources used for making promotion decisions. Given this constraint, the survey responses suggest that it was only a small minority of schools indicating breadth in the use of information sources. Multiple choice questions—either course-based or as standardized licensure or subject examinations—and preceptor ratings remain the most important grade determinants for each of the promotion or graduation decisions examined in the survey. Only the first and second year clinical skills courses tended to show any variability in the types of information used to determine student grades. Outside of clinical skills courses, it was rare that standardized patients, OSCEs, written records and other performance assessments had a significant role in determining student progress. Logbooks, case presentations and computer-based testing were seldom indicated as important for any of the student progress decisions.

     Although medical schools recently report some use of a broader array of assessment techniques,^{10, 12} our survey suggests that there is yet little use of these newer methods for purposes of making important progress decisions about students. Given the concerns previously raised about fidelity of MCQs to clinical practice³ and the well- documented problems with faculty ratings,⁴ we appear to be in need of much more work and discussion about what assessment tools are used, for what purpose they are selected, and how they are used. Perhaps innovative assessment strategies might be more often used for formative assessments to give students and/or faculty better feedback about student performance and perhaps to motivate learning, and to achieve improvements in instruction.¹⁴ Additionally, medical schools need to continue their efforts to promote and use existing alternative techniques for decision making, and as well, to consider new ones. In this regard, computer-based assessments may address some of the fidelity and standardization concerns that have been mentioned.

     This state of decision-making on student progress reflects a long legacy of student assessment in both basic science and clinical education. Despite their limitations, both MCQs and preceptor ratings have been in use for decades and have implicitly become the norm or standard against which newer methodologies are compared.¹⁵ They are familiar and comfortable to both faculty and students. New assessment methodologies have their own methodological weaknesses, which frequently are less familiar or evident to faculty. Other obstacles to innovation in decision-making are the resources required for implementing alternative assessments. Standardized patient programs, OSCEs and computer-based technologies require a significant investment of resources for adoption and maintenance.

     There are a number of limitations related to this study. The first limitation, previously discussed, is based on the nature of the survey questions, which asked respondents for the three most important data sources rather than all data sources. Thus, the survey might underestimate the breadth of information sources being used for decisions about student promotion. This is less of an issue for the question of the importance of various information sources for student progress decisions.

     A second limitation is related to the response rate. Almost a third of the medical schools surveyed did not return a completed questionnaire after numerous attempts to solicit participation. There were no significant differences on most of the indicators comparing respondents and non-respondents. The proportions of graduates in primary care specialties, while statistically significant, did not seem to be of pragmatic significance. However, with any survey there is a possibility of other unmeasured sources of bias that would differentiate respondents and non-respondents.

     The third limitation is respondent bias. While the survey was directed to administrators responsible for the undergraduate medical curriculum, the extent to which his or her responses accurately reflected the actual assessment practices of teaching faculty across the whole curriculum are unknown. Since multiple informant sources were not used in this study, it is difficult to determine the validity of these findings.

References

1. Kane, M. The assessment of professional competence. Evaluation and the Health Professions. 1992; 15: 163-182.
2. Fowell, S. & Bligh, J. Recent developments in assessing medical students. Postgraduate Medical Journal. 1998; 74(867): 18-24.
3. Veloski, J.J., Rabinowitz, H.K., Robeson, M.R. & Young, P.R. Patients don’t present with five choices: An alternative to multiple-choice tests in assessing physicians’ competence. Academic Medicine. 1999; 74(5): 539-546.
4. Albanese, M. Rating educational quality: Factors in the erosion of professional standards. Academic Medicine. 1999; 74: 652-658.
5. Newble, D. Assessing clinical competence at the undergraduate level. Medical Education. 1992; 26: 504-511.
6. Newble, D. & Jaeger, K. The effect of assessments and examinations on the learning of medical students. Medical Education. 1983; 17: 165-171.
7. Van der Vleuten, C. The assessment of professional competence: Developments, research and practical implications. Advances in Health Science Education. 1996; 1: 41-67.
8. Newble, D., Dawson, B., Dauphinee, D., Page, G., Macdonald, M., Swanson, D., Mulholland, H., Thomson, A. & van der Vleuten, C. Guidelines for assessing clinical competence. Teaching and Learning in Medicine. 1994; 6(3): 213-220.
9. Liaison Committee on Medical Education. Functions and Structure of a Medical School. Association of American Medical Colleges, Washington DC, 1997.
10. Association of American Medical Colleges. Emerging trends in the use of standardized patients. Contemporary Issues in Medical Education. 1998; 1(7): 1-2.
11. Association of American Medical Colleges. Computers and medical informatics in the curriculum. Contemporary Issues in Medical Education. 1998; 1(4): 1-2.
12. Mavis, B., Cole, B. & Hoppe, R. A survey of student assessment in U.S. medical schools: The balance of breadth versus fidelity. Teaching and Learning in Medicine. 2001; 13(2). (In Press).
13. Association of American Medical Colleges. Institutional Goals Ranking Report. Association of American Medical Colleges; Washington DC, 1998.
14. Mavis, B., Henry, R., Ogle, K. & Hoppe, R. The emperor’s new clothes: The OSCE reassessed. Academic Medicine. 1996; 71(5): 447-453.
15. Hull, A., Hodder, S., Berger, B., Ginsberg, D., Lindheim, N., Quan, J. & Kleinhenz, M.E. Validity of three clinical performance assessments of internal medicine clerks. Academic Medicine. 1995; 70(6): 517-522.

Note: An earlier version of this paper was presented at the Annual Meeting of the Group of ducational Affairs, Research in Medical Education Conference, Association of American Medical Colleges, Washington D.C., October 25, 1999.

Reference

Mavis BE, Cole BL Hoppe RB   A survey of inormation sources used for progress decisions about medical students. Med Educ Online [serial online] 2000;5:10. Available  from URL http://www.med-ed-online.org.

Correspondence should be addressed to:

Brian E. Mavis, Ph.D.
A-202 E. Fee Hall
Michigan State University
E. Lansing, MI 48840

(517) 353-2037 (Voice)
(517) 432-1798 (Fax)

mavis@pilot.msu.edu