Karyn D. Baum, M.D.

     Evidence-based medicine (EBM) has had significant worldwide impact on medical care and education. Before 1992, EBM did not exist as a Medline Medical Subject Heading (MeSH) term. As of March 2003, the number of papers indexed under that term was up to 10,502. EBM, often defined as “the conscientious, explicit, and judicious use of the best available evidence in making decisions about the care of the patient”,¹ gives practitioners tools to manage the ever-expanding world of medical literature and apply it to their patients.

     Educators across the world have answered the challenge of teaching these powerful decision-making devices to their medical students and residents. EBM has been taught in almost every form imaginable, ranging from morning report sessions,^2,3 to physical diagnosis rounds,⁴ to projects in ambulatory rotations,⁵ to “stand-alone” courses.⁶ A recent commentary in the Journal of the American Medical Association,⁷ which noted that the existing quality of evidence for teaching EBM is weak, implored researchers to maintain the standards of evidence that EBM educators seek to convey to their students. However, in an extensive review of graduate medical education training in EBM,⁸ it was observed that there were few assessments of the efficacy of current curricula; in the few instances where such data existed, outcomes assessed were not meaningful and skills improvements were modest. The author of the review concluded that it is essential to evaluate the efficacy of EBM curricula to achieve lasting, meaningful outcomes.

      What are these meaningful outcomes? Most studies have addressed the value of EBM curricula for learning the “skills” of EBM,^9-11 such as question formation, literature searching, and critical appraisal. Yet, our ultimate goal as educators is for our learners to use EBM in their daily practice of medicine, with no grade at stake. Unfortunately, changes in practice patterns have been difficult to accomplish at the continuing medical education level,¹² much less at the graduate and undergraduate levels.

     Adult learning theory and theories of behavior change provide much needed perspectives for the education of EBM as we strive to facilitate lasting behavior change in our learners. The Prochaska and DiClemente transtheoretical model of behavior change,¹³ often used in the description of successful smoking cessation and nutritional education programs, is a useful framework. This theory describes a continuum through which patients move as they achieve lasting behavior change, moving from unawareness of a problem, to awareness, to decision to change and planning, and finally to action and maintenance.

      If we apply this model to the instruction of EBM, the clear implication is that attitudes and motivation are instrumental if learners are to use EBM in their daily practice. The learners must have an internal drive to change. Therefore, to create an effective curriculum, we must address and evaluate not only the knowledge and skills objectives but also the attitudinal objectives. This study was designed to evaluate whether a half-day course on EBM results in improved attitudes towards the value of EBM, increased self-reported ability to practice it, and self-reported short-term behavior change.

Methods

     A half-day EBM workshop was designed to integrate EBM into the residency curriculum. The workshop, conducted in January-February 2002, was given five separate times over the course of five weeks to allow the maximum number of residents to attend the workshop. Each resident attended only one session, and all sessions were identical. The same three faculty members, each of whom had extensive experience with EBM as well as with teaching, taught all sessions. During the month after the workshop, no residents worked with those faculty members on the inpatient services.

     Seventy-three (73) Internal Medicine and Internal Medicine/Pediatrics residents in their first through fourth year of training attended one half-day session each. All residents in both programs who were not on vacation or on Intensive Care Unit (ICU) rotations attended the workshop. Due to curricular constraints, residents could not be randomized to a control group.

     Anonymous pre-workshop surveys (Figure 1) were given to all attendees who arrived within the first 30 minutes of the workshop. These surveys focused on their attitudes towards the use of EBM on a regular basis and their self-reported ability to perform several of the skills central to EBM. A 5-point Likert scale was used for all items; several questions had opposing polarity to maintain internal validity.

     The workshop was designed using principles derived from adult learning theories. It was comprised of multiple components designed to allow for maximum resident interaction and immediate use of newly introduced skills. First, leaders engaged the residents in a frank discussion of the benefits and barriers to EBM use. This discussion was followed by a 30-minute presentation on question formation and advanced search techniques. Next, using a trigger tape, the group practiced question formation. They then divided into small groups and performed literature searches on the questions they had formulated. Each group presented their search question and results and received feedback from the instructors and their fellow residents.

     At the end of the workshop, participants completed the same survey again. They were also asked, but not required, to develop and record two personal goals designed to integrate EBM into their medical practice that they would like to achieve over the next month. These forms included participants’ names. Approximately one month after the course ended, each participant was paged by the research assistant and asked if they had met their goals. If they could not recall their goals, they were prompted using their original forms. Results were reported in group form to allow blinding for the Principal Investigator. Residents were considered lost to follow-up if they failed to respond to four pages on differing days.

The study received approval from the Institutional Review Board at the University of Minnesota.

      Analysis - The data was analyzed in conjunction with the Boen Biostatistics Laboratory at the University of Minnesota, using SAS, a standard statistical software package. We calculated descriptive statistics (mean and SD) for each question for both sets of surveys. Nonparametric two-sample Wilcoxon Rank-Sum tests were performed to compare differences in pre- and post-workshop survey results.

Residents’ self-reported success in achieving their personal goals was reported in tabular form only.

Results

      Attitudes - The attitudes of residents towards EBM use in daily practice were significantly more positive on the post-workshop surveys (overall scores 3.5 pre-course vs. 3.7 post-course, when 5=most positive, p=0.034). The differences were largest in areas that had been targeted during the workshop, such as literature searches and filter use during those searches. Results for each question are displayed in Table 1.

      Self-Reported Ability - Residents reported significantly more confidence in their EBM skills after the workshop (overall score 3.0 pre-course vs. 3.3 post-course, where 5=most skillful, p=0.006). There were statistically significant (p<0.05) increases in confidence in ability in five of the seven areas assessed in this section of the survey. Results are displayed in Table 2. One of the two domains that did not show a statistically significant improvement was ability to evaluate the quality of a study. It is noteworthy that this area was not discussed during the course. Overall, residents reported increased ability in five of the six workshop-related skills improved during the workshop, whereas they did not in the control skill of study evaluation.

      Goal achievement - All 73 residents were asked to set two goals as outlined in the Methods section. Sixty-six (66) residents outlined goals at the conclusion of the workshop. All residents who set goals were included in the analysis regardless of whether we were able to collect follow-up data from them or not. Sixty-two (94.0%) returned their pages. Results are detailed in Figure 2. Of the 66 residents who set goals, 48 (72.7%) reported that they met one or both goals; 14 (21.2%) did not reach either of their goals and 4 (6.1%) were lost to follow-up.

Discussion

     This study demonstrated a statistically significant improvement in the attitudes of residents regarding the value of EBM after just a 4-hour interactive workshop. The residents also reported more confidence in their use of EBM, especially in the skills targeted during the workshop. Finally, there was self-reported evidence of short-term behavior change at one month, as over two-thirds of residents reported meeting a personal goal with respect to the practice of EBM.

     Evidence-based medicine can be a powerful tool for physicians and their patients. It has become a part of many medical school and post-graduate curricula.⁸ However, only recently has attention become focused upon the evaluation of the effectiveness of these teaching efforts. Most attempts to date have centered upon skill-building outcomes, measured either directly or through learner self-reports.^14-17 Studies that assess attitudes towards EBM are often discounted. In one review evaluating the effectiveness of EBM curricula, studies were excluded “if they simply… used some form of "happiness index."¹¹

     But EBM educators aspire to change more than test scores. We want to facilitate life-long practice change in our students. To do that we must examine the theories of andragogy and behavior change and use them to help us design and evaluate curricula. The transtheoretical theory developed by Prochaska and DiClemente, used to assess patients’ readiness to change, is a useful model for this situation.

     As physicians, we understand through this theory that a patient must be ready to change in order to stop smoking or start exercising succesfully.^13-18 As educators, it is a lesson we would do well to learn. This theory has provided a framework to assess physicians’ readiness to change in areas such as cancer screening,¹⁹ and it may help explain why teaching critical appraisal skills does not always lead to behavior change.⁹ Adult learners must have an internal drive for sustained behavior change in order for significant behavior changes to be maintained. Fortunately, educators can stimulate learners’ need to learn and readiness to know.²⁰ It is therefore imperative that we attempt to monitor, evaluate, and, if necessary, influence our learners’ readiness to change.

     This study addresses whether a short course on EBM can affect residents’ attitudes, confidence, and behavior. It also relates behavioral change in the form of reaching participant-designed goals with those same attitude changes. This study assesses attitudes using a quantitative approach. In the future, it would be useful to conduct qualitative studies. For example, a qualitative analysis of the goals set by the students could provide useful information, much as a focus group does. Studies of this type would allow us to evaluate the attitudes held by EBM learners more directly as we strive to better design curricula and create a readiness for change.

     Several methodological improvements can be suggested, using the experience gained through this study. Pre- and post-workshop surveys were not matched and hence paired t-tests could not be performed. Additionally, matching surveys to the goal results would allow a “dose response” to be assessed, by analyzing whether those whose attitudes improved the most also were more likely to reach their goals. Confirming such a dose response would strengthen the causal link. Although I was able to overcome the selection bias inherent in the study of elective seminars, having a control group complete the same surveys at the same times would be useful.

     I used self-reports to assess residents’ behavior after the workshop, specifically concerning the goals they designed for themselves. Self-reported behavior, while used in many educational studies to examine behavior change²¹ can also be unreliable as a measure of clinical behavior. Objective evidence of goal completion would have been preferable, but was not possible at this time. Finally, the sample size did not provide the statistical power to detect small changes, but the study remains a useful starting place from which to design larger, more rigorous trials.

     Evidence-based medicine is hopefully here to stay, but we are a long way from the goal of successfully integrating it into medical school, post-graduate, and continuing medical education and facilitating sustained behavior changes in our learners. As we attempt to increase the effectiveness of curricula, we would do both learners and educators a service if we address and evaluate the desire of our learners to change. Integrating theories of andragogy and behavior change into curricular design is one of the keys for successful curricular development and sustained learner behavior change. Further research evaluating the impact of thoughtful uses of these educational and change theories will allow educators to design more efficient and effective curricula.

Acknowledgements.

     The author would like to acknowledge Bradley Benson, M.D. and L. James Nixon, M.D. for helping to teach the workshop. I would also like to thank Robert Kane, M.D. and the Clinical Outcomes Research Center at the University of Minnesota for their help with the data entry. Finally, I owe a special thanks to Ilene Harris, Ph.D. for her thoughtful review of the manuscript.

References

Reference

Baum K. The impact of an evidence-based medicine workshop on resident’s attitudes towards and self-reported ability in evidence-based practice. Med Educ Online [serial online] 2003;8:4. Available from http://www.med-ed-online.org

Correspondence

Dr. Karyn Baum
University of Minnesota
MMC 741,
420 Delaware St SE,
Minneapolis, MN 55455