Michael H. Malloy, M.D.,M.S.
Department of Pediatrics
University of Texas Medical Branch
Galveston, Texas

Abstract - Background: Distance education methods have taken on greater importance as medical student education has moved off campus into the community. What the best methods are for conveying information to students at distant sites has not been determined.
Objective: To determine if students at distant community sites who received an otitis media study case by e-mail that was enhanced with a referral to a web-based otitis study case, performed better on otitis OSCE stations than students who received the same case not enhanced with visuals or referrals to a web-based otitis case.
Design/Methods: Students were randomized by community site to receive either the enhanced (E) or simple otitis study case (S). Students were e-mailed an otitis media study case during the 5th week of the rotation. Those randomized to the E-case received a case that started with a case scenario followed by a "Task" that instructed them to go to this web address: http://www.aap.org/otitismedia/www/vc/ear/index.cfm (American Academy of Pediatrics Otitis Web Site). They were then to select "Case 1" which was a continuation of the case scenario present on their e-mail. A list of learning objectives was also printed on the e-mail. Students receiving the S-case viewed the same case scenario and objectives, but were not instructed to go to the web page. All students rotated through two OSCE otitis stations. In the first station they interviewed a simulated patient(OSCE-SP) and counseled her on the management of her 12 month old with otitis. Within that station they viewed a video of a pneumoscopic exam of two ears, one ear with otitis and the other ear normal. At the 2nd otitis station the student presented the case to a faculty and was asked a series of questions about otitis media(OSCE-PR). Scores on the two stations were compared by group.
Results: There were 198 students who took the OSCE. 178 (90%) responded to a survey that indicated they had opened and read the e-mailed case. There were 87 students in the E-group and 91 in the S-group. The mean ±s.d. OSCE-SP station score for the E-group was 72.6 ±12.0 vs 75.4 ±9.8 for the S-group, p=0.09. For the otitis presentation station the scores for the E-group and S-group were respectively, 82.9 ±9.6 and 83.7 ±9.4, p=0.55.
Conclusions: These data suggest that enhanced visual distance education cases may offer no distinct advantage over simple paper-type case study guides.

Keywords: Distance education, medical education, web-based education.

     Distance education methods have been proposed as a means of delivering a more standardized curriculum to medical students participating in off-campus education programs in the offices of community preceptors.¹ Little research, however, is available concerning the best method of delivering distance education. The greater availability of more powerful computer hardware and the ability to connect distant sites through the internet has allowed for the development of stimulating visual and auditory educational materia^.2,3,4,5,6 Whether or not this type of educational material provides a better educational experience over and above its novelty has not been proven.

     Although extensive research does not exist evaluating the best methods for the delivery of distance education, extensive research has been attempted to evaluate the merits of computer-assisted instruction (CAI).^7,8 Friedman in his review of the "Top Ten Reasons the World Wide Web May Fail to Change Medical Education", suggests that the failure of the fulfillment of the promise of the Web is a function of failure of fully integrating CAI into the medical school curriculum, failure of testing students on material provided in CAI, and poor design of CAI. Computer assisted instruction relies on hypermedia. Hypermedia documents contain links to other pieces of text, as well as to other media, eg sounds and images. Research on hypermedia is also disappointing and as suggested by Dillon, "breakthroughs promised by hypermedia advocates are more mythical than real".⁸

     Nevertheless, advocates of CAI in medical curricula suggest that this form of education may have advantages. Based on experience with the PlanAlyzer program, Lyons et al. agree with the concept that, "computer technology has the power and capacity to facilitate independent learning and to teach problem solving unmatched by any medium other than real experience with patients".⁹ PlanAlyzer was designed to teach clinical problem solving in diagnosing anemia and coronary artery disease to second year medical students. The objective of the current study was to determine if third year medical students exposed to CAI that incorporated interactive teaching that would stimulate clinical problem solving was associated with higher scores on content related Objective Structured Clinical Exam stations than students exposed to CAI that did not offer an interactive process.

Methods

     The University of Texas Medical Branch at Galveston, implemented a Multidisciplinary Ambulatory Clerkship (MAC) in 1996 to provide students the opportunity to spend equal amounts of time over a 12 week period in the offices of community physicians in the areas of Family Medicine, Internal Medicine and Pediatrics. Over the ensuing years, a series of Pediatric cases were developed that consisted of the presentation of a case scenario, learning objectives for the case and references. One case a week was e-mailed to the students for the first 10 weeks of the 12 week course. There were no visual components or internet links within these cases. In 1999 one of the Pediatric cases, the otitis media case, was "enhanced" to include an internet linkage to what was originally the University of Colorado otitis web site. The web site has now been adopted by the American Academy of Pediatrics and is found at the following address: (http://www.aap.org/otitismedia/www//vc/ear/index.cfm). The case scenario of the original otitis media case, "simple Case", was modified to present the same scenario as Case 1 of the University of Colorado Otitis Media site. The learning objectives remained the same for the "simple" and the "enhanced" cases. For the 1999-2000 academic year community sites to which the students were assigned for the 12 week rotation were randomized to receive either the "Simple" or "Enhanced" otitis media case. Randomization was accomplished by alphabetizing by site and alternating assignments to the "Simple" or "Enhanced" case by site. The "Enhanced" case offered the following advantages. First, the case study was interactive requiring the students to respond to questions that concerned whether or not the patient should be seen in the office; what the diagnosis was, based on visualization of a video of a moving tympanic membrane; what the appropriate treatment would be; and the impact of the decisions on the cost of the care of the patient. At the conclusion of the case the student was scored on the choices that were made and information was given concerning the rationale behind the most appropriate selections. Students were e-mailed the introduction to the respective case to which their site had been randomized during the fifth week of the rotation (Appendixes A and B ). Attached to the e-mail was a survey requesting information about the case they had reviewed (Appendix C). Only students responding to the survey as an indication of their review of the cases were used in the analysis.

     During week 10 of the MAC, students return to the Galveston campus to take an Objective Structured Clinical Exam (OSCE). One of the 10 exam stations is an interview with a simulated patient acting as the mother(OSCE-SP station) of a 12 month old child that has otitis media. The student has 10 minutes to conduct the interview. Within that station the student views a video that consists of a pneumoscopy of a normal right ear and pneumoscopy of an ear with acute otitis media. The simulated patient evaluates the student on interviewing style, specific content items, and the management counseling provided by the student. At the next station, students present the interview, their assessment, and a management plan to a Pediatric faculty (OSCE-PR station). The student is evaluated by the faculty for presentation style; specific content items of the interview; and counseling content. The faculty then asks the student a series of 7 questions related to the clinical diagnosis, management, pathobiology of otitis media, and costs of antibiotics. A grade based on a 100 point scale is generated for each station.

     Statistical analyses were conducted using SAS. Testing for significant differences between means was conducted using t-tests. A general linear model was used for testing for significant trends across means. Analysis of variance was used to examine scores for the independent contribution of variance for group and term variables and for a group-term interaction variable. Chi-square tests for general association were used for testing for association of categorical variables. A p-value of 0.05 was arbitrarily assigned to be the level of statistical significance.

Results

     One hundred and ninety-eight students rotated through the MAC during the academic year 1999-2000. Of these, 178 (90%) responded to the e-mail survey indicating they had opened and read the e-mailed otitis case. There were 91 students who were e-mailed the simple case and responded, and 87 students who were mailed the enhanced case and responded.

     The mean s.d. score for the otitis-simulated patient station for students receiving the simple case was 75.4±9.8 compared to 72.6±12.0 for the enhanced group (Table 1). The difference in scores between groups approached but did not attain statistical significance (p=0.09). The differences in means between groups on the Otitis-presentation station and the Overall OSCE scores were much closer and did not approach statistical significance. The clinical performance grade, the written exam grade and the overall grade were examined by group and did not show any significant differences (Table 1).

     The otitis interview station scores were examined by group and academic year term. Although there was a significant difference in mean scores between groups for term 1, terms 2-4 did not show any differences (Table 2). There was a significant trend for increasing grades for each group over the course of the academic year (Table 2). The results of a general linear analysis of variance model showed there to be a significant trend for scores over the 4 terms (p<0.01), no significant differences in scores by group (p=0.09) and a nonsignificant group-term interaction variable (p=0.12).

     The student survey results are reported in Table 3. There were no significant differences in student responses by group for any of the questions. In response to the question about whether students found the case to be helpful in guiding their study, about 40% from each group strongly agreed with the statement. Slightly more of the students receiving the enhanced case agreed with the statement than students receiving the simple case. More students who received the simple case appeared uncertain about the helpfulness of the case in directing their study. About 90% of both groups strongly agreed or agreed that the time required to work the case was acceptable. More of the students receiving the enhanced case tended to disagree with the statement about the case not posing technical difficulties in accessing it (13.8%-enhanced group vs 8.5%-simple group).

Discussion

     The technology used for distance education has rapidly progressed in the last 10 years and has been recommended as a valuable resource to be incorporated into medical education teaching programs.^2-6 Basic science courses in medical schools appear to have taken the lead in this process compared to clinical clerkships. The Background Paper 2 of the Medical School Objectives Project noted the discrepancy between basic science course and clinical clerkship application of education technology.¹⁰ These authors suggested that the reason for this observation, "reflects a tendency on the part of the clinical faculty to maintain more traditional approaches to the education of medical students during the clerkship experiences." That report advocated that the discrepancy deserved special attention by deans and faculties.

     Although the research accomplished evaluating CAI and hypermedia in education in general has proven to be disappointing, the quantity of research done in the medical education arena, particularly as it relates to distance education is not overwhelming in volume.⁸ If, as Lyon suggests, CAI has particular advantages in teaching problem solving, further research into CAI should be a high priority.⁹

     Several recent studies report their experience with Internet-based teaching, but they fail to test whether or not student's performance is different from students using less sophisticated educational methods. Bacro et al, describe the development of an Internet-site for a review of the knee joint.¹¹ There is, however, no evaluation of its usefulness. Several reports describe the introduction of case-based education through the Internet, but are preliminary reports without an evaluative component.^3,6

     Metha et al, described an evaluation of an Internet-based oncologic teaching program for second year medical students with findings similar to ours.⁴ Students were asked to volunteer to participate in an experiment in which they would be randomized to a traditional learning module on lung cancer or an experimental web-based lung cancer module. The traditional module consisted of a one-hour lecture and three-hour case-based faculty panel discussion. The experimental module consisted of all the information in the course book and the slides used in the panel discussion placed on the Web. Pre and post-exposure tests were given. Forty-three students participated in the experimental group and 62 in the traditional group. There were no differences in the pre- and post-test scores between the groups. Both groups demonstrated an increase in scores from the pre-test to the post-test. The traditional group had an opportunity to review the experimental Web material prior to the final exam. A questionnaire after both groups had viewed the Web-based material found the students in the experimental group more enthusiastic about using the Web, but with complaints about the length of time required for downloading.

     Papa and Meyer implemented a CAI tool intended to improve clinical problem solving skills concerning chest pain.¹² They randomized 83 osteopathic students rotating through an Emergency Department clerkship to receive CAI or not. The CAI was intended to provide during the rotation exposure to 27 cases of chest pain requesting development of a chest pain protocol and providing feedback. The outcome measure was the development of a diagnostic chest pain protocol at the end of the rotation. The authors reported a higher score by the CAI group on the diagnostic protocol. There was no type of alternative intervention offered to the other students during the rotation. Thus, this study does not provide a comparison of different types of CAI.

     There are several limitations to the current study. There is no guarantee that the group receiving the "Simple" case did not see the otitis web-site. It is, however, extremely unlikely that this occurred. The randomization process was set up by site so that any exchange of information between students at the same site would be about similar e-mailed experiences. Sites were separated by at least 10 miles. We cannot, however, rule out that students surfing the web may have stumbled upon the site. Nevertheless, such a random happening appears highly unlikely. Another argument for the lack of difference in performance between groups is that the OSCE did not test for information that was unique to the "Enhanced" case web-site. The counter-argument is that the provision of unique information to one group and not the other, would simply be a test of whether or not the students read their e-mail. The intent of this project was to determine if students received any benefit from a visually stimulating interactive way of learning that provided similar content information that could have been obtained from reading. Friedman argues that one of the reasons that the World Wide Web may fail to change medical education is that computer-based educational material on the Web does not fully exploit the problem-solving and visual aspects of the medium.⁷ Here, in fact, is a study that attempts to compare a problem-solving and visual format offered on the Web to a simple informational content conferring method on the Web. The results of this study suggest that even when the question is formulated in a way that should exploit the advantages of the Web, no advantage is observed. Another argument for the lack of difference between groups is that the students learned what they needed to know for the OSCE from their preceptors and that the e-mailed cases were superfluous. If that is the case, then significant savings can be achieved by not having to create either "Simple" or "Enhanced" cases.

     The knowledge-enterprise industry is a major growth industry. Current spending in this industry is estimated at $735 billion and includes textbooks, software and services,13 Expenditures for the online component of this industry were estimated at $3.6 billion in 1999 and are expected to grow to $25.3 billion by 2003. The expenditure for higher education, college and above, is anticipated to grow from $1.2 billion in 1999 to $7 billion by 2003, a 483% increase in spending in 4 years. In the year 2000, $2.7 billion in venture capital went into educational companies and $2.5 billion (94%) went to electronic learning initiatives.¹⁴ Whether or not internet based, online, or electronic learning is less effective; is as effective; or is more effective than conventional educational methods, the industry is moving educational institutions into an electronic educational format. Educators thus must attempt to determine where this visually exciting and entertaining form of communication is most cost effective in the education process.

References

1. Malloy MH and Speer A. A comparison of performance between third-year students completing a pediatric ambulatory rotation on campus vs in the community. Arch Pediatr Adolesc med 1998;152:397-401.
2. Maulitz RC, Ohles JA, Schnuth RL, Lipsky MS, and Grealish RJ. The CyberDoc Project: using portable computing to enhance a community-based primary care clerkship. Acad Med 1996;71:1325-1328.
3. Ivanovic B, Koethe S, and Drogull S. Computer applications in medical education. Acad Med 1997;72:415-416.
4. Mehta MP, Sinha P, Kanwar K, Inman A, Albanese M, Fahl W. Evaluation of internet-based oncologic teaching for medical students. J Cancer Educ 1998;13:197-202.
5. McNulty JA, Halama J, Dauzvardis MF, and Espiritu B. Evaluation of web-based computer-aided instruc-tion in a basic science course. Acad Med 2000;75:59-65
6. Rawn C, Davidon R, and Meier A. Using webbased case presentations to supplement a surgery clerkship curriculum. Acad Med 2000;75:540.
7. Friedman RB. Top ten reasons the world wide web may fail to change medical education. Acad Med 1996;71:979-981.
8. Dillon A and Gabbard R. Hypermedia as an educational technology: a review of the quantitative research literature on learner comprehension, control, and style. Review of Educational Research 1998;68:322-349.
9. Lyon HC, Healy JC, Bell JR, O'Donnell JF, Shultz EK, Moore-West M, Wigton RS, Hirai F, and Beck JR. PlanAlyzer, an interactive computer-assisted program to teach clinical problem solving in diagnosing ane-mia and coronary artery disease. Acad Med 1992;67:821-828.
10. Moberg TF and Whitcomb ME. Educational technology to facilitate medical students' learning: back-ground paper 2 of the medical school objectives project. Acad Med 1999;74:1145-1150.
11. Bacro TRH, Murphy JC, and Reeves BA. Integration of an internet anatomy review of the knee joint in a rehabilitation sciences curriculum. J Allied Health 1997;26:159-162.
12. Papa FJ and Meyer S. A compter-assisted learning tool designed to improve clinical problem-solving skills. Ann Emerg Med 1989;18:269-273.
13. Grimes A. The hope and the reality. Wall Street Journal, March 12,2001. p R6.
14. Bransten L. Something ventured: where the big-money investors are placing their bets. Wall Street Journal, March 12, 2001. p R8.

Acknowledgement

The author would like to thank Dr. Steve Berman, a creator of the University of Colorado Web-Based Otitis Media Cases, for allowing the use of these cases for this project. This study was presented in part at the Annual Meeting of the Council on Medical Student Education (COMSEP), March 2001 and at the Pediatric Academic Societies Meet-ing, April 2001.

Correspondence

Michael H. Malloy, M.D., M.S.
Department of Pediatrics
University of Texas Medical Branch
Galveston, Texas 77555-0526
Phone: (409) 772-2815
FAX: (409) 772-0747
E-mail: mmalloy@utmb.edu

Appendix A

1. Electronic-mail letter sent to students receiving the "Simple" case:

Dear MAC students:

Listed below is the pediatric case for Week 5. Please fill out the Evaluation at the end of the end of the case and return it to Dr. Malloy.

WEEK 5 CASE:

It is 8:30 a.m. Monday, February first, and after arriving at your office the receptionist transfers a……… What pertinent historical and physical exam findings would help you make the diagnosis of a simple upper respiratory infection vs acute otitis media vs sinusitis?

LEARNING OBJECTIVES

1. Be able to differentiate by history and physical exam findings the various types of common up-per respiratory problems including otitis media.
2. Know the various microbiologic causes of upper respiratory disease in children including otitis media.
3. Have a basic understanding of the management of upper respiratory problems in children includ-ing otitis media.

REFERENCE

1. Nelson's Essentials 3rd. edition: pp 341-5 and pp 459-489.
   

APPENDIX B

2. Electronic-mail letter sent to students receiving the "Enhanced" case:

Dear MAC students:

Listed below is the pediatric case for Week 5. Please fill out the Evaluation at the end of the end of the case and return it to Dr. Malloy.

WEEK 5 CASE:

It is 8:30 a.m. Monday, February first, and after arriving at your office the receptionist transfers a……… What pertinent historical and physical exam findings would help you make the diagnosing of a simple upper respiratory infection vs acute otitis media vs sinusitis?

1. Task: Now go to the following Web Site and select Case 1 which will provide you with the above introduction and then direct you through a series of questions and management decisions about the case. You may view other cases on this Web page. They will also help you in understanding, diagnos-ing, and managing otitis media in children. http://www.aap.org/otitismedia/www/vs/ear/index.cfm.

LEARNING OBJECTIVES

1. Be able to differentiate by history and physical exam findings the various types of common up-per respiratory problems including otitis media.
2. Know the various microbiologic causes of upper respiratory disease in children including otitis media.
3. Have a basic understanding of the management of upper respiratory problems in children includ-ing otitis media.
   

REFERENCE

1. Nelson's Essentials 3rd. edition: pp 341-5 and pp 459-489.
   

Appendix C

Survey questions sent to students with electronic mail.

I found this case helpful in guiding my study activities.

a. ___ Strongly agree
b. ___ Agree
c. ___ Uncertain
d. ___ Disagree

2. The length of time required to work on the case was acceptable.

a. ___ Stongly agree
b. ___ Agree
c. ___ Uncertain
d. ___ Disagree

3. The case did not pose any technical difficulties in the process of accessing it.

a. ___ Strongly agree
b. ___ Agree
c. ___ Uncertain
d. ___ Disagree