Patrick D. Bridge, PhD^*, and Kenneth A. Ginsburg, MD^†

^*Department of Family Medicine
^†Department of Obstetrics and Gynecology
Wayne State University
Detroit, MI

Abstract - During the clinical phase of undergraduate medical education (UME) students are often geographically dispersed and assigned to preceptors throughout the community. Monitoring, documenting, and evaluating their clinical experiences and achievement of clinical objectives in this venue becomes a challenge, especially for large UME programs. The purpose of this manuscript is to discuss a method for developing and implementing a school-wide evaluation system for the clinical phase of UME. This type of evaluation system links students' clinical experiential data with the objectives of a clerkship, using technological advances, such as the Personal Digital Assistant (PDA), Internet, and intranet. Clerkship directors are provided real-time reports on student's progress toward achieving clerkship objectives and are able to monitor the clinical activities of the clerkship. Students on the other hand, will be empowered to take more control of their educational experiences by monitoring their own progress.

Background and Rationale

     Evaluating students' achievement of educational goals and objectives during the pre-clinical years of undergraduate medical education (UME) is a relatively clear-cut process, compared to evaluating the clinical phase of UME. During the pre-clinical years, information is primarily delivered in large and small group lectures and labs. These settings are more conducive to a traditional education process, which allows a course director to coordinate the delivery of curricula specific to course objectives, and evaluate students on the core of information delivered.

     During the clinical phase of UME, the clinical setting is the classroom and students' experiences become the primary mechanism for achieving clinical objectives. Unlike the controlled environment in the pre-clinical years, during the clinical phase students are often geographically disbursed and assigned to preceptors throughout the community. Monitoring and evaluating their clinical experiences and achievement of clinical objectives in this venue becomes a challenge, yet is a critical component to the clinical phase of UME. If a clerkship director is unaware of the experiences students encounter, or if students are not exposed to a required experience, how can students be held accountable for achieving the clerkship objectives?

     Students' exposure to a required experience during the clinical phase of UME does not assess clinical competency, but documenting and monitoring those experiences remains a major component in the education and accreditation process.^1-7 For a clerkship director, the information can provide a framework for establishing continuity between the clinical objectives and test content (content validity). If a clerkship director is confident that students are exposed to the required experiences, he or she can develop a test based on those experiences. If students are not exposed to a required experience, a clerkship director may be required to modify the curricular objectives or pursue alternate methods (e.g., simulated patient) to ensure exposure and validity of the testing process.

     Documenting the clinical experience also gives students the necessary data to reflect on the clinical encounter. Reflection requires the student to retrospectively review the learning encounter. Westberg and Jason⁸ view reflection as a component of collaborative education, which both the teacher and learner share a common goal to ensure that learner goals are achieved. Because of the vast experiences a student may encounter in a given day and the demands of his or her physician-teacher, reflection cannot always occur immediately following an encounter. Documenting clinical experiences enhances students' ability to reflect at a later time.

     Curriculum administrators must also ensure that clinical educational goals, as defined by the medical school and carried out through students' clinical experiences, are monitored and achieved. If goals are not being met, interventions may be required. In fact, according to a standard from the Liaison Committee on Medical Education (LCME): 7

A system for monitoring the achievement of clinical educational goals must be developed, and students must be evaluated in this framework. If the level of diversity of student interactions with patients does not meet the school-based criteria, specific mechanisms must be in place to adjust the criteria or to alter the educational program. Either may be done only within ensure continued educational quality. (p.12)

     Integrating an evaluation system into a medical school to document and monitor students' clinical experiences is important for the valid assessment of student performance, to enhance the educational experience for the student, and to monitor students' achievement of clinical goals and objectives. The purpose of this manuscript is to discuss a method for developing and implementing a school-wide evaluation system for the clinical phase of UME. It is our goal that medical school administrators and clerkship directors confronted with this issue can learn from our experiences and strategies as they work to improve the evaluation of students' clinical performance.

Planning and Design Overview

     To successfully integrate a school-wide evaluation system for the clinical phase of UME, an "integrated" approach was proposed. This type of system centralizes students' experiential data within the school of medicine (SOM), but decentralizes the responsibility of data collection, data entry, data interpretation, and utilization of information to the clerkship director and student. An important component of the integrated approach is the linkage of students' clinical experiential data with the objectives of the clerkship using technological advances, such as the Personal Digital Assistant (PDA), Internet and intranet. Students are issued a PDA (hand-held computer) with a database containing defined criteria (data fields) that students collect during their encounters with patients. Students input the data and transfer the information, using a PDA modem, to a central location in the SOM. When the data are downloaded, standard reports are automatically generated which assess the scope of learning experiences, as well as students progress toward achieving clerkship objectives. Information is instantly uploaded to a secured web site for the clerkship director and student to review (see Figure 1).

Figure 1: Integrated Approach for Evaluating Students Clinical Experiences

     The structure of this system allows a clerkship director to review students' clinical experiences and progress toward achieving clinical objectives on a real-time basis, and it empowers students to take more control of their educational experiences by monitoring their own progress. This is a key factor for the success of the project, as well as a unique component that links today's technology to the educational process in the clinical phase of UME.

Development and Implementation

     Phase I - The development and implementation process consisted of four phases. Phase I assessed the efficiency of the PDA as a data collection tool. In the 1999-2000 academic year, a pilot project was initiated for undergraduate medical students in the Obstetrics and Gynecology (OB/GYN) and Family Medicine clerkships to test the small-scale functionality of the PDA to document students' clinical experiences. The Palm III Connected Organizer was chosen for the pilot project because of its ease-of-use, relatively low cost, open architecture and availability of end-user software to develop the database for input and storage of student experiences.

     Using Pendragon Forms Professional Version 2.0,⁹ a patient encounter database program was designed. The data fields included patient demographics, primary and secondary diagnoses and procedures, level of visit, degree of participation in the encounter, educational value of the encounter, time stamp (verify time and date of encounter), comment field, preceptor/site and student identification, as well as a preceptor signature field. The students collected data on their PDA's during their clinical experience and returned to the respective departments to download the encounter data. Students' placed their PDA into a cradle attached to a desktop computer and pressed a button on the cradle. The data was transferred (synchronized) from the PDA to a Microsoft Access database. At this time the database could be queried as needed.

     Ten students in the pilot completed the clerkship in Family Medicine and had recorded over 900 patient encounters, while 30 students in the OB/GYN clerkship recorded over 700 encounters. The downloading (synchronization) process was extremely efficient, with the average download of data taking between 15 and 30 seconds. After each download, students were given basic reports summarizing their clinical experiences.

     The process of using the PDA in the clinic setting generated common themes. Students found the PDA to be a more efficient method of collecting patient encounter data than traditional paper logs, were satisfied with the experience, and would recommend the use of the PDA for other students'. The use of handheld computers in Phase I was found to be an effective method for collecting students' experiential data from multiple locations; thus the implementation of phase II was initiated using the OB/GYN clerkship as the pilot.

     Phase II - The second phase of the project occurred in Spring 2001. This consisted of developing and operationalizing an educational strategy that would monitor students' clinical experiences and link the experiences to measurable objectives in the clerkship. This strategy was designed to identify an educational framework that was common and clinically relevant to all of the clerkships. A review of our clerkships' objectives identified common clinical themes: all students must have the skill to conduct a history and physical (H&P), experience patients with health conditions specific to the clerkship/specialty, apply critical decision making skills, and conduct/observe procedures specific to the clerkship/specialty.

     A set of four general clinical objectives was developed from this framework, which could be used as a template across all the clerkships (see Table 1). The objectives require students to: 1) conduct an H&P on a specified number of patients with conditions relevant to the specialty/clerkship; 2) apply the necessary clinical decision making skills with those patients; 3) conduct specific procedures relevant to the specialty/clerkship; and 4) observe specific procedures being conducted that are relevant to the specialty/clerkship. For each objective, students are required to achieve a specific participation level. For example, the required H&P's must be at least expanded problem focused, detailed, or comprehensive. And the student must be able to at least apply the necessary clinical decision making skills to diagnose the patient. Students' record the participation levels along with other patient encounter data in their PDA's. Additionally, students evaluate the value of the educational experience and document the educational level in the PDA.

     Phase III - The third phase of the project was implemented in the summer of 2001, which integrated the technical components of the evaluation system into the SOM infrastructure. This included the development of the front-end and back-end systems.

     The front-end systems involved the development of the database for the PDA and then testing the data input process and transfer of data using a 56k PDA modem from remote locations, to a centralized server in the SOM. The database consists of 15 fields, including patient demographics, patient diagnosis, level of clinical decision-making, procedures, educational value, as well as the participation rating scale associated with each clinical field.

     The back-end technology includes remote receiving, storing, and reporting of data, using Palm HotSync software, a dedicated Windows NT Server and Oracle database. The Palm HotSync server software is used to allow hundreds of PDA's to be synchronized to a central location with enterprise data and servers, including email, calendars, software programs, databases, and database servers.¹⁰ The HotSync server software will allow multiple databases to be downloaded to the students. Therefore, as students rotate through the clerkships, the clerkship specific database will be downloaded to their PDA when they perform a HotSync to the network from remote locations. The Oracle database was chosen for compatibility with the current technology in the SOM. Once in the Oracle database, a set of standard reports developed in a JSP (Java Server Pages) are automatically generated and posted to a secured website for the clerkship directors and students to view on a real-time basis.

     The reports were designed to measure three distinct yet interdependent areas. The first set of reports are global measures assessing the clinical encounters all students in the clerkship are experiencing (e.g., diagnoses, procedures, participation levels, etc.). The second set of reports are more detailed, and assess the types of clinical encounters at the site level. The final sets of reports are student-specific, evaluating their real-time progress toward achieving the clinical objectives of the clerkship.

To test the front and back-end systems, a mock OB/GYN patient database was developed representing 30 patient encounters. Algorithms to assess the validity of the reports were designed using the patient encounter data. The data was entered and downloaded from three remote locations using the PDA modems.

     The results from the phase III testing were encouraging, but not without problems. Fortunately, because 90% of the data fields consisted of drop down lists, the data was easy to enter in the PDA. The download process time was not as fast as using the desktop PC and PDA cradle. The average download time was approximately 2-3 minutes per 20 patient encounters. While the download time does not seem unreasonable, it may become burdensome for students in placements with large volumes of patients. Additionally, as expected in the first testing phase, there were numerous errors found in the programming of the reports. These errors were corrected and a second testing phase was initiated, which further validated the reports.

     Phase IV - The final phase will focus on the pilot of the evaluation system in the OB/GYN clerkship in the fall/winter 2001. During this phase, approximately 35-40 students will receive a pilot orientation program covering the use of PDA's and guiding policies, as well as the PDA unit and modem. Each PDA will contain a preloaded database with the necessary data fields to measure each clinical objective. Students will record their clinical experiences using the PDA and be required to download the data to the centralized server twice a week, for a minimum of sixteen synchronizations over a two-month period. From the server, standard reports will be generated and posted on the SOM Intranet, summarizing students' clinical experiences and progress toward achieving the clerkship objectives. Students will be required to access their personal reports and review their progress at least weekly. In order to maintain compliance during the pilot, participation is a mandatory for all OB/GYN students.

     A formative evaluation phase will also be conducted with the students, to identify any issues or problems they may be having during the pilot, as well to obtain their input into the advantages and disadvantages of the pilot program. Following the outcomes from OB/GYN pilot and any modifications to the program, the Family Medicine clerkship will be integrated into the PDA program. Upon successful implementation of Family Medicine, each clerkship/clinical rotation will be integrated in a stepwise manner.

Conclusion

     The purpose of this manuscript was to discuss our current experiences in developing an evaluation system using an integrated approach for monitoring students' achievement of clinical objectives and planned strategies to achieve this goal. Recent LCME standards are mandating that medical schools have a system for monitoring both the achievement of clinical educational goals and the equality of student experiences across clinical sites. Mandating an evaluation system is not unreasonable, and in fact, is a necessity which can benefit the SOM, clerkship director, preceptor, and more importantly the student. But the development and implementation of such a system is not a simple task.

     While students' exposure to a required experience does not in itself assess clinical competency, documenting and monitoring those experiences remains a major component in the education and accreditation process. Successful development and implementation of the evaluation system will provide clerkship directors and preceptors real-time feedback on students' progress toward achieving clerkship objectives. It will ensure they are being provided the necessary experiences to meet the objectives, through an automated link from student experiential data to a standard report format. Students, on the other hand will be empowered to take more control of their educational experiences by monitoring their own progress. This type of system could also improve collaboration, support, and communication between the clerkship director, student and preceptor, enhancing the quality of student education.

     Evaluating students' achievement of clinical objectives is a challenge for most medical schools, especially for large undergraduate medical education programs. It is our hope that medical school administrators and clerkship directors confronted with this issue can learn from our experiences and strategies. We hope our efforts in developing an evaluation system to monitor and evaluate students' achievement of clinical objectives will continue to enhance student's clinical education and improve the undergraduate medical experience for all involved.

References

1. Dolmans D, Schmidt A, Van der Beek F, Beintema M, Gerver WF. Does a student log provide a means to better structure clinical education? Med Educ 1999,33,89-94.
2. Kaprielian Johnson V, Michener JL. Tracking medical students' clinical experiences with a computerized medical records system. Fam Med 1994;26, 425-427.
3. Alderson T, Oswald NT. Clinical experience of medical students in primary care: use of an electronic log in monitoring experience and in guiding education in the Cambridge community based clinical course. Med Educ 1999; 33:429-433.
4. Ferrell BG. Demonstrating the efficacy of the patient log book as a program evaluation tool. Acad Med 1991:66:9, 49-51.
5. McVeigh S, Williams R. Pangaro L. An internet home page to log students' patient encounters. Acad Med 1997:72(5), 417-418.
6. Patricoski CT, Shannon K, Doyle GA. The accuracy of patient encounter logbooks used by family medicine clerkship students. Educ Research and Methods 1998:30(7), 487-489.
7. Liaison Committee on Medical Education. LCME accreditation standards web site. Available at: http://www.lcme.org/stndtext.htm. Accessed on June 17, 1999.
8. Westbert J, Jason H. Collaborative clinical education: The foundation of effective health care. New York: Springer Publishing, 1993.
9. Pendragon Software Corporation. Libertyville, IL. 1999.
10. Palm, Inc. Palm web site. Palm HotSync server software. Available at: http://www.palm.com/products/accessories/enterprise/server.html. Accessed on October 22, 2001.

Acknowledgement

This project was funded through the Office of Academic and Student Programs, Wayne State University School of Medicine.

Reference

Bridge PD, Ginsburg KA. An integrated approach for evaluating students' achievement of clinical objectives. Med Educ Online [serial online] 2001;6:9. Available from URL http://www.med-ed-online.org

Correspondence:

Patrick D. Bridge,
Wayne State University,
Department of Family Medicine,
101 E. Alexandrine,
2nd floor, Detroit, MI 48201

Telephone: (313) 577-5076
Fax (313) 577-3070
Email pbridge@med.wayne.edu.