Pediatric Resuscitation: Evaluation of a Clinical Curriculum
Joseph V. Dobson, MD*, David S. Brancati, DO†, Rollin Nagel, PhD‡
An effective team response to acute pediatric life-threatening situations requires experience, practice, self-confidence, and knowledge. However, in many areas of the country medical professionals have limited contact with unstable pediatric patients.1-7 This can be attributed to factors which have decreased infant and childhood mortality such as improved anticipatory guidance and preventive medicine, as well as a trend of regionalization of pediatric care into tertiary care centers.8,9 Pediatric and emergency medicine residents routinely have clinical rotations in pediatric emergency and critical care and receive training in pediatric advanced life support.10-13 Despite this training, they often express a lack of confidence in dealing with acute emergencies and a significant proportion demonstrate poor critical care procedural skills. 14-17 Additionally, recent trends in pediatric training have emphasized increased levels of resident supervision, which may further decrease a resident’s experience with the direct and independent care of the critically-ill child.14,17-19
To meet the obligations of resident education while at the same time providing sufficient supervision and quality patient care, residency review committees have required that training programs evaluate and certify that their trainees attain specific core competencies.11,20 This includes competency in pediatric critical care and emergencies. Appropriate supervision is necessary for quality pediatric care and education. However, it presents challenges for the educator seeking to give resident physicians experience and comfort with the ability to independently handle a critical situation.13
Pediatric mock resuscitations have been used in many institutions as a method to train health care providers, but there have been few studies to establish their utility or define effective teaching techniques.14,21,22 Educators seeking to establish competency-based teaching curriculums would benefit from access to the materials and methods of proven existing programs. In 1999, we developed a pediatric resuscitation curriculum utilizing principles of instructional design aimed specifically at the adult learner. The pediatric resuscitation curriculum was developed following the Instructional Systems Design (ISD) Model. 23-26 This model of instructional design involves defining a set of learning objectives that are based upon established curriculum goals or core competencies defined by the residency. Evaluation procedures are selected which best assess the acquisition of each learning objective. The goal was to provide instruction in management of life threatening situations, utilizing a variety of teaching techniques. The objective of this study was to assess the impact of this six hour pediatric resuscitation curriculum on a resident’s comfort in the evaluation and treatment of critically ill pediatric patients.
Study Design - The study was conducted retrospectively in conjunction with the evaluation of an educational intervention using a consecutive sample of residents rotating through the Pediatric Emergency Medicine Service. The study was approved by the St. Vincent Mercy Medical Center Institutional Review Board. Primary outcomes measures were pre- and post- intervention comfort levels in thirteen areas of pediatric resuscitation.
Study Setting and Population - Participants included a consecutive sample of pediatric, combined medicine/pediatric and emergency medicine residents that were participating in an established pediatric resuscitation curriculum from July, 2001 to June, 2002 at St. Vincent Mercy Medical Center. This is a community based, tertiary care referral center with an annual ED volume of greater than 70,000 visits per year, 25% of which are pediatric patients. All resident’s participating in the study had taken a pediatric advanced life support course within the previous two years.
Study Protocol - At their pre-intervention assessment, residents anonymously indicated their comfort levels in performing 13 areas of pediatric resuscitation. They then participated in a structured curriculum that included didactics, hands-on demonstrations and mock resuscitation scenarios (Table 1).
Each resident was required to demonstrate key procedural skills for pediatric resuscitation before continuing with mock resuscitation scenarios. During the simulated resuscitations, the students had the opportunity to take both the roles of leader and participant. The scenarios were designed to test the student’s knowledge of both the factual knowledge and procedure oriented learning objectives (Table 2) while working with a specific clinical encounter. After receiving the educational training, each participant was required to take a knowledge-based multiple choice test on pediatric resuscitation. Finally, each participant was evaluated on their knowledge of both the knowledge and procedural learning objectives via an oral examation. For post-intervention assessment residents evaluated their comfort level in each of the 13 pediatric resuscitation areas and the degree of effectiveness for each of the four educational methods (didactic lectures, procedural demonstration, mock code scenarios, oral exam) in helping them learn the objectives. Pre-intervention assessments were recorded on the first day of the monthly rotation in pediatric emergency medicine. The curriculum was given during the first 2 weeks of the rotation and post-intervention assessments, oral and written examinations were conducted during the last week of the monthly rotation.
Measurements - Data from the seven-digit Likert scale (from ‘Significantly Uncomfortable’ to ‘Significantly Comfortable’ with a ‘Neutral’ midscale) was collected at pre- and post-intervention assessments in each of the following thirteen areas of pediatric resuscitation: the ability to both lead and participate in the resuscitation of a critically-ill pediatric patient; resuscitation of respiratory failure requiring airway management; resuscitation of shock requiring fluid resuscitation; resuscitation of pulseless arrest (asystole); resuscitation of bradycardia with hypotension; evaluation and management of selected dysrhythmias (supraventricular tachycardia and ventricular fibrillation); performance of bag-valve-mask ventilation with and without chest compressions utilizing properly sized equipment and proper technique; performance of endotracheal intubation with rapid-sequence-induction utilizing properly sized equipment and proper technique; performance of intraosseous needle insertion utilizing properly sized equipment and technique; performance of synchronized cardioversion and asynchronized defibrillation of unstable cardiac dysrhythmias; selection and dosing of drug regimens used for rapid sequence induction for intubation; correct usage of a length based resuscitation tape; and termination of resuscitative efforts. Data from a four-digit Likert scale (significant, moderate, minimal, insignificant) was collected at the post-intervention assessment to determine the degree of effectiveness for each of the four educational methods (didactic lectures, demonstration of key pediatric procedural skills, mock resuscitation scenarios, oral exam) in helping residents learn the objectives. An objective measure of knowledge was documented as a passing score on the written exam (80% of questions answered correctly on a multiple choice test). Additional objective measures of performance included a final evaluation of each resident’s ability to fulfill the stated learning objectives (Table 2) during their participation in the mock code scenarios as well as an assessment of their ability to successfully perform all critical actions in the oral examination. Residents were scored on whether they ‘completely’, ‘partially’, or ‘poorly’ achieved each learning objective.
Data Analysis - Paired pre- and post-intervention comfort responses were analyzed using the Wilcoxon Sum Ranks Tests. Sign tests were used for comparing effectiveness between educational methods. Fisher’s exact tests were used to compare the number of residents with previous experience for each of the 13 pediatric resuscitation areas with type of resident (emergency medicine versus pediatric and medicine/pediatric residents). A Mann Whitney U test was used to assess the difference in the total number of pediatric resuscitation areas of previous experience with type of resident. Pearson correlations were calculated to compare 1) amount of previous experience and pre-intervention comfort levels for each of the 13 areas, and 2) year of residency and number of areas with previous experience. Cronbach’s alpha was calculated to assess the internal consistency of the 13-item comfort scale. All analyses were conducted using SPSS/PC 11.5 (SPSS, Inc., Chicago, IL).
All residents who rotated in pediatric emergency medicine from July, 2001 to June, 2002 participated in the pediatric resuscitation curriculum. No resident refused to participate or failed to complete the curriculum. A total of 31 residents completed the pediatric resuscitation curriculum and participated in this study. Twenty (64.5%) were either pediatric or medicine/pediatric (P) and the remaining 11 (35.5%) were emergency medicine (EM) residents. Seventeen (54.8%) were in their first year, nine (29%) in their second, and five (16.1%) in their third year of residency. While the study population represented a mix of training programs (pediatric, combined medicine-pediatrics, and emergency medicine), all residents were fairly homogenous in previous experience with pediatric resuscitation. The majority had little prior direct exposure to critically-ill pediatric patients and there was no relationship between reported experience and either type of residency or year of residency. Eighty-three percent had been directly involved in fewer than five pediatric resuscitations. Of the 30 residents reporting their previous experience, two residents indicated no previous experience in any of the areas and two residents reported experience in all 13 areas (median = 9 areas of previous experience). There was no relationship between the number of areas of reported experience and either type of residency (emergency medicine versus pediatric) or year of residency. Emergency medicine residents were more likely to have reported experience in equipment sizing using a length-based pediatric resuscitation tape than the pediatric residents (EM = 80%, P = 0%, Fisher’s exact test p < 0.001). The different residency groups did not differ in experience for any of the remaining 12 areas. There was an association between previous experience and the pre-intervention comfort levels for eleven of the thirteen pediatric resuscitation areas with significant correlations ranging from 0.38 to 0.68 (p < 0.04). The correlation between experience and pre-intervention comfort level was not significant for ‘resuscitation of respiratory failure requiring airway management’ (r = 0.32, p <0.09) and ‘resuscitation of shock requiring fluid resuscitation’ (r = 0.30, p <0.12).
The thirteen areas of pediatric resuscitation studied comprised the breadth of topics covered during the pediatric resuscitation curriculum, providing evidence of content validity for the areas evaluated. In addition, significant correlations between pre-intervention comfort assessments and their previous experience in the respective area provides concurrent validity evidence for these comfort levels. Residents reported an increase in comfort in both their knowledge and procedural skills in all areas studied (Table 3), demonstrated by the significant Wilcoxon Signed Ranks test results. There was high internal consistency (a =0.95) for the 13-item comfort assessment based on the residents’ pre-intervention responses. Before training more than 80% of the residents were comfortable in performing only one of the 13 areas, bag-valve-mask ventilation (83.9%). Only 35.5% were comfortable performing cardioversion/defibrillation. After the course, 80% or more of the residents were comfortable in performing every aspect of pediatric resuscitation except having the ability to lead a resuscitation (77.4%). All residents but one received passing scores on their written examinations (97%). During the mock resuscitation scenarios and final oral examination, all residents who participated in the course were scored on whether they ‘completely’, ‘partially’, or ‘poorly’ achieved each learning objective and critical action. 100% of the residents were assessed to have ‘completely’ met the learning objectives and critical actions
The majority of residents indicated that all four educational techniques utilized were ‘significant’ in their effectiveness in helping them learn the objectives: didactic lectures (71%), demonstration of key pediatric procedural skills (87%), mock resuscitation scenarios (97%), and oral exam (74%). Comparisons of effectiveness ratings between each of the educational methods indicated that residents rated the mock resuscitation scenarios as more effective than didactic lectures (p < 0.008) and oral exam (p < 0.04). The remaining educational methods did not differ in effectiveness for these residents. While didactic lectures were felt to be effective, it is interesting to note that a significant percentage of learners felt that the teaching strategies that encourage active participation and feedback were more effective in acquiring resuscitation knowledge and skill.
Expertise and comfort in dealing with life-threatening situations is gained through knowledge, experience, and feedback on performance. 1-3,14,27 Historically, residents training in pediatrics or emergency medicine gained much of their experience in resuscitation through direct exposure. However, studies show that most finish their graduate education without attaining sufficient knowledge and experience with the procedures and approach to the critically-ill child.12,15-17,19,28 A well-structured pediatric resuscitation curriculum helps residents gain expertise by simulating the critically ill child and allowing the health care provider gain experience prior to the actual event of a pediatric emergency. This study measures the effectiveness of this type of training by assessing whether it succeeds in increasing a resident’s comfort in their knowledge and procedural skills. We choose to assess comfort levels as the primary outcome measurement since it had been kept confidential and blinded during the course of instruction. The validity of the measurement scale was assessed by correlating pre-intervention comfort levels with a resident’s prior experience in each of the areas of pediatric resuscitation. Residents with less experience in each area ranked themselves as less comfortable in that aspect of resuscitation. Residents who are more comfortable with pediatric life and death issues are less likely to avoid these situations, and are more likely to gain the experience and expertise to manage the critically ill child.29
Few studies have evaluated the impact of dedicated instruction in pediatric resuscitation on the confidence, knowledge, or skills of residents training in pediatrics or emergency medicine. Capelle and Paul conducted mock codes with pediatric residents during their ward rotation.14 They compared residents who participated in the codes with those who did not (their control group). A survey assessing level of training, previous experience, and level of confidence in managing specific problems encountered in code situations was administered before and after the intervention. Their mock code program did not contain didactics, assigned reading, or specific evaluation procedures to document knowledge or procedural skills. Residents who participated in the resuscitation scenarios demonstrated an improvement in confidence in their ability to supervise a code, obtain IV access, and intubate during a medical emergency. They concluded that residency programs are not currently meeting the educational and confidence needs of pediatric residents. Nadel, et al studied the impact of a structured curriculum which included didactics, skill practice stations, and mock resuscitations.22 Residents who went through the curriculum had improved fund of knowledge, technical skills, and confidence in their leadership role when compared to residents who did not take the course. Our study confirms the conclusions of these prior studies and expands on the number and variety of resuscitation-related knowledge and skills evaluated. The curriculum used in this study can serve as a model for educators wishing to establish competency-based resuscitation courses that meet residency standards requiring documentation of key competencies, knowledge and procedural skills.
The pediatric resuscitation curriculum was developed following the Instructional Systems Design (ISD) Model. This concept of designing instruction for adult learners dates back to efforts of psychologists during World War II to develop effective procedures for training military personnel in the shortest time possible.23-26 The results reported in this study represent a part of the self-evaluation process of ISD and are the first to evaluate this model as an effective method of developing instruction aimed at teaching competency to in pediatric resuscitation. The process of instructional design involves defining a set of learning objectives that are based upon established curriculum goals or core competencies defined by the residency. Evaluation procedures are selected which best assess the acquisition of each learning objective. As an example, a learning objective requiring acquisition of specific points of factual knowledge might be tested with a simple multiple-choice question, while a procedural learning objective such as intraosseous needle placement is best evaluated via skill demonstration. Instructional strategies are developed only after the learning objectives and evaluation procedures are defined. These may include a variety of teaching techniques (slide presentation lecture, procedure demonstration, simulated patient encounters) and should focus on specific learning objectives. The effectiveness of the instruction is evaluated both by how well the residents succeed in attaining the learning objectives and by having the residents give feedback regarding the effectiveness of curriculum. Course instruction is revised based upon this evaluation process until the instruction reaches the desired standards of quality. The entire process is geared toward acquisition of a teaching strategy that is effective and can objectively evaluate a resident’s acquisition of the learning objective.
Limitations and Future Questions
The measures of resident performance reported in the study, while sufficient to show competency following completion of the course, do not document improvement directly attributable to course instruction. Inclusion of a pre-course written examination and assessment of procedural skills would have supplied stronger objective evidence of improved knowledge and procedural skills. Due to the structure of the curriculum, it would have been difficult to collect blinded data on demonstrated performance during mock resuscitation scenarios or oral examinations. Other than the written examination, measures of proficiency were based upon the instructor’s subjective assessment of an individual’s achievement during the scenarios, and so may be subject to reporter bias. Use of checklists would have more precisely document procedural expertise than the evaluation methods used in the study.
While each resident’s pre- and post-intervention assessments were kept strictly confidential, the process of instructional design requires that the teaching strategy be continually reevaluated and improved based upon resident feedback and an objective evaluation of its effectiveness. We made every attempt to keep the instructors blinded to the results of the self-assessments, but we could not blind them from the other evaluation tools used in the curriculum, nor could we inhibit the continual two-way feedback that is an important part of this teaching strategy. For these reasons, it was impossible to remain completely blinded to the learner’s self-perceived comfort levels, since this was often apparent by their demonstrated performance on the other evaluation methods. Nevertheless, the students were encouraged to answer the confidential self-evaluations carefully and honestly and were assured that their responses would have no bearing on their final evaluations. The structure and content of the curriculum was developed for a defined group of learners. It’s applicability to other health-care personnel of varying level and specialty is uncertain. Adaptation of the teaching methods reported in this study should be tailored to each individual group.
Research has not proven that improving a physician’s comfort in pediatric resuscitation through simulated patient encounters will translate to improved performance in actual resuscitations. It would be valuable to evaluate whether a resuscitation program results in any improvement in performance in actual resuscitations. A further limitation of our study is that it was not designed to evaluate retention of knowledge, procedural skills or confidence following completion of the course. Previous studies have shown that participants in advanced life support courses have significant deterioration of their skills and knowledge over time, returning to pre-training levels within 1 year.30-33 This would support the premise that a resuscitation program must be on going in order to be effective, but the exact timing and content of follow-up “refresher” mock resuscitations is currently undefined.
Implementation of a pediatric resuscitation curriculum improves pediatric and emergency medicine residents’ comfort with the evaluation and treatment of critically ill pediatric patients. This curriculum can be used in residency training to document the acquisition of core competencies, knowledge and procedural skills needed for the evaluation and treatment of the critically ill child. The results reported in this study support using this model of instructional design to implement educational strategies, which will meet the requirements of graduate education.
Abstract presentations of this data:
SAEM Midwest Regional Meeting, 9/13/02, Toledo, Ohio
Dobson JV, Brancati DS, Nagel R. Pediatric resuscitation: Evaluation of a clinical curriculum. Med Educ Online [serial online] 2003;8:15. Available from http://www.med-ed-online.org
Joseph V. Dobson, MD