Google
 
 		  

Evaluating a Residency Program Using Reflections of Recent Resident Graduates and their Pediatric Colleagues

Robert K. Kamei, MD* and Theodore C. Sectish, MD†

*Department of Pediatrics
University of California
San Francisco School of Medicine
San Francisco, CA.

†Department of Pediatrics
Stanford University School of Medicine
Palo Alto, CA.

Abstract: Background and Purposes: In response to the new Accreditation Council for Graduate Medical Education (ACGME) mandate for residency programs to use feedback to improve its educational program, we piloted a novel evaluation strategy of a residency program using structured interviews of resident graduates working in a primary care practice and their physician associates.
Methods: A research assistant performed a structured telephone interview. Quantitative data assessing the graduate’s self-assessment and the graduate’s clinical practice by the associate were analyzed. In addition, we performed a qualitative analysis of the interviews.
Results: Thirteen resident graduates in primary care practice and seven physician practice associates participated in the study. Graduate self-assessment revealed high satisfaction with their residency training and competency. The associates judged our graduates as highly competent and mentioned independent decision-making and strong interpersonal skills (such as teamwork and communication) as important. They specifically cited the graduate’s skills in intensive care medicine and adolescent medicine as well as communication and teamwork skills as important contributions to their practice.
Conclusions: The ACGME Outcomes Project, which increases the emphasis on educational outcomes in the accreditation of residency education programs, requires programs to provide evidence of its effectiveness in preparing residents for practice. Direct assessment of the competency of our physician graduates in practice using structured interviews of graduates and their practice associates provide useful feedback information to a residency program as part of a comprehensive evaluation plan of our program’s curriculum and can be used to direct future educational initiatives of our training program

Keywords: Medical education, residency training, evaluation, ACGME resident competencies, feedback

     In 1999, the American Committee for Graduate Medical Education (ACGME) mandated a transition in the accreditation process of residency training programs from the traditional emphasis on “structure-and-process” components to a focus on educational “outcomes”. The ACGME reasoned that simply verifying a program offered specified curricular components did not assure that the residents were actually achieving adequate levels of competency and that program quality was high. “Measuring program quality by examining structure and process is not a direct or complete measure of the quality of the educational outcomes of a program.”1

      Furthermore, the ACGME directed residency programs to use feedback from resident performance and outcome assessments in their evaluation of the educational effectiveness of the residency program. However, resident evaluations performed during residency training are still not a direct measure of the quality of their physicians and training program. We assume that if residents are determined to be competent during residency, they will be competent in actual clinical practice and their residency-training program has provided them with the skills they require for their career.

      The experiences of residents during residency training may not necessarily ensure their success in the work force. It has been stated by some that residency graduates of our medical educational institutions need at least one year of further training to be able to work in the “real world” of a generalist practice.2 While residents trained in large academic medical centers may have a great deal of experience caring for seriously ill children with unusual diseases, these training programs are often criticized for not teaching the practical skills required in a modern day clinical practice. These concerns about graduate medical education rarely include specific statements about what areas of “real life” practice that our graduates are well or inadequately prepared for.

      Therefore, care must be taken when using competency data of physicians in residency training to guide a curriculum designed for preparing physicians for clinical practice. To try to more directly assess our educational outcomes, we interviewed graduates from a university-based pediatric residency training program, who were working in a primary care setting. Although training programs have previously reported surveying their residents and resident graduates 3-11, we limited our inquiry to the evaluation of residents in primary care pediatrics practice one to two years after graduating from a residency program. We thought this group might have specific views on their training that were different from those of their colleagues who pursued other careers in pediatrics such as research or subspecialty practice. In addition, we obtained an independent “peer-review” opinion about our graduates and our training program from the unique perspective of a more established physician associate of the graduate. We are unaware of any other studies that asked an independent source to evaluate the quality of residency graduates and their training program.

Methods

      The Institutional Review Board of the University of California, San Francisco, approved the following study protocol. In August of 1999, resident graduates from the UCSF Pediatric Residency Training Program classes of 1997 and 1998 were sent a mailing requesting their participation in the research study. In addition, they were asked to identify a physician associate in their practice who could assess some of their clinical skills and would to agree to participate in the survey. Participation in the surveys was voluntary and no incentives for participating were offered. To be eligible to participate, graduates must have been working at least 60% of full time (0.6 FTE) in a primary care practice, and be employed by the practice for at least nine months. No further contact was made with the resident graduates if they returned the letter declining participation in the study.

      Structured telephone interviews were conducted with resident graduates and associates by an independent research consultant who kept their responses confidential. Interviews with our graduates were organized into two sections: practice demographics and graduates’ perspectives about their training. Resident graduates were asked to reflect on their residency training and rate themselves on a scale from 1 (least) to 9 (highly) on the following measures: “How prepared were you for primary care practice?” ”How satisfied were you with the entire training experience?” “Reflecting back to the time immediately after graduation, how competent did you feel to practice primary care?” and “How competent do you feel now?”

      Practice associates’ structured interviews included questions regarding the graduates’ competencies, as well as recommendations for residency training. They were asked to rate their colleague on a scale from 1 (not satisfied) to 9 (very satisfied) the following measures: “How satisfied are you with your colleague’s preparation for the work of your practice?” “How satisfied are you with the current overall competence of your colleague?” “How satisfied were you with your colleague’s ability to: Communicate with their patients and families? Communicate and work effectively with other members of the patient care team, such as subspecialty consultants, or social workers? See patients and manage care? Manage patients with chronic illness? Appropriately refer to subspecialists and to manage patients with the help of subspecialists?”

      Both quantitative and qualitative data were analyzed for this report. Descriptive statistics for continuous variables, including mean and standard deviations were calculated. Statistical analysis comparing the self assessment of each resident graduate’s present clinical competence compared to reflecting back on their competence after graduation was performed using a paired t-test, with significance at the p<0.05 level. The qualitative analysis of the structured interview data consisted of categorizing recorded answers into common themes, and noting each occurrence.

Results

     Resident Graduates Assessment - A total of 18 resident graduates were eligible for the study, out of 51 residents graduating from the program in 1997 and 1998. (Table 1) Thirteen resident graduates agreed to participate in the study. Of the 13 graduate participants, we were able to interview seven physician practice associates. The number of associates was less than the number of resident graduate participants for the following reasons: three graduates did not identify an associate; one associate was on leave from the practice at the actual time of the study; two practice associates could not be reached despite multiple attempts.

      Our surveyed graduates worked in a wide variety of primary care practice settings. Four resident graduates worked in a Health Maintenance Organization (HMO) setting and 4 worked in single-specialty practice group. (Table 2) Six resident graduates worked in practice setting with two to five partners. Three graduates worked in practices with six to 12 partners, while only two were in settings with greater than 12 partners. Two graduates, both working in an HMO setting, did not specify the number of partners or pediatricians in their practice.

      Resident graduates reported (on a scale from 1 [not] to 9 [very]) their preparedness to clinical practice as 6.96 and their overall satisfaction with their residency training as 7.27 (Table 3). Six graduates reported feeling more competent, one graduate felt less competent, and the remainder reported no change in competence after at least 1 year in practice. The differences in competence for each graduate were not statistically significant. (p< 0.14)


     Practice Associate Assessments - Practice associates rated our resident graduates very highly in all areas, ranging from a low mean score of 7.67 for efficiency to a high mean score of 8.92 communication with the patient care team (on a scale from 1 [not] to 9 [very]). (Table 4) The comparison of Graduates and Associate ratings of preparedness and competency are shown in Figure 1. In every instance, the mentor rated their colleague’s preparedness and competency at a higher level than the graduate’s self-assessed rating.

     Qualitative Analysis of the Structured Interviews - Several themes emerged from the interviews with graduates and their associates regarding the strengths and weaknesses of the training program.

      Graduate Interviews - Graduates identified hospital wards and ICU settings as venues of educational strength for the program. In particular, five of the 13 graduates cited the training in the Newborn and Pediatric Intensive Care Units as a strong training experience. Since some of the graduates had to cover the NICU and/or attend deliveries in their practices, the NICU experience provided excellent training for these circumstances. “A lot of nursery time (even NICU) helps my well baby experience.” Two residents specifically mentioned that they wished for more experience in the NICU setting; “ I would have liked more experience with resuscitation and with feeder/grower kids.” One resident practiced in a rural setting, and the ICU training substantially contributed to her ability to care for patients. Several residents expressed satisfaction with being able to treat very sick children in a variety of settings and circumstances. Three residents felt there were too many rotations in the Intensive Care Units. Even though there was a concern that too much time in the curriculum was spent in pediatric subspecialty areas, there was consensus with the value of the broad exposure of the training experience.

      Five residents expressed satisfaction with community rotations. One of the most frequently cited criticisms of the training program was the general pediatrics outpatient training. Eight graduates wished for increased exposure to “bread and butter” pediatric skills and routine well child visits. The continuity clinic experience in general was felt to be a positive experience, but it was criticized for not always reflecting actual general pediatric practice.

      Six of the graduates specifically cited deficiencies in their training of behavior and developmental pediatrics and experience with communication/mediation skills with patients and families. “There should be increased focus on mediation with families, need to learn how to ally with parents, how to reconnect with parents after a misdiagnosis and mediate feelings.” Training in dermatology, orthopedics, and minor trauma procedures, including suturing and casting were each mentioned as being deficient by one of the residents. (Table 5)

     Practice Associate Interviews: - When asked about the attributes that they valued most when hiring a new pediatrician, three of the seven associates spoke about the overall quality of the training experience the residents received. Many indicated they usually only hired from selected institutions that they believe provide a high level of training.

      Clinical acumen was mentioned by many, but perhaps surprisingly, associates focused their answers more on interpersonal skills, such as communication skills, flexibility, interaction with team members, work ethic, organizational skills, commitment, willingness to seek help, and a friendly personality. None of the associates expressed concern about their junior colleague’s lack of developmental and behavioral training. However 3 associates recognized that the “ability to identify a sick child” and strong nursery and intensive care skills were important attributes of the new graduate.

      Areas of “on the job training” for a beginning physician in practice were discussed with the associates. Although not specifically asked, five of the seven associates made reference to “independence-like” issues. They defined this skill as the graduate’s ability to rely less on tests and more on clinical judgment. This was fostered through the graduates’ increased exposure to the practice of pediatrics in their particular setting and more independent work on nights and weekends. Other important skills that employers thought were developed “on the job” included efficiency and time management, communication with patients and families, and counseling about behavioral and developmental concerns. Most associates thought residents should have more general pediatric outpatient training during residency to be better prepared for practice. Some believed that a rotation with a community pediatrician should be mandatory.

      The associates felt the skill of “learning the system” of the pediatric practice was one of the easier skills to acquire. They considered this to be something unique to each practice and probably not easily taught during residency. They suggested that residency programs should instead facilitate the development of flexibility and adaptability to work in many different work situations.

Conclusions

      This study represents our initial attempt to understand whether we are accomplishing our goal of preparing our graduates for clinical practice. A previous survey of graduates of pediatric programs reported that 81% felt that their formal medical education did a good or excellent job of preparing them for being a physician.6 Our study confirmed that our resident graduates had a high degree of satisfaction of our residency program and were doing well in their primary care careers. We chose to use peer evaluation to further assess our resident graduates and in the process, obtain a unique perspective of the residency program and curriculum.

      The interviews with practice associate suggested that communication and teamwork are highly valued skills in practice. Other attributes that are increasingly important for physicians, such as flexibility and adaptability, were also found to be important in our survey results.12 As a result, we are considering how to formally and informally emphasize these curricular issues in our program. Teaching modules about cultural sensitivity, practicing evidence-based medicine, continuous quality improvement, community medicine and patient advocacy have recently been required for pediatric residency training, but were not specifically mentioned by our graduates or their associates in this study.

      As with other surveys of pediatric resident graduates4,10, behavioral and developmental pediatrics training was frequently cited by our resident graduates as inadequate. Study participants also mentioned the importance of clinical learning in general pediatric settings that accurately reflect modern primary care practice during residency.

      We thought that our study participants would mention efficiency and time management problems more frequently. Practice associates generally did not expect new graduates to work at the same pace of more experienced practitioners and this was not commonly raised as a problem. At the same time, we found that efficiency was the lowest score given to resident graduates by the associates.

      Both graduates and their practice associates recognized that some of the graduate’s recently developed clinical skills were very important to their new primary care practice. Specifically mentioned were graduates’ intensive care experiences and knowledge of adolescent medicine. Although residents directed towards primary care careers may complain about the amount of Neonatal Intensive Care and Pediatric Intensive Care time in their training program, we learned that our graduate’s level of expertise in this area was seen as a very important skill. This insight has been a very helpful to our Residency’s Curriculum Committee and represents a perspective not always represented and considered. As residency programs think about implementing novel curriculum, we must not lose sight of maintaining the educational importance of these more traditional areas of pediatrics.

      Our residency program trains physicians who pursue many different types of careers, such as in subspecialty pediatrics, public health, basic science research as well as primary care. These different careers make it difficult for a single program to train physicians who can immediately step into the role of pediatric practitioners without a period of transition. Even the nature of each individual primary care practice is sufficiently different from others that time to adjust is inevitable. All but one graduate reported a difference in their level of competency from their residency graduation to the end of their first year of practice. However, most new hires, whether in medicine, business or other industries, would report some amount of “start-up” time necessary to feel comfortable. As Smith argues, “to try to reproduce the learning opportunities of the first year (in practice) at any other time in medical education is unnecessary and unlikely to succeed.” 13

      This study was limited by the small numbers of our graduates who were immediately engaged in a full time pediatric practice. In addition, the number of physician associates identified by our graduates further limited the sample size of that part of our study. Although the physician associate may have a biased viewpoint on the graduate, as we have found in this study, they can still provide important information. This evaluation technique, could serve as part of a 360-degree evaluation, a technique of evaluation being promoted in residency training to assess competency.14 360-degree evaluations are valid, however, when there are sufficient numbers (10-20) of peer and staff evaluators.15 These results should not be interpreted in isolation but rather as part of a comprehensive evaluation system of the residency program.

      We found our study methodology to be time consuming; obtaining the informed consent necessary for our research protocol from the very busy study participants and then arranging a convenient time for the telephone interview was difficult. However, the personal interview gave us very rich responses to open ended questions and was an important first step to developing future evaluations of our graduates in clinical practice. As we gain additional experience with this novel type of evaluation program, we will use different methods such as asynchronous surveying via the Internet to make it easier for our study subjects to participate and increase our response rates.

     The perspectives of pediatricians in practice and the reflections of resident graduates not only provided us with reassuring information about their preparedness for practice, but more importantly, we found this method provided us with information useful to guide the curriculum of our training program. The evaluation of our resident graduates in practice are another form of evaluation that should be used to assess the effectiveness of our pediatric residency training programs and allow program improvement to better suit the needs of our learners and the communities that they eventually serve.

Acknowledgements:

We wish to thank Dan Kaleba, E.D.D. who performed the data collection for the research project, the faculty members of the Pediatric Medical Education Center at UCSF Department of Pediatrics for their initial comments and support, and Dr. David Irby for his suggestions on the research design and written manuscript. This work was supported by an educational grant from the Departments of Pediatrics, University of California, San Francisco, and the Lucille Packard Children’s Hospital.

References

  1. Accreditation Committee for Graduate Medical Education, Outcomes Project Available at: (http://www.acgme.org/outcome/project/proHome.asp).
  2. Shine KI. Educating Physicians for the real world. In: Ginsberg E., ed. Urban Medical Centers: Balancing Academic and Patient Care Functions. Boulder, Colo: Westview Press; 1996:5-11.
  3. Nelson RL, McCaffrey LA, Nobrega FT, et al. Altering Residency Curriculum in Response to a Changing Practice Environment: Use of the Mayo Internal Medicine Residency Alumni Survey. Mayo Clin Proc. 1990;65:809-17.
  4. Taras H, Nader P. Ten Years of Graduates Evaluate a Pediatric Residency Program. AJDC. 1990;144:1102-1105.
  5. Kiel DP, O'Sullivan PS, Ellis PJ, Wartman SA. Alumni Perspectives Comparing a General Internal Medicine Program and a Traditional Medicine Program. J Gen Intern Med.1991; 6:544-52.
  6. Cantor JC, Baker LC, Hughes RG. Preparedness For Practice, Young physician’s views of their professional education. JAMA. 1993; 270:1035-40.
  7. Biro, FM; Siegel,DM; Parker,RM; Gillman,MW. A Comparison of Self-Perceived Clinical Competencies in Primary Care Residency Graduates. Pediatr Res. 1993;34:555-559.
  8. Parrino TA, Kern DC. The alumni survey as an instrument of program evaluation in internal medicine. J Gen Intern Med. 1994; 9: 92-95.
  9. Seelig CB, DuPre CT, Adelman, HM. Development and Validation of a Scaled Questionnaire for Evaluation of Residency Programs. South Med J. 1995;88: 745-750.
  10. Camp B, Gitterman B, Headley R, Ball V. Pediatric Residency as Preparation for Primary Care Practice. Arch Pediatr Adolesc Med. 1997;151:78-83.
  11. Macnab A, Martin J, Duffy D, Murray G. Measurement of how well a paediatric training programme prepares graduates for their chosen career paths. Med Educ.1998;32:362-6.
  12. Blumenthal D, Thier S. Managed Care and Medical Education. JAMA. 1996;276:725-727.
  13. Smith Laurence. First Year of Practice. Acad Med. 1996;71: 580-581.
  14. Accreditation Committee for Graduate Medical Education, Toolbox Available at:(http://www.acgme.org/outcome/assess/toolbox.asp).
  15. Wooliscroft JO, Howell JD, Patel BP, Swanson DB. “Resident-Patient Interactions: The Humanistic Qualities of Internal Medicine Residents Assessed by Patients, Attending Physicians, Program Supervisors, and Nurses.” Acad Med 1994;69(3):216-223.

Reference

Kamei RK, Sectish TC. Evaluating a residency program using reflections of recent resident graduates and their pediatric colleagues. Med Educ Online [serial online] 2003;8:6. Available from http://www.med-ed-online.org

Correspondence

Robert K. Kamei, M.D.
Department of Pediatrics
UCSF Box 0110
University of California, San Francisco
San Francisco, California, 94143-0110

Telephone: (415) 476-9185
Fax: (415) 476-4009
Email: kamei@itsa.ucsf.edu

 

Medical Education Online Editor@Med-Ed-Online.org