Strategies for Evaluating Medical Residents

If you are an academic otolaryngologist, teaching probably comes naturally to you, but can you tell which of your residents are struggling? Over the last few years, educators have become more interested in identifying residents in need of extra attention long before the yearly Otolaryngology Training Examination (OTE) results come in and well before their five-year training periods are over.

In 1999, the Accreditation Council for Graduate Medical Education (ACGME) began requiring that all residency programs evaluate their trainees according to six core competencies: patient care, medical knowledge, practice-based learning and improvement, systems-based practice, professionalism, interpersonal skills, and communication. The requirements changed again in July 2014 when the ACGME began evaluating otolaryngology programs according to its Next Accreditation System, which uses outcome measures rather than duration of training as the basis for accreditation. Residents must now show competency in 16 areas that the ACGME has termed “milestones.”

These requirements spurred educators to begin thinking about new ways to evaluate residents, said Gregory Grillone, MD, professor and vice chairman of otolaryngology-head and neck surgery at Boston University School of Medicine, where he is also chair of the Graduate Medical Education Committee. “That reinforced our need to provide more frequent and useful feedback to residents that would help them improve in areas where they had deficiencies,” he said.

Dr. Grillone and other otolaryngology educators share their tips for evaluating residents below.

Test, Test, Test

A few years ago, to determine how well their residents were acquiring factual knowledge from didactic lectures, Dr. Grillone and his colleagues at Boston University decided to administer regular quizzes. Residents were given three to four multiple choice questions before and after lectures, which they answered using automatic transponders that collate data electronically. Dr. Grillone and his colleagues described their method in a paper recently published in The Laryngoscope (2014;124:E309-E311). (See “Benefits of Pre- and Post-Lecture Questions,” below.)

“The questions and answers would be embedded in my presentation. If they did the reading and still didn’t get the questions right at the beginning of the lecture, they would know the answer by the time the presentation was over,” Dr. Grillone said.

Scores were tracked over the course of an academic year and then compared with scores on the OTE from that year. Overall OTE scores demonstrated a significant correlation to the scores from the lecture-based quizzes.

The study also showed that residents who had previously scored poorly on the OTE but were then assigned an individual learning improvement plan that included weekly reading lists and regular meetings with assigned faculty mentors to discuss reading topics showed an improvement in scores on the next OTE exam.