Efforts Under Way to Improve Assessment of Operative Competency

At the 2007 annual meeting of the American Academy of Otolaryngology Head and Neck Surgery (AAO-HNS), investigators from Johns Hopkins University and the University of Kansas reported on a needs assessment that identified a number of issues related to improving the formal assessment of operative competency among otolaryngologists-head and neck surgeons. These issues included (1) the difficulty of defining surgical competency, (2) recognition that the number of surgical cases performed may not determine surgical competency in all residents, (3) the potential problem that residents who require remediation are often identified only late in their residency, and (4) the dilemma that program directors and department chairs want to certify surgical competency in their residents, but they fear legal and other ramifications if they do not sign off on those residents who are surgically challenged.

This needs assessment was in response to the mandate by the Accreditation Council for Graduate Medical Education (ACGME) that now requires all graduate medical education specialties to incorporate into training programs demonstrated competency in six major core areas. One of these core competencies is patient care, which includes surgical competency. The mandate was generated, in part, by the need to show demonstrated surgical competency to the public, health care payers, and accrediting bodies such as medical licensure boards.

Ultimately, a number of centers will need to gain experience with the use of tools to test their reliability and validity, and the American Academy of Otolaryngology-Head and Neck Surgery as well as the American Board of Otolaryngology will have to indicate their agreement with this idea before otolaryngology programs will want to implement them widely.

-Robert Weatherly, MD

At the 2008 annual AAO-HNS meeting, investigators reported on what has been done over the past year to develop and implement assessment tools for operative competency in otolaryngology trainees.

Development of Assessment Tools

According to Nasir I. Bhatti, MD, of the Department of Otolaryngology-Head and Neck Surgery at Johns Hopkins University School of Medicine in Baltimore, the main task for improving assessment of operative competency is to develop assessment tools that are practical and feasible.

Although he said that attempts in the past have been made at developing more objective measurements of competency, such as examining electromagnetic waves from surgeons’ hands or assessing surgeons’ eye movements, these types of measurements are not very applicable to resident programs or to the everyday life of a surgeon.

click for large version

Figure 1. These 12 core procedures are necessary for attaining competence as a primary surgeon. Assessment tools have been developed for each core procedure.

Therefore, Dr. Bhatti and his colleagues at Johns Hopkins are collaborating with investigators from Canada and the United Kingdom to come up with objective measurements that are both feasible and can be generalized to residency programs worldwide.

To this end, Dr. Bhatti and colleagues identified 12 core procedures necessary for attaining competence as a primary surgeon (Figure 1) and developed assessment tools for each core procedure. The aim is to conduct pilot studies to implement and test these assessment tools for each procedure.

The tools they developed include a modified Objective Structured Assessment of Technical Skills (OSATS) instrument, a modified global assessment tool that assesses overall surgical performance based on 10 items, and a skills checklist for each procedure that allows formative feedback based on a number of tasks and subtasks, depending on the complexity of the procedure.

The first pilot study to examine these tools using two core procedures in a laboratory setting-mastoidectomy and endoscopic sinus surgery (ESS)-showed the validity, reliability, and feasibility of the global assessment tool and skills checklist to measure surgical competency.

According to Dr. Bhatti, these assessment tools will allow faculty to give formative and educational feedback, will help identify residents who need remediation, and will encourage faculty evaluators to establish benchmarks for satisfactory performance.

click for large version

Table 1. Otolaryngology Programs Using E-Value for Assessment

Implementation of Assessment Tools

One of the key issues in using the assessment tools is to make sure they are not too cumbersome for faculty to use. They [the tools] will need to be easy to use and they will have to capture the information that we want by accurately measuring operative competency in a meaningful way, said Robert Weatherly, MD, Associate Residency Program Director of the Department of Otolaryngology-Head and Neck Surgery at University of Kansas Medical Center in Kansas City.

One way to make these tools accessible is to add them to the online program E-value. This program would allow faculty to complete an online evaluation with 48 hours after an operation and also sends out an email reminder to nonresponders after 48 hours to complete the evaluation. It also allows faculty to provide a comment on, for example, why a student received a low score. Currently, a number of otolaryngology programs in the United States are using E-value for assessment (Table 1).

According to Dr. Weatherly, all these tools still require a lot of additional work before they can be implemented. Ultimately, a number of centers will need to gain experience with the use of tools to test their reliability and validity, and the American Academy of Otolaryngology-Head and Neck Surgery as well as the American Board of Otolaryngology will have to indicate their agreement with this idea before otolaryngology programs will want to implement them widely, he said.

Nasir I. Bhatti, MD, said that the main task for improving assessment of operative competency is to develop assessment tools that are practical and feasible

Remediation, Not Termination

The ultimate goal of identifying residents who are surgically challenged is remediation. According to Douglas A. Girod, MD, of the Department of Otolaryngology-Head and Neck Surgery at the University of Kansas Medical Center in Kansas City, the goal is to graduate competent residents and to avoid punitive measures and stigmatization.

Remediation needs to be tailored to correcting the deficiencies particular to each resident. For example, a resident having difficulty with manual dexterity may be asked to practice working with their hands in an anatomy lab. If the problem is a matter of decision-making skill, the resident may be required to work more closely with one of the better teaching surgeons.

Whatever needs to be done, identifying the problem early and intervening early are key. You want to make sure you have a method of identifying surgically challenged residents early, said Dr. Girod. Whatever evaluation method you use, make sure it is designed to catch residents early, that you intervene early, and that you document everything.

Accurate and thorough documentation is particularly important in those rare and unfortunate cases where remediation is not possible and a surgical resident may be asked to switch medical specialties.

©2009 The Triological Society