Efforts Under Way to Improve Assessment of Operative Competency

To this end, Dr. Bhatti and colleagues identified 12 core procedures necessary for attaining competence as a primary surgeon (Figure 1) and developed assessment tools for each core procedure. The aim is to conduct pilot studies to implement and test these assessment tools for each procedure.

The tools they developed include a modified Objective Structured Assessment of Technical Skills (OSATS) instrument, a modified global assessment tool that assesses overall surgical performance based on 10 items, and a skills checklist for each procedure that allows formative feedback based on a number of tasks and subtasks, depending on the complexity of the procedure.

The first pilot study to examine these tools using two core procedures in a laboratory setting-mastoidectomy and endoscopic sinus surgery (ESS)-showed the validity, reliability, and feasibility of the global assessment tool and skills checklist to measure surgical competency.

According to Dr. Bhatti, these assessment tools will allow faculty to give formative and educational feedback, will help identify residents who need remediation, and will encourage faculty evaluators to establish benchmarks for satisfactory performance.

click for large version

Table 1. Otolaryngology Programs Using E-Value for Assessment

Implementation of Assessment Tools

One of the key issues in using the assessment tools is to make sure they are not too cumbersome for faculty to use. They [the tools] will need to be easy to use and they will have to capture the information that we want by accurately measuring operative competency in a meaningful way, said Robert Weatherly, MD, Associate Residency Program Director of the Department of Otolaryngology-Head and Neck Surgery at University of Kansas Medical Center in Kansas City.

One way to make these tools accessible is to add them to the online program E-value. This program would allow faculty to complete an online evaluation with 48 hours after an operation and also sends out an email reminder to nonresponders after 48 hours to complete the evaluation. It also allows faculty to provide a comment on, for example, why a student received a low score. Currently, a number of otolaryngology programs in the United States are using E-value for assessment (Table 1).

According to Dr. Weatherly, all these tools still require a lot of additional work before they can be implemented. Ultimately, a number of centers will need to gain experience with the use of tools to test their reliability and validity, and the American Academy of Otolaryngology-Head and Neck Surgery as well as the American Board of Otolaryngology will have to indicate their agreement with this idea before otolaryngology programs will want to implement them widely, he said.