Will Virtual Oral Board Exams Remain an Option After the COVID-19 Pandemic?

In the spring of 2020, as it was becoming clear to health providers, educators, and organizations that the rampant spread of COVID-19 was going to necessitate numerous changes in the name of public safety, some member boards of the American Board of Medical Specialties (ABMS) began to explore the possibility of temporarily converting their in-person oral board certification examinations to virtual formats. Candidate polls were taken, and pilot exams were run to test the waters.

The positive results of these, coupled with the harsh reality of a prolonged global pandemic, convinced several ABMS members to devise a virtual oral board exam for residents seeking their certification in 2020, 2021, and beyond.

Moreover, the unconventional testing method isn’t likely to go away any time soon. “Of the 14 [ABMS] boards that offer oral examinations, about one-third are committed to continuing with remote exams, one-third are leaning toward returning to in-person exams, and one-third are still considering the options,” reported Greg Ogrinc, MD, MS, senior vice president, certification standards and programs at ABMS. The American Board of Otolaryngology–Head and Neck Surgery (ABO-HNS) is among the boards that have taken the plunge, and as ABO-HNS executive director Brian Nussenbaum, MD, MHCM, notes, the shift to a virtual format has been both worthwhile and complex.

“The decision of changing from an in-person oral exam to a remote one was made in an intentional way after considering all of the factors involved,” Dr. Nussenbaum said. “What was certain at the time was that it was unsafe from a public health standpoint to conduct the exam in person during the pandemic. What was uncertain was for how long those circumstances would last. So, with that uncertainty, and not wanting to delay the board certification process for our candidates indefinitely, we decided to move forward.”

Venturing into Virtual

Each ABMS board that undertook the transition from in-person to virtual exams faced its own unique set of challenges, because, as Dr. Ogrinc explained, “Each board has a different exam format. Some use standard cases, some use real cases that the candidate submits, and others may use a combination of both. Each board chooses the format of cases that makes the most sense for its specialty.”

In the case of ABO-HNS, the board already benefitted from two distinct advantages. First, the American Board of Surgery and the American Board of Ophthalmology had both already conducted pilot exams, “so we were able to get some great advice from those boards, which had learned through their administrations with smaller groups,” explained Dr. Nussenbaum.

This sharing of information was key, added Dr. Ogrinc, who was impressed by the cooperative learning environment fostered among ABMS member boards. “They have done a remarkable job of taking an improvement approach to the remote oral exams,” he said. “Each started with a clear plan, learned from one another during the monthly AMBS remote oral exam learning collaborative, and assessed and made changes as they implemented their programs.”

The board provided several training sessions as well as a post-exam debriefing, which was quite helpful from a community standpoint.  —Andrew J. Holcomb, MD

The board’s second advantage concerned its exam schedule. Under pre-COVID-19 circumstances, the primary certification exam—generally taken by 300-plus candidates—is administered just a couple of days after the board’s much smaller neurotology exam, which had just 25 registered examinees in 2020. So, board leadership decided to separate the exam administrations by more than the usual few days. “We thought this would be a good opportunity to administer the virtual exam with the neurotology group in December 2020, and then schedule the primary exam for February 2021,” Dr. Nussenbaum said. “That way, we would be able to apply whatever we learned from the smaller group experience to the larger one. We would have time to make necessary changes and adjustments.”

Following the decision, approximately 5% of candidates who were eligible to take the February 2021 oral certification exam deferred, for reasons that might or might not have been related to the virtual format, noted Dr. Nussenbaum. In addition, given the unusual circumstances, the board granted all deferral requests with no questions asked and no time lost from candidates’ five-year board eligibility period. After deferrals, 308 candidates took the primary certification virtual exam.

Adapting the Format

As it turned out, the board’s strategy to administer the smaller neurotology virtual exam well in advance of the primary one was a crucial call. “Although our method worked well with the small group for the neurotology exam, we also learned that the process and the technology didn’t allow for easy scalability of the process to the numbers that we needed to administer the primary certification exam,” Dr. Nussbaum said. “That meant that we had to fit the exam format construction to the constraints of the technology. Fortunately, that two-month window allowed us to rethink the format structure and determine what changes needed to be made.”

Customarily, the in-person version of the ABO-HNS certification exam is approximately four hours long. During the process, the candidate visits five different stations, each of which presents three protocols during a 40-minute period. Concerned about the challenging logistics of so much movement of multiple candidates into multiple virtual rooms, as well as the likelihood of participant “Zoom fatigue,” the board scaled down the exam. Instead of five stations, there was to be just a single video conference room in which all the examiners were present. The three cases per station was reduced to two shorter-in-length cases. The newly designed virtual exam clocked in at 90 minutes.

Given the reduction in cases to analyze, great care was taken with the exam’s content selection, noted Dr. Nussenbaum. “We worked very closely with our psychometrician in a tremendous team effort to ensure that the redesign would still allow us to create a psychometrically valid exam, enabling us to make consequential pass/fail decisions on the candidates,” he explained.

Andrew J. Holcomb, MD, head and neck surgeon at Estabrook Cancer Center, Nebraska Methodist Hospital, and assistant professor in the department of surgery at Creighton University in Omaha, Neb., experienced the new virtual format firsthand when he took the exam in February 2021. Although Dr. Holcomb attests to major advantages offered by the revamped approach, he did view the shortened time per protocol as a disadvantage. “It challenged many candidates to adequately progress through the prompts in the allotted time,” he said. “There were prompts that would address only a portion of the clinical scenario, which was confusing to candidates both in terms of the approach and grading implications. That said, I did feel that the exam tested a broad range of clinical knowledge and judgement, and that the tone felt similar to what I would have imagined an in-person environment to provide.”

More feedback is forthcoming when the ABO-HNS virtual board exam is administered again in October 2021. Among the seated examinees will be Meha Goyal Fox, MD, who recently completed fellowship training in rhinology and skull base surgery in the department of otolaryngology–head and neck surgery at the University of Kansas Medical Center, Kansas City, Kan. Dr. Fox also has some concerns about the shortened format. “It leaves less time for examinees to explain their decisions, which could be detrimental,” she said. “The folks I’ve spoken to who took the exam in February said they felt a little rushed on the cases.”

Pros and Cons

Without a doubt, going virtual enabled the board to avoid the public health risks associated with bringing in people from all over the country and gathering them together in one central location. But there were other benefits. With the virtual method, examinees enjoyed the convenience of not having to travel. “This exam takes place during fellowship for many candidates, while others are working to build their practices, and it can be difficult to take time off even for a weekend exam,” noted Dr. Holcomb. Also key was the fact that candidates saved on travel-related costs such as flights, vehicle rentals, accommodations, and other expenses.

Of the 14 [ABMS] boards that offer oral examinations, about one-third are committed to continuing with remote exams, one-third are leaning toward returning to in-person exams, and one-third are still considering the options.  —Greg Ogrinc, MD, MS

Still, as Dr. Nussenbaum acknowledged, examiners did express disappointment over the missed opportunity to gather in person. “This in-person experience for examiners results in a remarkable sense of camaraderie, and that wasn’t there because everybody was in their own places at their computer and interacting only with their exam team members,” he said.

Meeting the technological requirements of the new format is another unavoidable challenge. “To take the exam virtually, [participants] have to make sure that their computer/hardware, software, webcam, and microphone are working properly; that they have access to high-speed internet; and that if they don’t have an ethernet cable, that their Wi-Fi is adequate and reliable,” explained Dr. Nussenbaum. “If an examiner has an IT problem, any of the others on the exam time can continue the exam and deliver the protocols. But if a candidate has a technology problem during the exam, it could lead to the need to reschedule to a future exam date.”

The idea of rescheduling the board exam is a worrisome prospect for any board certification candidate. “The oral board exam is only offered once a year, so having to reschedule due to technological issues can have severe consequences for employment, licensing, and credentialing,” pointed out Dr. Fox.

Given these potential consequences, administrators went the extra mile to ensure a smooth process technologically, taking pains to educate candidates and examiners, test equipment and connections, and answer any questions in advance. Board staff provided multiple individual check-ins with every exam candidate, including right before exam time. “The board provided several training sessions as well as a post-exam debriefing, which was quite helpful from a community standpoint,” Dr. Holcomb said.

The technological demands of the process might even have provided a friendly nudge toward a more environmentally friendly exam administration protocol, reflected Dr. Nussenbaum. “If you look at it from a green standpoint, it’s a lot of people not using planes and cars to get to the exam,” he said. “Also, previously, the exam materials, scoring, and evaluation forms were all on paper. When we changed to virtual, we also changed from a nearly completely paper exam to a fully electronic format.”

The electronic delivery has been so successful and well-received, in fact, that ABO-HNS intends to continue using a paperless exam even if in-person exams resume at some point. “It would work in an in-person environment, as long as you have a laptop with you,” said Dr. Nussenbaum.

Looking Ahead

The next ABO-HNS exam, in October 2021, will be virtual, but according to Dr. Nussenbaum, no decisions beyond that have been made. “The board will think about what to do in 2022,” he said. “We’re planning to have one more virtual administration to incorporate the things we learned from the February 2021 exam, continue to improve the process and the experience for the examiners and candidates, and then we’ll make a decision shortly afterward based on operational considerations, feedback, and public health safety.”

Having taken the virtual exam, Dr. Holcomb concluded that the advantages of the format outweigh the disadvantages. “Given the significant benefits of convenience and cost, and as long as the threat of COVID-19 resurgence or future pandemics remains, this format remains the safest for candidates and examiners.”

Anticipating the October exam, Dr. Fox has been keeping her ear to the ground. “The folks I have spoken to who took the exam in February didn’t report any technological issues, and the [ABO-HNS] does provide a time to check technology with their system prior to the time of examination,” she said. “I think that if the virtual format is effective, then the board should consider instituting it permanently.”

So far, examiner feedback on the virtual oral exam format has been positive. Moreover, scoring results from February’s test revealed a failure rate consistent with past in-person exams. For this ultimate success, Dr. Nussenbaum credits the board’s “exceptional” group of volunteers. “Without their incredible dedication and commitment to learning this new electronic virtual platform and process, this couldn’t be done,” he concluded. “The incredible, collective team effort from ABO-HNS staff and the examiners was truly inspirational to me.”

---

Linda Kossoff is a freelance medical writer based in Woodland Hills, Calif.