The Potential for Artificial Intelligence Tools in Residency Recruitment

According to Electronic Residency Application Service (ERAS) statistics, there were more than 1400 dermatology applicants in 2022, with an average of almost 560 applications received per program.^1,2 With the goal to expand the diversity of board-certified dermatologists, there is increasing emphasis on the holistic review of applications, forgoing filtering by discrete metrics such as AOA (American Osteopathic Association) membership and US Medical Licensing Examination (USMLE) scores.³ According to the Association of American Medical Colleges, holistic review focuses on an individual applicant’s experience and unique attributes in addition to their academic achievements.⁴ Recent strategies to enhance the residency recruitment process have included the introduction of standardized letters of recommendation, preference signaling, and supplemental applications.^5,6

Because it has become increasingly important to include applicant factors and achievements that extend beyond academics, the number of data points that are required for holistic review has expanded. If each application required 20 minutes to review, this would result in 166 total hours for complete holistic review of 500 applications. Tools that can facilitate holistic review of candidates and select applicants whose interests and career goals align with individual residency programs have the potential to optimize review. Artificial intelligence (AI) may aid in this process. This column highlights some of the published research on novel AI strategies that have the potential to impact dermatology residency recruitment.

Machine Learning to Screen Applicants

Artificial intelligence involves a machine-based system that can make decisions, predictions, and recommendations when provided a given set of human-defined objectives.⁷ Autonomous systems, machine learning (ML), and generative AI are examples of AI models.⁸ Machine learning has been explored to shorten and streamline the application review process and decrease bias. Because ML is a model in which the computer learns patterns based on large amounts of input data,⁹ it is possible that models could be developed and used in future cycles. Some studies found that applicants were discovered who traditionally would not have made it to the next stage of consideration based primarily on academic metrics.^10,11 Burk-Rafel et al¹⁰ developed and validated an ML-based decision support tool for residency program directors to use for interview invitation decisions. The tool utilized 61 variables from ERAS data from more than 8000 applications in 3 prior application cycles at a single internal medicine residency program. An interview invitation was designated as the target outcome. Ultimately, the model would output a probability score for an interview invitation. The authors were able to tune the model to a 91% sensitivity and 85% specificity; for a pool of 2000 applicants and an invite rate of 15%, 1475 applicants would be screened out with a negative predictive value of 98% with maintenance of performance, even with removal of USMLE Step 1 examination scores. Their ML model was prospectively validated during an ongoing resident selection cycle, and when compared with human review, the AI model found an additional 20 applicants to invite for interviews. They concluded that this tool could potentially augment the human review process and reveal applicants who may have otherwise been overlooked.¹⁰

Rees and Ryder¹¹ utilized another ML screening approach with the target outcome of ranked and matriculated compared with ranked applicants based on ERAS data using 72 unique variables for more than 5000 applicants. Their model was able to identify ranked candidates from the overall applicant pool with high accuracy; identification of ranked applicants that matriculated at the program was more modest but better than random probability.¹¹Both the Burk-Rafel et al¹⁰ and Rees and Ryder¹¹ models excluded some unstructured data components of the residency application, such as personal statements, medical student performance evaluation letters, transcripts, and letters of reference, that some may consider strongly in the holistic review process. Drum et al¹² explored the value of extraction of this type of data. They created a program to extract “snippets” of text that pertained to values of successful residents for their internal medicine–pediatrics residency program that they previously validated via a modified Delphi method, which then were annotated by expert reviewers. Natural language processing was used to train an ML algorithm (MLA) to classify snippets into 11 value categories. Four values had more than 66% agreement with human annotation: academic strength; leadership; communication; and justice, equity, diversity, and inclusion. Although this MLA has not reached high enough levels of agreement for all the predetermined success values, the authors hope to generate a model that could produce a quantitative score to use as an initial screening tool to select applicants for interview.¹² This type of analysis also could be incorporated into other MLAs for further refinement of the mentoring and application process.

Knapke et al¹³ evaluated the use of a natural language modeling platform to look for semantic patterns in medical school applications that could predict which students would be more likely to pursue family medicine residency, thus beginning the recruitment process even before residency application. This strategy could be particularly valuable for specialties for which there may be greater need in the workforce.

AI for Administrative Purposes

Artificial intelligence also has been used for nonapplication aspects of the residency recruitment process, such as interview scheduling. In the absence of coordinated interview release dates (as was implemented in dermatology starting in the 2020-2021 application cycle), a deluge of responses to schedule an interview comes flooding in as soon as invitations for interviewees are sent out, which can produce anxiety both for applicants and residency program staff as the schedule is sorted out and can create delays at both ends. Stephens et al¹⁴ utilized a computerized scheduling program for pediatric surgery fellowship applicants. It was used in 2016 to schedule 26 interviews, and it was found to reduce the average time to schedule an interview from 14.4 hours to 1.7 hours. It also reduced the number of email exchanges needed to finalize scheduling.¹⁴

Another aspect of residency recruitment that is amenable to AI is information gathering. Many would-be applicants turn to the internet and social media to learn about residency programs—their unique qualities, assets, and potential alignment of career goals.¹⁵ This exchange often is unidirectional, as the applicant clicks through the website searching for information. Yi et al¹⁶ explored the use of a chatbot, which mimics human conversation and exchange, on their institution’s pain fellowship website. Fellowship applicants could create specific prompts, such as “Show me faculty that trained at <applicant’s home program>,” and the chatbot would reply with the answer. The researchers sent a survey to all 258 applicants to the pain fellowship program that was completed by 48 applicants. Of these respondents, more than 70% (35/48) utilized the chatbot, and 84% (40/48) stated that they had found the information that was requested. The respondents overall found the chatbot to be a useful and positive experience.¹⁶