How did someone who got the number 1 "overall score," which includes both the application and interview scores, end up so relatively far down the final list?
Yes, interesting. The way this happens is because we don't blindly accept our "formula" for combining the interview score, the USMLE score, and the clinical score. Let's review what characteristics we want to recruit, as I posted before (in quotes below):
“Will we enjoy working with this person, can we trust them to do the best job possible for our patients, can we be confident in their ability to interact with our clinical colleagues, will this person help us recruit great residents in the future, will we be excited to see this person at a meeting after they graduate, are they on a career trajectory that could inspire future residents and maintain our brand?
• Enjoy working with: obvious
• Trust them to do the best job: potential for developing excellent clinical skills, demeanor, motivation
• Interactions with clinical colleagues: If the most arrogant surgeon comes down to review a case, will this person be able to handle it well?
• Help us recruit: are they likeable, present themselves well, will they pursue the best possible fellowships
• Excited to see: obvious
• Inspiring career trajectory: Leadership (private practice, organized radiology, or academics), Academics, Community service."
As I mentioned in that post, the selection committee jots down a score for "Leadership" which gets at a combination of research/academic ambitions and inspiring career trajectory. We review this score as we look at the candidates and use it to move up or down. But it doesn't actually fit into a "formula".
The person who had the #1 "overall" score based on our mathematical formula that combines the interview score, USMLE score, and clinical score was an individual who went to a large mid tier medical school out west which we know well, who got all honors at that school where honors was only given to 15-30% of individuals (so relatively good discriminator of performance), and who scored over 265 on step 1 and step 2 (doing better on step 2 than step 1). On the interview score, the person did fine but the feeling was the person had modest career ambitions. That person went to undergrad at the same institution as medical school, was probably going to settle in that area, and we perceived that person to have little true desire for academics or leadership, based on historical activity and interview impressions. The committee had this assessment:
• Enjoy working with: YES!
• Trust them to do the best job: YES!
• Interactions with clinical colleagues: YES!
• Help us recruit: Very likeable, but probably would do fellowship at same institution the person came from to eventually practice in that area. That institution is fine but not really a "name".
• Excited to see: YES!
• Inspiring career trajectory: Looked to us like this person will be a great colleague in a private practice in the area that person grew up.
So based on that, we dropped that person to #11. We just didn't think that in the end this person was going to have an inspiring career trajectory THAT WOULD BE READILY APPARENT TO APPLICANTS. In other words, the person might in fact be very inspiring on the local level that he/she chooses to be at--but hard to point to that person to our future applicants and say, "Look at this graduate, he/she is practicing in a community practice in a mid-size area where he/she grew up--isn't that inspiring!".
Now, to be fair, based on the combination of the size of our program and history, and because we always interview and rank our candidates based purely on our metrics and don't worry about our "rank to fill" number (in other words, as I mentioned before, I will rank people highly who I sense would rather be somewhere else geographically), we have never filled all of our positions in the top 15 choices. So ranking this person #11 meant the person was still going to match with us if he/she ranked us first, unless somehow we magically got all of our top choices--which never happens.
The person ended up matching at the same institution that he/she did undergrad and med school at--I expect he/she will be in one of the big "super-practices" that we know that operate in that geographical area (at which some of our graduates currently work). It's where that person grew up, chose to stay for undergrad, chose to stay for med school, and chose to stay for residency. I would be very surprised if that person doesn't do his/her fellowship there.
Applicants can't be easily distilled into "numbers". We use scores and numbers to help guide our thinking and keep us honest (it's a way to combat unconscious bias which is hard to recognize). But we aren't beholden to the scores and numbers if the consensus is that the candidate has some special appeal or is less of a good fit for us.
And another question about the same step in the process: I know that you've said that you personally select the applications that go on to get an interview. But after the interview stage is over, much of a relative impact do you have (as program director) in terms of the final ranking compared to the other faculty members who have interviewed the candidates? Do you get a disproportionately high say in the final ranking, or is everyone on the ranking committee given an equal say in how any applicant gets ranked?
I'm passionate about our residency and the selection process, so objectively I would say that I probably advocate forcefully for a particular outcome--but my selection committee is used to me and there are other passionate voices, and others who are quietly effective at shaping group consensus. We work well together and know each others' blind spots and preferences. In the end, if I'm the outlier, I'm overruled. Where I hold the most power as PD and chair of this selection process is when the selection committee is fairly equally split about a candidate. If half love someone, and the other half are "meh", then it really does matter where I stand since I usually break that tie.