Shortlist on evidence, the same way every time.

Shortlisting is where most hiring decisions actually get made. By the final interview you’re choosing between a handful of people. The bigger choice was which handful. Yet shortlisting is often the least structured step: a recruiter skims a pile, goes with a feeling, and nobody can say afterwards why one person moved forward and another didn’t. A fair shortlist uses the same criteria for everyone, looks at evidence rather than impressions, and leaves a record someone else could check.

1. Agree the criteria before you look at anyone

Criteria set after you’ve seen candidates tend to describe the candidate you already liked. Decide them first, with the hiring manager, and write them down.

Good shortlisting criteria are:

• Tied to the job. Each one should connect to something the person will do in the first few months.
• Observable. You can point to something in the interview or application that shows it.
• Few. Three to five skills. Beyond that, reviewers start weighting them by instinct.
• Levelled. Say what level you expect. A beginner bar for a trainee role is not the same as an advanced bar for a team lead.
• Weighted, if they aren’t equal. If handling difficult customers matters twice as much as product knowledge, say so up front.

Keep hard requirements, like shift availability or right to work, separate from skills. They’re yes-or-no facts, and a person should read them rather than letting them silently lower a score.

Never include anything related to protected characteristics, and be careful with proxies for them, such as gaps in employment, graduation year or where someone lives, unless the job genuinely requires it.

2. Review evidence, not impressions

Confidence, polish and a familiar accent are easy to mistake for skill. The fix is to ask, for each criterion, “what did they actually say or do that shows this?”

A simple habit: before you rate a skill, find the sentence that supports your rating. If you can’t find one, you’re rating an impression.

The evidence column is also what you’ll need if anyone later asks why a decision was made.

3. Calibrate your reviewers

Two reviewers reading the same interview will often disagree. That’s normal. What matters is whether you notice and correct it.

Before a large batch:

1. Pick five to eight completed interviews that cover a range, from clearly strong to clearly weak.
2. Have every reviewer rate them independently against the criteria.
3. Compare ratings side by side. Discuss the biggest gaps: what did each person see, and which reading is closer to the criteria?
4. Write down what you agreed, for example “an answer that names the steps but not the outcome is intermediate, not advanced.”

Repeat briefly every few weeks, or whenever a new reviewer joins. Reviewer drift is gradual and hard to see from the inside.

4. Take a second look below the line

Any cut-off, whether it’s a score, a gut call or “top 20”, will put some good people on the wrong side. Build in a regular second look.

• Sample below the line. Each week, read a handful of candidates who just missed out, not only the ones far below.
• Look for patterns. If people who just missed share something unrelated to the job, like a particular college, region or language, stop and check the criteria.
• Watch for short answers. A nervous candidate, or a weak connection that dropped words, can produce a short answer that hides a strong person. Check the recording or transcript before setting them aside.
• Move people back when you’re wrong. A shortlist is a working list, not a verdict.

If you use a pass mark, the second look is how you learn whether it’s in the right place. The pass mark guide covers how to set and adjust one.

5. Document every decision

A short record protects candidates and your team. For each person, you should be able to say:

• Which criteria they were assessed against, and at what level.
• The rating for each, and the evidence behind it.
• Who made the shortlisting decision, and when.
• If a decision went against the ratings, why.

This doesn’t have to be long. One line of evidence per skill and a name against the decision is enough for most roles. It also makes the next round better, because final interviewers can start from what’s already known.

Common mistakes

• Shortlisting on résumé keywords. It rewards people who know how to write résumés, not people who can do the job.
• Letting the first few candidates set the bar. Whoever you review first becomes the unspoken standard. Calibrate on a mixed set instead.
• Treating the shortlist as final. Candidates just below the line deserve a second look before you close the role.
• Keeping no record. If you can’t explain a decision a month later, you can’t check it was fair.

Where AI screening fits

AI scoring can make shortlisting more consistent, as long as a person still decides. In LessRounds, each skill is scored 0 to 100 against the level you set, with the reasoning and a quote from the transcript behind every score, which covers much of steps two and five. Candidates below the pass mark are set aside without being contacted, and anyone on your team can move them back at any time, which makes the second look quick. Scoring reads the words of the answer, not faces or voices, and never marks down language or grammar. It won’t make a process perfectly fair, so keep calibrating and keep checking below the line.