Skip to content

How AI Interview Scoring Works (and How to Check It)

In short

AI interview scoring turns a transcript into ratings against a rubric, each with a reason. Trust a score only if you can see the rubric, find the quote behind it and change the result. Tracy AI Interviewer does not score, rank or reject. She asks follow-ups in the person's own field, then gives your reviewer a recording, a timestamped transcript and what the person said to each question, with quotes. Your reviewer scores on your own rubric, from words they can replay.

How does AI interview scoring work?

AI interview scoring works in four steps. Someone writes a rubric for each question or skill, the answer is transcribed, a language model rates the transcript against the rubric, and the tool shows the rating with a reason. The ratings are then combined into one number, and a person can accept or change it. How AI interviews work covers the interview side.

Four vendors describe scoring this way in their own documentation, read on 14 Nov 2026, each with a different scale. HiPeople rates each competency with behaviourally anchored rating scales and defines in advance what strong, medium, weak or absent evidence looks like. Talkpush runs one evaluator per skill against anchor descriptions for every level and takes the mean on a 0 to 5 scale by default. Truffle scores each answer against its question's rubric and shows a 0 to 100 match score with the reasoning beside it. VidCruiter rates each transcribed response from 1 to 5 and says hiring teams approve every rubric before any candidate is rated.

SystemScaleWhere a person steps in
HiPeopleStrong, medium, weak or absent evidence per competencyUsers can override the AI's decision
Talkpush0 to 5 per skill, mean by defaultRecruiters can override any score
Truffle0 to 100 match score per answer"You decide what happens to every candidate"
VidCruiter1 to 5 per responseRubrics approved first; hiring managers can review, adjust or override a result

Each row is how the vendor describes its own scoring. Vendor documentation read 14 Nov 2026.

All four keep a person in the loop, and that step is the part of the pipeline you can inspect.

Four steps: rubric, transcript, rating with a quote, and a person who replays the quote and keeps or changes the rating.
Four places where a person checks an AI score, from the rubric to the final call. Illustration.

A score is worth reading when a person can follow it back to the quote and change it.

Where can an AI interview score go wrong?

An AI interview score goes wrong in four common places, and none of them belongs to one tool.

A vague rubric comes first. A rating can only be as specific as the anchors it is rated against, and "good communication" gives the model and the reader nothing to hold on to.

Delivery can also count as content. A fluent speaker can sound like a stronger one. Talkpush's rubrics tell the model not to raise or lower a score for how assertively or fluently the candidate spoke, so the risk is real enough to write a rule against.

An average can hide a gap. If a role needs five skills, ratings of 5, 5, 5, 5 and 0 average 4, the same as 4, 4, 4, 4 and 4. Talkpush's default is the mean, with weights optional.

Last, a number can arrive with nothing behind it. If a report shows 82 and no quote, the reviewer either trusts it or ignores it.

The research that favours scoring is about structured interviews. A 2022 re-analysis of published meta-analyses by Sackett and colleagues gives mean validity of .42 for structured interviews and .19 for unstructured ones, where validity is a correlation with job performance ratings. That result is about interview ratings in general, so ask any vendor for evidence about its own ratings. The four vendor pages read on 14 Nov state no link between their scores and later job performance.

How do you check an AI interview score?

Check a score against the words. Any rating should lead you to a quote and a moment in the recording within a minute, and you should be able to disagree with it. These are six checks you could run, our own suggestions rather than a tested protocol:

  1. Ask for the rubric for your role before the first interview, and edit it. Anchors for each level work like the ones on a human scorecard; structured interview scoring and the interview rating scale show how to write them.
  2. Click from each rating to its quote. A rating with no quote is an opinion.
  3. Replay a sample, say one rating in ten. Watch the minute it points to and write your own rating before reading the model's.
  4. Read the per-skill ratings as well as the total, so a gap does not disappear into an average.
  5. Check for delivery effects by giving the tool two practice answers with the same content, one polished and one hesitant, and comparing the ratings.
  6. Keep a log of who changed a rating and why, so a second person can review it.

A person decides who moves forward. The score is one input to that decision, next to the structured interview you already run.

What should AI evaluation interview questions ask for?

A question that asks for a case, a decision or a number gives any rating something to hold. "Tell me about a report you rebuilt because the numbers disagreed" produces evidence. "Are you detail-oriented?" produces a claim.

A follow-up such as "How did you check it was right?" separates the person who did the work from the person who watched it. Our guide to writing interview questions an AI can follow up on has patterns for it, and candidate screening questions lists examples by skill.

What Tracy gives the person who scores

Tracy does not score, rank or reject anyone, so there is no model rating for your team to second-guess. She does the part that makes a human score possible. She asks follow-ups in the person's own field, waits for the end of the thought and asks for the missing detail. Bring your own questions, or describe the role and Tracy drafts them for you to edit.

Each result then holds a recording of both sides, a timestamped transcript, what the person said to each question and the integrity signals shown for review. Your reviewer scores against the rubric your team already uses, such as an interview scorecard, and replays the minute behind any point. What an AI interview report contains covers what a reviewer should expect to see in one.

Create one interview for your role and check the first results against their recordings.

Recreated result report with sample data: recording, transcript with a follow-up question, and three questions with what was said and quotes.
A Tracy result report: recording, transcript with the follow-up, and what the person said with quotes, for a reviewer to score. Illustration with sample data

Each point in the report sits beside the person's own words and a place in the recording, with no score for the reviewer to defer to.

What stays with your team

The person interviewed is told on the join screen that the conversation is recorded and transcribed. Nothing is decided in the interview. Your team reads the evidence, scores it and decides, and can explain the decision from the recording.

Common questions

What is AI interview scoring?
A tool transcribes an interview answer, rates it with a language model against a rubric and shows the rating with a reason. Most tools then combine the ratings into one number a recruiter can accept or change.
What is an AI interview evaluation system?
The rubric, the transcript, the model that rates against the rubric and the report that shows ratings with reasons, plus the step where a person reviews. Check each part, not only the final number.
Is AI interview scoring accurate?
It depends on the rubric and on what the score was checked against. The vendor pages reviewed state no link between their scores and later job performance, so ask for it and test the ratings against recordings yourself.
Is AI interview rating fair?
A rating can be unfair if the rubric is vague or if delivery is counted as content. Test it with two answers of the same content in different styles, and keep a person in the decision.
Does Tracy score candidates?
No. Tracy does not score, rank or reject. It reports what the person said to each question, with a recording and a timestamped transcript, and your reviewer scores.
Can a recruiter override an AI interview score?
The four vendors reviewed all describe an override or review step. Confirm yours, and keep a log of who changed what and why.

See the evidence behind one result

Create one interview for your role and check each point in the report against the recording. The first 3 results are free.

Try it free (opens in a new tab) Book a demo (opens in a new tab)

Your first 3 results are free. No card needed.

Sources