Fact-checked on 3 Nov 2026 against the public platform pages and announcements listed in Sources.
TLDR
- Tracy AI Interviewer can take the conversation layer of expert vetting, asking field-specific follow-ups of every applicant and leaving reviewers the recording, a transcript timed to it and integrity signals with evidence.
- Tests show answers. Follow-ups show whether the expertise is real and the person is the one who earned it. Use both, plus identity and trial-task checks.
Vetting domain experts for AI training projects breaks in a predictable way as you scale. Ten applicants, and a domain lead talks to all ten. A thousand applicants, and you lean on a form, a test and a credential check. Each of those can pass someone who can write a fluent answer and cannot do the work, and fail someone who can do it and writes plainly.
This guide sets out a layered way to vet experts at volume, shows what a follow-up adds that a test cannot, and says where Tracy AI Interviewer fits and where it does not.
Why does vetting break at scale?
Public pages from the platforms that do this show the shape of the work. Outlier, an AI training platform, says its onboarding takes 30 to 90 minutes and "includes creating your account, selecting your areas of expertise, completing skill screenings so we can match you with the right projects, and identity verification". Its minimum qualification, it says, is typically undergraduate-level expertise, so a screening has to do the separating.
The tasks are demanding. Outlier lists writing a challenging prompt, creating grading rubrics and rating and ranking answers, and asks experts to write a difficult question in their field, "one that would confuse an AI model into writing an incorrect answer." DataAnnotation describes remote contract roles "matched to your skills", across fields from finance and law to data science. For the same field, one expert writes a prompt a model gets wrong and a rubric two graders score alike. Another writes a vague one. You find out which after the project has started, unless the vetting asked.
The stakes are visible in the market. Mercor, which says it connects human expertise with AI labs and enterprises, announced a $350 million Series C at a $10 billion valuation on 27 Oct 2025. The AI Insider reported in September 2025 that micro1 secured a $35M Series A at a $500M valuation. We cite both as market context: they show how much money now rides on expert supply. They say little about how a platform vets, apart from micro1's AI recruiter, covered under the alternatives below.
What should vetting check?
Five things, and each is a place a test alone falls short.
| What to check | What a test shows | What a follow-up adds |
|---|---|---|
| Domain depth | A right or wrong answer | How far the knowledge goes past the first answer, and where it stops |
| Reasoning | A final answer | The path to it, and what would change it |
| Prompt and rubric writing | A written sample | Whether the expert can defend the rubric against a case they did not plan for |
| Catching model errors | Whether they flagged a planted error | Why it is wrong, and how they would make a grader catch it |
| Integrity | A score, from whoever answered | Whether the person can keep going on their own answer, live |
Our guide to AI trainer interview questions has 25 questions across these areas. For a data scientist as the expert, see the data scientist question bank.
What does a layered vetting process look like?
No single layer carries the load. A process that holds at volume stacks five.
- Identity and credentials. Confirm the person is who they say they are and holds what they claim. This is its own step, and an interview does not replace it.
- A screen. A short test or questionnaire that removes the clearly unqualified, cheaply, for everyone.
- A conversation in the expert's field. A two-way interview where each follow-up builds on the last answer. This is where depth and ownership show.
- A paid trial task. Real project material, reviewed closely. It shows how they work, not only how they talk.
- Ongoing quality checks. Review early outputs, and keep sampling after onboarding.
Layer 3 is the costly one at volume, because each conversation takes a domain lead's time. Tracy can run that layer: the same conversation with every expert who passes layer 2, each starting when they click the link at any hour, with your leads reading the results instead of holding the calls.
What does a follow-up chain add? An example
Here is an example we wrote to show the shape of a chain, in statistics. It is not a transcript of a real interview. The opening question gives the expert a model answer to assess.
Interviewer: A model answered: "The sample mean of n independent Cauchy draws converges to the true mean by the law of large numbers." Is that right?
Expert: No. The Cauchy distribution has no mean, so there is no true mean to converge to.
Interviewer: Why does the law of large numbers not apply?
Expert: It needs a finite expectation. For Cauchy, the integral for the mean does not converge absolutely.
Interviewer: What does the sample mean of 100 Cauchy draws look like?
Expert: It has the same Cauchy distribution as a single draw. Averaging does not narrow it, which is why a few extreme values keep showing up however many you take.
Interviewer: How would you write a rubric item so a grader catches this error?
Expert: I'd make the check "states that the mean does not exist" a required item, and add a note that "converges by the law of large numbers" is a fail, even when the final sentence sounds right.
The first answer is where a test would stop. The second shows the expert knows the condition behind the rule, and the third whether they can say what actually happens, which is hard to fake from a half-remembered summary. The fourth checks the skill the project needs most: turning an error into something a grader can score.
Now the same opening with a weaker expert, also written by us.
| Question | Weaker answer | What it shows |
|---|---|---|
| Is the model's statement right? | "Yes, the law of large numbers says averages converge." | Recalls the rule, misses its condition. |
| Why would it not apply? | "Maybe the sample is too small?" | Guessing at a cause. |
| What does the mean of 100 draws look like? | "Close to zero, probably." | Cannot say what actually happens. |
The weaker expert might have scored well on a multiple-choice test of the same topic. Three follow-ups tell you more than the test did.
How do you check the expert is the one answering?
Fluent written answers are easy to get from a model now. Two checks work together.
First, the follow-up itself. A question built on what the expert just said cannot be pre-written, and an expert who is relaying someone or something else slows down when the next question goes a layer deeper. Our guide to AI interview cheating detection covers this in full.
Second, integrity signals with evidence. Tracy captures the face and where the expert is looking, response timing, tab switching, leaving fullscreen and an external screen, and nothing is decided during the interview: afterwards the signals are assessed together and the evidence goes to your reviewer. Tracy's cheating-detection guide names five signals flagged for a closer look: tab changes, abnormal eye tracking, live face detection, the average delay before an answer and external screens. The guide says a signal calls for review and does not establish that someone cheated. A person looks at the moments flagged and decides. Do not treat a signal as proof: people read, look away and lose a connection for ordinary reasons.
How does Tracy fit in the layers?
Tracy AI Interviewer (see what an AI interviewer is) runs layer 3. It runs two-way video interviews. With CV check on, Tracy reads the expert's CV and the project's must-have requirements and asks 3 to 5 questions tailored to them, on top of your own. Follow-ups are in the expert's own field and build on what was said, and Tracy waits for the end of a thought, so a long technical answer is not cut off. Questions are generated from the project description and can be edited or replaced, so the field-specific questions you already use can go in as written.
Four details matter at volume:
- Experts join from a browser with nothing to install, whenever it suits them, which suits experts in many time zones.
- People are told about the recording before it starts.
- Each expert is interviewed when they click the link, with no scheduling, and a candidate who drops can rejoin the same session within 10 minutes.
- Tracy connects through an API and an MCP server, so interviews can be created and read from the tools a team already uses, and enterprise teams can ask for a customised setup. For published prices across tools, see AI interview cost.
Your domain leads get the recording of both sides with a transcript anchored to it by timestamp and a written assessment per question with its reasoning, so they can check a claim against what the expert said. Tracy does not reject anyone: a person reviews the evidence and decides. Our guide to AI interview reports shows what a reviewer should look for.
When a different route is better
- If you would rather have a marketplace source and screen experts for you. Marketplaces such as Mercor say they connect experts with AI labs and enterprises, and micro1, according to The AI Insider, recruits its experts through its own AI recruiter, Zara. You trade control of the vetting for speed, which suits a team that needs experts this month and has no pipeline of its own.
- If you need to see the expert do the task. Run a paid trial on real material. An interview shows how they think; a trial shows what they produce.
- If you need identity documents or credentials checked. Use a dedicated check. Tracy is not a credential check.
- For a senior reviewer or a lead role. Put a domain lead in the conversation. Tracy fits the first conversation with everyone, not the final one.
- For people who have not agreed to be recorded. Tracy is not for talking to someone who has not agreed to be recorded.
For the technical side of a coding or system-design check, see AI technical interviews. If your problem is volume before vetting begins, screening when there are too many applicants is the place to start.
Questions
How do you vet a domain expert for AI training work?
Why are tests not enough to vet experts?
Can an AI interviewer vet domain experts?
How do you check an expert is not using AI during the interview?
What should you do after the interview?
Try it free and run the conversation layer with your next expert applicants, or book a demo.
Sources
All sources retrieved 3 Nov 2026.
- Outlier, home page: common tasks, minimum qualifications, onboarding with skill screenings and identity verification.
- DataAnnotation, home page: remote contract roles matched to skills.
- Mercor, Unlocking human potential in the AI economy, 27 Oct 2025: Series C and valuation, as the company states them.
- The AI Insider, Micro1 Secures $35M Series A at $500M Valuation to Scale Human Expertise for AI Training, 18 Sep 2025: funding of micro1 and its AI recruiter, Zara, as reported.
- Tracy AI Interviewer, product page: two-way video interview, questions from the project description or your own, CV check, follow-ups, recording notice, recording and transcript timed to it, an assessment per question, integrity signals, API and MCP.
- Tracy AI Interviewer, home page: a person reviews the evidence and decides; integrity signals with their evidence.
- Tracy AI Interviewer, pricing: API and MCP connections and a customised setup for Enterprise.
- Tracy AI Interviewer, Connect page: MCP server and API connection to the tools a team already uses.
- Tracy AI Interviewer, AI interview cheating detection: five integrity signals flagged for a closer look, and that a signal calls for review without establishing cheating.