TLDR Tracy AI Interviewer asks each medical expert questions in their own specialty, follows up on what they said and waits for the answer, so you see how they reason and not only what they know.
Use the 18 questions below, grouped by what they test: reasoning with missing information, reviewing a model's answer, scope and safety, rubric writing, specialty depth and project fit.
Every result keeps a recording and a timestamped transcript, with integrity signals and their evidence, for a person to review. Licences and credentials are checked separately.
A clinical-review project lands on a Monday and hundreds of applicants have CVs that name the right specialty. A CV says what someone trained in. It does not say whether they can explain why a model's answer is unsafe, or whether the person on the call is the person who will do the work. Medical expert interview questions for AI training projects have to test that gap.
These ask for clinical reasoning applied to review work: reading a model's answer, writing the criteria for judging it, and knowing when to stop. Each has a follow-up and a note on what a strong answer contains. Job seekers can use them to prepare, and should answer the follow-up too, because that is where the interview starts.
Two limits. The questions test reasoning for review work, give no medical advice and are not a clinical standard. And they do not check a licence: Tracy does not verify a licence or a degree, so check credentials separately against the register in the expert's country. For the generalist version of this job, see our guide to AI trainer interview questions. For the rest of the process, see expert vetting.
The work asks for judgement on a model's answer, not only medical knowledge
A public healthcare benchmark, HealthBench, shows the work at scale. Its preprint, submitted in May 2025, says responses are evaluated with conversation-specific rubrics created by 262 physicians (source). Specialty matters as well: a cardiologist and a dermatologist should not get the same case question.
The questions test reasoning with missing information, reviewing a model's answer, scope and safety, rubric writing and agreement, specialty depth and currency, and project fit (hours, time zone, steadiness over a long batch, care with patient information). A clinician on your team should adjust each one to the specialty and the task.
Eighteen medical expert interview questions, grouped by what they test
Pick one or two questions from each group. Ask the follow-up after you hear the first answer, write down what a strong answer contains before the first expert joins, and give every expert the same opening questions.
Group 1: reasoning with missing information
| Ask | Follow up with | Listen for |
|---|---|---|
| 1. A patient says their chest has felt tight for two days. Before you say anything useful, what do you want to know? | Which answer would change your next step the most? | Asks about onset, what it feels like, context and who the patient is. Orders the questions by impact. Says what a text conversation cannot settle. |
| 2. Tell me about a case where your first impression was wrong. What changed your mind? | What would you have needed to see earlier? | A real case with a turning point. Owns the miss. Names a check they now do. |
| 3. Two reasonable clinicians manage the same kind of case differently. Describe one in your field. | What should a model say about that disagreement? | Separates real uncertainty from error. Wants options and reasons, not an invented consensus. |
Group 2: reviewing a model's answer
| Ask | Follow up with | Listen for |
|---|---|---|
| 4. A model told a patient: "This is usually harmless and will pass by itself." What do you check first? | What would make you reject it even if every word is true? | Looks at what is missing: when to seek care, who the answer is for. Names a reason to reject that is not a factual error. |
| 5. What is the difference between an answer that is wrong and one that is unsafe? | Give me an example from your own reviewing. | Draws a defensible line. Knows a correct answer can be unsafe. Real example, patient details removed. |
| 6. How do you grade an answer that is correct but written for a clinician when a patient asked? | Would your score change if the asker were a nurse? | Grades against the asker, not a textbook. Says what changes with the reader. |
| 7. A model says "current guidelines recommend" something. How do you check it? | Which version, and for which country? | Opens the source and reads the sentence. Checks date, population and strength of wording. |
Group 3: scope, safety and patient information
| Ask | Follow up with | Listen for |
|---|---|---|
| 8. When should an answer tell the person to see a clinician instead of answering? | Give a case where that advice would be too cautious. | Sees both failures. A referral can be the useful answer or a way of saying nothing. |
| 9. What do you do when a task asks you to judge something outside your specialty? | How would your lead know the flag was fair? | Flags it, gives a one-sentence reason, offers a nearby task. Does not guess to keep the queue moving. |
| 10. What would you never put into a task, a note or a chat with us about a patient? | How do you write a rubric example, then? | Builds an invented case. Removes identifiers without being asked. |
Group 4: rubrics and agreement with other reviewers
| Ask | Follow up with | Listen for |
|---|---|---|
| 11. Pick a question patients often ask in your field. Write three criteria for judging an answer. | Which of your three could two reviewers read differently? | Criteria scored yes or no. Notices their own vague wording. |
| 12. A rubric line says "recommends appropriate follow-up." What is wrong with it? | Rewrite it so a second reviewer scores it the same way. | Says "appropriate" hides the judgement. Splits it into a trigger and a time frame. |
| 13. You and another reviewer score the same answer far apart. What do you do? | When do you hold your score? | Compares reasons, not numbers. Holds on a fact they can show, moves on taste. |
Group 5: specialty depth and currency
| Ask | Follow up with | Listen for |
|---|---|---|
| 14. What is a common belief in your specialty that is out of date or contested? | How would you spot a model repeating it? | Something specific, with how settled it is. Describes the tell in a model's wording. |
| 15. How do you keep up with your field, and what did you last change because of it? | How did you judge that source? | A named source and a concrete change. |
| 16. Where does practice differ between countries or health systems in your field? | How would you mark an answer that fits one system and not another? | A real first-hand difference. Marks an answer that does not say where it applies. |
Group 6: working on the project
| Ask | Follow up with | Listen for |
|---|---|---|
| 17. Walk me through twenty reviews in a two-hour block. How do you keep your standard steady? | How would you notice it slipping? | A routine with breaks or spot checks. Knows when they get faster and worse. |
| 18. How many hours a week, in which time zone and for how long can you commit? What would interrupt it? | What happens when a clinic shift runs over? | Specific numbers and a plan for the bad week. Says early if it will not work. |
For a six-question call, take 2, 4, 7, 9, 12 and 16, and ask question 18 of everyone. These are also the medical specialist interview questions to adapt: keep the group, change the case to the specialty.
A strong answer names what it would check, and a weak one stops at the fact
A weak expert is often right and runs out of reasons when the question moves from a fact to a decision. These pairs are written by us as examples, not quotes from real experts.
| Question | Weak answer | Strong answer |
|---|---|---|
| 4. What do you check first in the model's reassurance? | "It's accurate, so I'd score it high." | "It's accurate and never says when to get care. For a patient asking about a symptom, that gap gets marked down." |
| 2. What changed your mind about the case you got wrong? | "I don't think I've had one." | "A case I labelled routine. The detail I skimmed was in the history, and I now read the history first." |
Listen for ownership. "We found" describes the team. "I checked, and this is what I do now" describes the person.
A follow-up built on the expert's own answer shows whether they have actually done the work.
Run the same interview for every applicant without losing the follow-up
A live call for every applicant does not survive a project that draws hundreds of them. Tracy runs the interview for each expert, and each setting below is on the product page.
- Brief or your own questions. Describe the role and Tracy drafts questions you can edit, or paste the 18 above. Style runs from quick to deep.
- CV check, as a setting. With it on, Tracy can read the CV and ask about what it claims.
- Follow-ups that wait. Tracy asks for the missing detail in the expert's field and waits for the end of a thought. A director at an accelerator residency programme said: "Tracy sometimes brings up exactly those questions I might have forgotten to ask, which has been a pleasant surprise."
- Start on click. Each expert opens a browser link with no account and no calendar, at any hour. The join screen says the conversation is recorded and transcribed. Candidates rate Tracy 92 on NPS.
- A result you can check. A recording, a timestamped transcript and a written assessment for each question sit behind every result, so a reviewer can go from a statement to the moment it was said.
- Integrity signals, with the evidence. Tracy captures tab changes, gaze, face detection, delay before answers, fullscreen exits and external screens, and shows the evidence to a human reviewer. A signal is not proof, and your team decides. See AI interview cheating detection.
Pricing is $1 per credit, one credit per started 20 minutes of an interview that produced a result, so a 30-minute interview takes two credits. The first 3 results are free with no card, there are no seats, and a session with no answers is never charged. See the pricing page, or the Connect page for the API and MCP server.
A reviewer can go from each stated point in the result to the moment the expert said it.
Keep three checks with people
- The credential check. Verify a licence or degree separately, against the right register, before the interview or at approval. A strong interview from someone with no valid licence is still a reject.
- A clinician on your side. Have one of your clinicians read five or six transcripts and score them blind. If they disagree with the written assessments, change the questions before you invite the next hundred experts.
- The decision. A person decides from the recording, transcript, assessments and any integrity signals. Tracy rejects no one.
If you vet ten experts a month, a clinician on a video call is likely cheaper than any tool (AI interview vs human interview). The case for Tracy starts when applicants outnumber your reviewers' hours. Pilot on one specialty with 15 to 30 applicants, and read AI interview reports first.
Questions
What should you ask a medical expert in an AI training interview?
How do you test clinical reasoning without giving medical advice?
Can an AI interviewer interview medical experts?
Does Tracy check a medical licence?
How long should the interview be?
Run these questions on your own applicants. The first 3 results are free, with no card. Try it free · Book a demo
Sources
- HealthBench, arXiv 2505.08775, submitted 13 May 2025: rubrics created by 262 physicians, retrieved 9 Nov 2026.
- Tracy AI Interviewer pricing, retrieved 5 Nov 2026.
- Tracy AI Interviewer Connect page, retrieved 5 Nov 2026.
- Tracy AI Interviewer home page, customer proof, retrieved 9 Nov 2026.