Fact-checked on 3 Nov 2026 against the public job posts and research listed in Sources.
TLDR
- Tracy AI Interviewer can read each data scientist's CV, add 3 to 5 questions specific to it and put your question bank to every applicant, with follow-ups that depend on the previous answer and a written assessment per question.
- Use the 20 questions below as a bank: pick four or five, one per area, and always ask the follow-up.
Most lists of data scientist interview questions are written for candidates: a question and a model answer to memorise. A candidate who has read the same list gives you the model answer, and you learn nothing about whether they did the work behind it.
So this guide is written for the interviewer. Each question comes with the follow-up to ask once you have heard the first answer, and with what a strong answer contains. The follow-up is built on what the candidate just said, so it cannot be rehearsed.
That is the part Tracy AI Interviewer takes over when many people apply: it asks the opening question, listens, and asks the follow-up that answer calls for, for each applicant, at any hour.
What does the role need, according to two public job posts?
Job posts differ, but two current ones show the common core. Reddit's Senior Data Scientist, Ads posting asks for a strong understanding of "statistical modeling, machine learning algorithms, causal inference and experimental design", experience with SQL, and lists communicating findings to stakeholders as a responsibility. Airbnb's Senior Data Scientist, Guest & Host posting asks for "experimentation, causal observational analysis, and machine learning techniques" and for "an ability to communicate complex technical concepts to a non-technical audience."
Both are senior roles at product companies, so treat them as a sample, not a standard. They still point to the same five things to test:
- Statistics and experiments: can they reason about evidence and uncertainty?
- Modelling: can they choose, check and defend a model?
- Data handling and SQL: do they trust data too early?
- Metrics and product sense: do they know what a number is for?
- Communication and ownership: can others act on what they say?
Data scientists are also hired as experts for AI training work. DataAnnotation, for one, lists a Data Scientist role in Data & ML, shown on its home page as "Up to $150+ / hr". For that kind of role the same five areas apply, with extra weight on spotting confident wrong answers. Our guide to AI trainer interview questions covers that side, and how to vet domain experts at scale covers the process around it.
Why does the follow-up matter more than the question?
Structured interviews, where every candidate gets the same questions and you score against the same criteria, do well in research. A 2022 re-analysis in the Journal of Applied Psychology (Sackett and colleagues) found structured interviews to be the top-ranked selection procedure for predicting job performance. The authors also report that most of the procedures that ranked high in earlier summaries remain high in rank, with mean validity estimates reduced by .10 to .20 points. That research is about interviews run by people, not AI interviewers, and it says nothing about any one question.
The practical point is the structure: same core questions, same criteria. The follow-up is where you adapt to the answer. Keep the opening question fixed so candidates can be compared, then let the follow-up go where the answer leads.
How to use these questions
- Pick four competencies for the role from the five above.
- Choose one opening question for each.
- Keep the follow-up beside it and ask it only after you hear the first answer.
- Write down what a strong answer contains before the first candidate joins.
- Score the evidence after the interview, with the notes or transcript open.
For a junior role, look for sound reasoning and the ability to learn. For a senior role, expect named trade-offs, past mistakes and examples of changing a decision. Do not turn trivia into a harder interview.
The questions and follow-ups below are our own, written for this guide.
Statistics and experiments
| Opening question | Follow-up | A strong answer |
|---|---|---|
| 1. An A/B test shows a 2% lift in conversion with a p-value of 0.04. Do you ship? | How many other metrics did you look at, and does that change your answer? | Asks about the planned hypothesis, sample size and run time. Raises multiple comparisons, novelty effects and the gap between statistical and practical significance. |
| 2. Explain a p-value to a product manager. | What would a confidence interval tell them that the p-value does not? | Uses plain words, avoids "the probability the hypothesis is true", and says an interval shows size and uncertainty. |
| 3. A feature rolled out by region, so you cannot randomise. How do you estimate its effect? | What would make you distrust that estimate? | Names a method such as difference-in-differences or matching, states the assumption it needs and says how to check it. |
| 4. Your two experiment groups already look different before treatment. What now? | How would you tell a randomisation bug from bad luck? | Checks the assignment logic and the split ratio, runs balance checks and does not quietly re-weight and carry on. |
A weak answer to this group is a rule recited with no judgement: "p below 0.05 means ship." Ask what they would do if the lift were real but tiny, and listen for whether the cost of acting enters the answer.
Modelling
| Opening question | Follow-up | A strong answer |
|---|---|---|
| 5. Walk me through a model you took to production. | Which choice would you change, and what did it cost? | Separates their own work from the team's, names a constraint and a real regret, and says how it was monitored. |
| 6. A fraud classifier has 98% accuracy. Is it good? | What would you measure instead, and who sets the threshold? | Notices class imbalance, asks about precision and recall and the cost of each error, and says the threshold is a business decision. |
| 7. The model scored well offline and badly after launch. Where do you look? | What would you have logged beforehand to find this faster? | Considers leakage, differences between training and serving data, drift and changed user behaviour, in a sensible order. |
| 8. When would you pick a simpler model than the best validation score? | Tell me about a time you did. | Weighs interpretability, latency, upkeep and stability against a small gain, and gives a real case. |
Listen for ownership. "We built a gradient boosting model" tells you nothing. "I chose it over the logistic baseline because the lift held up on the last two months of data" tells you what they decided.
Data handling and SQL
| Opening question | Follow-up | A strong answer |
|---|---|---|
| 9. Revenue per user dropped last week on the dashboard. What do you check first? | The raw table looks fine. What next? | Checks definitions, pipeline changes, duplicates and late-arriving data before touching a model. |
| 10. Write, or talk through, a query for each user's second purchase date. | What changes if a user has two purchases in the same second? | Uses a window function or ranking, handles ties and NULLs, and states the table's grain. |
| 11. A join doubled your row count. Why? | How would you catch that automatically next time? | Spots a many-to-many key, then adds a uniqueness or row-count check. |
| 12. 30% of values are missing in a key feature. What do you do? | When is dropping those rows defensible? | Asks why the values are missing before imputing, and runs a sensitivity check. |
For this group, a short exercise beats talk. Ask for the query on a shared screen or as a small task, then use the follow-up to discuss it. A candidate who wrote working SQL but cannot say what happens on a tie has memorised a pattern.
Metrics and product sense
| Opening question | Follow-up | A strong answer |
|---|---|---|
| 13. How would you tell whether a new search ranking is better? | Which metric could improve while the experience gets worse? | Pairs a main metric with guardrails, and separates a proxy from the outcome it stands for. |
| 14. Pick one metric for the health of a subscription app. | How could a team game it? | Names a leading and a lagging measure and the trade-off between them. |
| 15. A stakeholder wants a forecast by Friday and you have six weeks of data. What do you say? | What do you deliver instead? | Says what the data can and cannot support and offers a range with stated assumptions. |
| 16. Tell me about an analysis that changed a decision. | What did the decision-maker do differently? | Links the work to a specific action and says how they knew it mattered. |
Communication and ownership
| Opening question | Follow-up | A strong answer |
|---|---|---|
| 17. Explain the main result of your last project to someone who has not seen the data. | What would you leave out, and why? | Gives the answer in one sentence first, then the evidence, with no jargon. |
| 18. Tell me about a result you reported that turned out to be wrong. | How did you find out, and who did you tell? | Owns the error, says when and how they told people, and names what changed in their process. |
| 19. A senior colleague insists on a method you think is wrong. What do you do? | What evidence would change your mind, and theirs? | Proposes a test or a comparison, represents the other view fairly and is open to being wrong. |
| 20. How do you check your own work before you share it? | Which check has caught the most mistakes? | Names concrete checks: a baseline, re-running from raw data, a second pair of eyes, a sanity range. |
Question 18 is worth protecting when time is short. Everyone has a mistake to report, and how they tell the story shows whether they check their work.
What does a full follow-up chain look like?
This is an example we wrote to show the shape of a chain. It is not a transcript of a real interview. It starts from question 1.
Interviewer: An A/B test shows a 2% lift in conversion, p equals 0.04. Do you ship?
Candidate: Yes, it's significant, so I'd ship it.
Interviewer: How many metrics did the team look at in that test?
Candidate: Probably around ten. Conversion was the one that moved.
Interviewer: So what does a p-value of 0.04 mean when you looked at ten?
Candidate: One of them being under 0.05 by chance is fairly likely. I'd want conversion to have been the planned primary metric, and I'd check whether the lift holds in a rerun or on a holdout before shipping.
Interviewer: And if it was planned, and holds, what else could make it a bad ship?
Candidate: Novelty. A new layout can lift clicks for a week. I'd look at the lift by day, and check retention and refunds as guardrails.
The first answer is the one a list gives you. The third and fourth answers show who has run experiments. The three follow-ups take a minute or so. Follow-ups on a candidate's own answer are also a quiet check that the answer is theirs; our guide to AI interview cheating detection goes further.
Strong and weak answers side by side
Another example written by us, for question 6, the 98% accuracy question.
| Weak answer | Strong answer | |
|---|---|---|
| First answer | "98% is very high, so it's a good model." | "It depends on the base rate. If 98% of transactions are legitimate, predicting 'not fraud' every time scores 98%." |
| After "What would you measure instead?" | "F1 score." | "Precision and recall at a threshold set by cost. Missing fraud and blocking a good customer cost different amounts, so I'd ask the business for both." |
| After "Who sets the threshold?" | "The data science team." | "We propose one with the trade-off curve; the fraud and customer teams own the decision." |
The weak candidate is not wrong about F1. They stop when the question moves from a term to a decision.
Can you ask these questions of every applicant?
A live screen for every applicant is a real cost in interviewer time when a role draws many applications. Our guide to screening when there are too many applicants covers the options.
One option is to let an AI interviewer run the first conversation (see what an AI interviewer is if the term is new, or AI interview software compared). Tracy AI Interviewer runs two-way video interviews. Each applicant starts their own interview when they click the link, at any hour, in a browser with nothing to install, so a pool of hundreds is interviewed without a calendar. Three things matter for this role:
- It reads the CV. With CV check on, Tracy asks 3 to 5 questions tailored to the candidate's CV and the role's must-have requirements, on top of your questions. A claimed production model or experiment becomes a question about that model or experiment.
- It follows up and waits. Follow-ups build on what the candidate said, and Tracy waits for the end of a thought, with longer pauses allowed on hard technical questions, so a candidate working through a trade-off is not cut off.
- It can use your questions. Questions are generated from the job description and can be edited or replaced, so the 20 above fit as written. Interview style runs from quick to deep and from technical to behavioural.
What your team gets back is the recording of both sides with a transcript anchored to it by timestamp, a written assessment per question with the reasoning behind it, and integrity signals (where the candidate looked, response timing, tab switches, leaving fullscreen, an external screen) with the evidence, for a person to review. A signal is not a verdict. People are told about the recording before it starts, and Tracy does not reject anyone: a person reviews the evidence and decides. Our guide to AI interview reports shows what a reviewer should expect to see.
Tracy also connects through an API and an MCP server, so interviews can be created and read from the tools a team already works in, and enterprise teams can ask for a customised setup. For published prices across tools, see AI interview cost.
When to pick a test or a person instead
An AI interviewer is not the right tool for every part of a data science process.
- To see someone write and run code against real data, use a take-home or live exercise. The interview is for the conversation about it. For software roles, see AI technical interviews and the software engineer question bank for how to pair a task with follow-ups.
- For a final round on a senior role, a person should be in the room. Judgement on an ambiguous business problem, and the fit with the team, are worth an hour of a senior colleague's time once the shortlist is small.
- For a candidate who has not agreed to be recorded, use a live call. Tracy is not for talking to someone who has not agreed to be recorded.
Tracy AI Interviewer fits the stage before those: the first conversation with everyone who applies, so the shortlist your seniors meet has already been asked the follow-ups.
Questions
What should you ask in a data scientist interview?
How many questions fit in a 30-minute data science screen?
What is the difference between a strong and a weak answer?
Should a data scientist interview include a coding test?
Can an AI interviewer ask data scientist interview questions?
Try it free and run these questions on your own applicants, or book a demo.
Sources
All sources retrieved 3 Nov 2026.
- Reddit, Senior Data Scientist, Ads, posting updated 2 Nov 2026: skills, SQL, communication responsibilities.
- Airbnb, Senior Data Scientist, Guest & Host, posting updated 28 Oct 2026: experimentation, causal analysis, communication.
- DataAnnotation, home page role listings: Data Scientist role and hourly rate as listed by the platform.
- Sackett, Zhang, Berry and Lievens, Journal of Applied Psychology, Revisiting meta-analytic estimates of validity in personnel selection, 2022: structured interviews as the top-ranked selection procedure.
- Tracy AI Interviewer, product page: two-way video interview, questions from the job description or your own, CV check, follow-ups, recording notice, recording and transcript timed to it, an assessment per question, integrity signals, API and MCP.
- Tracy AI Interviewer, home page: a person reviews the evidence and decides; integrity signals with their evidence.
- Tracy AI Interviewer, pricing: API and MCP connections and a customised setup for Enterprise.
- Tracy AI Interviewer, Connect page: MCP server and API connection to the tools a team already uses.