Skip to content

Find the ML engineers who can say why a model fails

In short

These machine learning engineer interview questions test whether a candidate can frame a problem, trust an evaluation and keep a model working after launch, with a follow-up for each of the 15. Tracy AI Interviewer asks them of every applicant, coding task included, follows the answer into the reasoning, and returns the recording and transcript. Your ML lead reads the evidence and decides.

What good machine learning engineer interview questions do

Say a machine learning engineer role draws 400 applications. Every CV lists PyTorch, scikit-learn and a Kaggle result. The CV cannot tell you who has found out why a model that scored well offline did badly in production, and who has only tuned a notebook. Good machine learning engineer interview questions ask what they did when the live numbers disagreed with the test set, and each question below has a follow-up that goes after the cause.

ML leads, hiring managers and recruiters can run these 15 machine learning engineer interview questions in a first technical conversation of 20 to 30 minutes. A take-home or a deep modelling review is better kept for your own team to do later.

Tracy AI Interviewer asks that second question of every applicant, coding task included. If a candidate would retrain, the next question is how they would know the data was the cause. For the wider process, see technical interviews with Tracy. For the neighbouring role, see AI engineer interview questions.

What should a machine learning engineer interview test?

It should test whether the candidate can frame a problem, trust an evaluation and keep a model working after launch. Algorithm names come second.

Fifteen machine learning engineer interview questions, grouped by what they test

From each group, select one or two. Ask the follow-up when the first answer ends. Fix what a strong answer contains ahead of the first interview.

Group 1: Framing the problem

  1. 1. A product team asks for "a model to predict churn". What do you ask before you build anything?

    Follow up with
    What would make you say a model is not needed?
    Listen for
    The decision it supports, how churn is defined, the time window, a simple rule as baseline, the cost of each error.
  2. 2. How do you choose a metric when one class is rare?

    Follow up with
    A model has 98% accuracy and is useless. How?
    Listen for
    Precision and recall, the cost of each error, a threshold set with the business. Accuracy hides the rare class.
  3. 3. How do you set a baseline?

    Follow up with
    How do you know a gain over it is real?
    Listen for
    A simple baseline on the same split. Variation across seeds or folds, not one run.

Group 2: Data and evaluation

  1. 4. Offline results are strong and production results are poor. Where do you look?

    Follow up with
    How do you tell which cause it is?
    Listen for
    Leakage, training and serving differences, drift, label delay. Compares feature distributions.
  2. 5. How do you split data for a model that predicts the future?

    Follow up with
    What goes wrong with a random split?
    Listen for
    A split by time. Future information leaking into training. The same user on both sides.
  3. 6. One feature is extremely predictive and you are suspicious. Why?

    Follow up with
    How do you check?
    Listen for
    It may only exist after the outcome. Checks when the value becomes available, runs without it.

Group 3: Modelling

  1. 7. A model overfits. What do you do?

    Follow up with
    What if you cannot get more data?
    Listen for
    Looks at validation curves. Regularisation, a simpler model, early stopping, augmentation.
  2. 8. When would you choose gradient-boosted trees over a neural network?

    Follow up with
    When would you change your mind?
    Listen for
    Tabular and smaller data, speed, explainability. Neural networks for images and text.
  3. 9. Write code that computes precision and recall from predicted and true labels.

    Follow up with
    What do you return when there are no positive predictions?
    Listen for
    Counts true and false positives and false negatives. Handles a zero denominator and says the convention.

Group 4: Production

  1. 10. How do you deploy a model so you can roll it back?

    Follow up with
    How do you know the new one is better in production?
    Listen for
    Versioned artefacts, a shadow or canary release, a live metric, a rollback path.
  2. 11. How do you monitor a model after launch?

    Follow up with
    Inputs drift but accuracy looks fine. What do you do?
    Listen for
    Input and prediction distributions, delayed labels, alert thresholds, retraining triggers.
  3. 12. Latency doubled after a model change. What do you check?

    Follow up with
    What would you trade away?
    Listen for
    Model size, batching, feature fetching, hardware, caching. Trades accuracy for speed with the owner.

Group 5: Judgement

  1. 13. Tell me about a model you shipped that did not work as expected.

    Follow up with
    What did it teach you about the data?
    Listen for
    A real case, what they measured, what they changed.
  2. 14. How do you explain a prediction to someone who is not technical?

    Follow up with
    What do you refuse to claim?
    Listen for
    Plain words, uncertainty stated. No causal claim from a correlated feature.
  3. 15. How do you decide a model is good enough to ship?

    Follow up with
    Who signs off?
    Listen for
    Tied to a decision and the cost of errors. Guardrails, review of risky cases, a monitoring plan.

Weak and strong answers to the same questions

Each row answers one question two ways.

Table 6: Weak and strong answers to the same questions, written as examples.

Weak answerStrong answer
Question 4, the offline gap"The production data is noisier.""I would first check for leakage, then compare feature distributions between training and live traffic, and check whether labels arrive late."
Question 6, the suspicious feature"It is a strong feature, so keep it.""I would ask when that value is known. If it is only filled in after the outcome, it is leakage, and I would retrain without it."
Question 13, the model that failed"We retrained it and it improved.""Live accuracy fell because a partner changed a field. We added an input check and an alert before retraining, so we would catch it earlier."

A follow-up chain on the model that fails live

This chain is an illustration. The data and the candidate are fictional, and no real interview sits behind it. It starts from question 4.

Tracy: Offline results are strong and production results are poor. Where do you look?

Candidate (fictional): I would retrain on more recent data and see if it helps.

Tracy: How would you find out whether the data is the cause?

Candidate: Compare each feature's distribution in training with live traffic, and check it is computed the same way in both.

Tracy: Everything matches. What next?

Candidate: Check for leakage, a feature only known after the outcome.

Transcript card: offline results strong, production poor; candidate says retrain; Tracy asks how to find the cause.
Tracy's second question asks how the candidate would find out whether the data is the cause, so your ML lead hears a method or a guess. Illustration with sample data

A second question built on the candidate's own answer shows whether they have a method for finding the cause.

What a list of questions cannot do alone

The chain above shows the gap. A list can contain "How would you find out whether the data is the cause?", but that question only exists once the candidate has said they would retrain. It is asked in response to an answer, so it cannot go out as a form. For 400 applicants, a 30-minute call each is 200 hours, which is five working weeks for one person. Tracy asks the second question as each applicant answers the first, and your ML lead reads the chains instead of holding the calls.

Four steps: you pick questions, Tracy asks every applicant, Tracy follows up, your ML lead decides.
Tracy asks the questions you picked, coding task included, and follows each answer into the reasoning, so your ML lead reads evidence from every applicant. Illustration.

Every applicant gets the same opening question and a follow-up built on their own answer.

What Tracy adds to machine learning engineer interviews

01

Reach every applicant

Each applicant can start on click, at any hour, with no scheduling and no recruiter on the call. A browser link is all they need, with no account or calendar. The join screen states that the conversation is recorded and transcribed; see what the candidate goes through.

02

Ask the next question

Tracy asks in the candidate's own field, follows what they said and waits for the answer. A coding task can be part of the interview, and the follow-ups cover approach and trade-offs. Question 9 works as a short task.

03

Keep the evidence

Each result includes a recording, a timestamped transcript and a written assessment per question with its reasoning, plus what the candidate said to each question and the result fields. See the answers and fields a report holds.

One customer cut its average time to hire from 44 days to 7, and Tracy did the work of 53 full-time recruiters in one month.

Tracy drafts the questions from the role you describe, and you edit them or bring your own; see where the drafts come from. CV check is a setting. The interview also captures integrity signals: tab changes, gaze, face detection, delay before answers, fullscreen exits and external screens. Your ML lead sees the evidence behind each, and a signal is not proof.

Recreated Tracy result report for a fictional ML engineer: recording, transcript and quoted evidence per question.
The ML lead can set the "what they said" line against the recording and transcript before moving anyone on. Illustration with sample data

Your ML lead can open the recording and the transcript behind each answer before they decide.

Keep three decisions with your ML lead

  1. Who moves forward: a senior ML engineer reads the report and decides, working from the candidate's quoted answers.
  2. Calibration: a second ML engineer reads five transcripts without your lead's notes. If the two reads diverge, revise the questions before the next hundred invitations.
  3. The modelling round: a real dataset, a take-home or a paper discussion stays with your team.

For two hires a year, a call with your own ML lead is simpler; see when your own lead's call is enough. With hundreds of applicants, volume hiring is where Tracy fits. To try it, put three of these questions into one open ML role and have your ML lead read the first batch of reports. For the data side of the same team, see data engineer interview questions, and for the coding language, Python interview questions.

Common questions

What are good machine learning engineer interview questions and answers?
Ask about framing a problem, evaluation, leakage, production monitoring and a model that failed. The "listen for" column gives the points a strong answer contains, and the follow-up beside each question tests the first answer.
Can Tracy run a coding task in a machine learning interview?
Yes. A coding task fits inside the same interview, and the follow-ups ask about approach and trade-offs. What the candidate said to each question is in the report, with the recording and transcript.
Does Tracy decide who to hire?
No. Your ML lead reads what was said, with the evidence, and decides.

Run these questions on your open ML role

Paste three questions into a Tracy interview and read the first report. The first 3 results are free.

Try it free (opens in a new tab) Book a demo (opens in a new tab)

Your first 3 results are free. No card needed.

Sources