TLDR Tracy AI Interviewer interviews every AI engineer applicant, follows up on trade-offs and failure cases, and can run a coding task inside the same interview, so you read how they reason about systems that fail.
Use the 15 AI engineer interview questions below, each with a follow-up and what to listen for.
A recording, a timestamped transcript and a written assessment per question sit behind every result, and your team decides.
Next step: paste five of these questions into a Tracy interview.
We make Tracy AI Interviewer.
Applications for an AI engineer role arrive with the same words on every CV: LLMs, RAG, fine-tuning, evals. Your head of engineering, or the person running lab operations, cannot interview all of them. The CV shows what someone has built with. It does not show how they behave when the system gives a confident wrong answer. AI engineer interview questions should test that.
These are written for the interviewer. They ask about evaluation, retrieval, cost, safety and shipping, and each has a follow-up and a note on what a strong answer contains. They avoid trivia about one framework, which goes out of date in a quarter. Candidates can use them to prepare.
Tracy AI Interviewer runs the part that does not scale. She asks each applicant in their own field, follows up on what they just said and waits for the end of the thought, with a coding task inside the same interview when you want one. For the wider engineering process, see AI technical interviews, and for specialist vetting, expert vetting.
Fifteen AI engineer interview questions, grouped by what they test
Pick one or two questions from each group. Ask the follow-up only after the first answer. Choose your opening questions for the role and write down what a strong answer contains before the first candidate joins. Ask for a system the candidate shipped, because a rehearsed answer about a famous model is easy to find and a failure story is not.
Group 1: evaluation
Table 1: Evaluation, with follow-ups and what a strong answer contains.
| Ask | Follow up with | Listen for |
|---|---|---|
| 1. You changed a prompt and the demo looks better. How do you know it is better? | What would make you revert it? | A fixed set of cases, a baseline, a comparison on the same inputs. Names a regression they would accept or not. |
| 2. How do you build an evaluation set for a task with no single right answer? | How do you keep it from going stale? | Real examples, written criteria, a second reviewer, refreshed from failures. Avoids one model grading its own output without checks. |
| 3. Tell me about a metric that looked good and was wrong. | What did you use instead? | A real case, the cause, and a change of metric or sample. |
Group 2: retrieval and data
Table 2: Retrieval and data, with follow-ups and what a strong answer contains.
| Ask | Follow up with | Listen for |
|---|---|---|
| 4. A retrieval system returns plausible but wrong passages. Where do you look? | How would you tell chunking from embedding trouble? | Reads failing queries, separates retrieval from generation, checks chunk boundaries and filters before swapping models. |
| 5. When would you fine-tune instead of improving the prompt or retrieval? | What did it cost you the last time? | Names the trigger, such as style or latency at volume, and the cost: data, drift, maintenance. |
| 6. How do you treat training or evaluation data that may contain personal information? | Who do you ask first? | Removes or avoids it, asks the data owner and the legal team, keeps a record. |
Group 3: trade-offs in production
Table 3: Trade-offs in production, with follow-ups and what a strong answer contains.
| Ask | Follow up with | Listen for |
|---|---|---|
| 7. A feature costs too much per request. What are your levers? | Which would you pull first? | Shorter context, caching, a smaller model on easy cases, batching. Measures quality after each. |
| 8. Latency doubled after a release. How do you find the cause? | What would you add so you see it sooner? | Traces by stage, compares versions, finds tokens or retries. Adds a dashboard or alert. |
| 9. How do you decide between an API model and one you host? | What would change your answer? | Volume, privacy, latency and maintenance. Gives a number or a threshold, not a preference. |
Group 4: failure cases and safety
Table 4: Failure cases and safety, with follow-ups and what a strong answer contains.
| Ask | Follow up with | Listen for |
|---|---|---|
| 10. Tell me about a failure your system had in production. | What did you change afterwards? | A specific incident, the signal that caught it, a fix and a test added. |
| 11. How would you stop a model from acting on instructions hidden in a document it reads? | What would you not rely on? | Separates instructions from data, limits tool permissions, asks for confirmation. Says a prompt alone is not enough. |
| 12. A model refuses a harmless request. How do you handle it? | How do you know the fix did not open something else? | Collects examples, changes one thing, re-runs the safety cases as well as the helpful ones. |
Group 5: shipping with a team
Table 5: Shipping with a team, with follow-ups and what a strong answer contains.
| Ask | Follow up with | Listen for |
|---|---|---|
| 13. How do you explain to a product manager why a feature cannot be exact? | What do you offer instead? | Plain language about error rates, a fallback or a review step, an agreed threshold. |
| 14. You are asked to ship in a week and the evals are not ready. What do you do? | What would you refuse to skip? | Proposes a smaller launch, keeps the key checks, says what is accepted and who agreed. |
| 15. What did you last learn from reading someone else's code or paper, and what did you do with it? | What did you decide not to copy? | A specific source and a specific change. Can say why it did not apply to their case. |
A strong answer names what would change its mind, and a weak one names a model
The difference shows up in the second answer. These pairs are written by us as examples of how it sounds, not quotes from real candidates.
Table 6: Weak and strong answers to the same questions, written as examples.
| Weak answer | Strong answer | |
|---|---|---|
| Question 1, the better prompt | "The outputs read more naturally now." | "I ran it on the same 80 cases as before. It improved tone and dropped accuracy on dates, so I would not ship it yet." |
| Question 4, wrong passages | "I would switch to a bigger embedding model." | "I would read twenty failing queries first. If the right passage is not retrieved at all it is retrieval, not generation." |
| Question 10, a production failure | "We had no major failures." | "The bot quoted a refund policy from an old document. We added a freshness filter and a test for it." |
Listen for ownership. "The model hallucinated" tells you about the model. "I read the failing cases and found the filter" tells you about the person.
One follow-up chain shows where the real answer starts
This is an illustration with sample data, not a real interview. The candidate is fictional. It shows the shape of what Tracy runs: an opening question from your list, the answer, and a follow-up built on it. It starts from question 1.
Tracy: You changed a prompt and the demo looks better. How do you know it is better?
Candidate (fictional): I run it on a fixed set of cases and compare with the old prompt.
Tracy: What was in that set the last time you did this?
Candidate: Eighty real support questions. The new prompt was friendlier but got dates wrong in eleven of them.
Tracy: What did you decide?
Candidate: I did not ship it. I added date cases to the set and tried again.
The first answer is the textbook one. The second shows whether it ever happened. Roy Junyoung Jang, Director, Residency & Investment at Antler Korea, says Tracy "sometimes brings up exactly those questions I might have forgotten to ask, which has been a pleasant surprise."
A follow-up built on the candidate's own answer shows whether they have done the work.
Run the interview for every applicant without losing the follow-up
A live call for every applicant does not survive a hundred of them, and a specialist pool is often the scarcest resource in the building.
- Your questions or ours. Describe the role and Tracy drafts questions from the job description, which you edit (see the product page).
- A coding task in the same interview. Live coding, debugging or multiple choice, with follow-ups on approach, trade-offs and failure cases.
- CV check, as a setting. With it on, Tracy can read the CV and ask about what it claims, so "built a RAG system" becomes a question about that system. See how the CV check works.
- Start on click. Each candidate opens a link in a browser, with no account and no calendar, at any hour, and is told on the join screen that the conversation is recorded and transcribed. See the participant experience.
- A result you can check. A recording, a timestamped transcript and a written assessment per question sit behind every result. See AI interview reports.
It is $1 per credit, one credit per started 20 minutes of an interview that produced a result, so a 34-minute interview costs $2. The first 3 results are free with no card, there are no seats, and a session with no answers is never charged. See pricing. Customisation takes days: send us one role and we will build the interview plan.
A reviewer can open the recording and transcript behind each assessment.
Keep four checks with people
- Your own engineer. Have one score five or six transcripts blind. If they disagree with the written assessments, fix the questions before inviting the next hundred.
- The work. Tracy asks about approach and trade-offs. Whether a design or a piece of code meets your bar is a reviewer's call.
- Integrity signals. Tracy captures tab changes, gaze, face detection, delay before answers, fullscreen exits and external screens, and shows the evidence to a person. A signal is not proof, and with tools like chat assistants everywhere, it is the follow-up that tests the candidate. See AI interview cheating detection.
- The decision. A person decides from the evidence.
For ten applicants, a video call with one of your engineers is simple and works well; see AI interview vs human interview. The case for Tracy starts when applicants outnumber your engineers' hours. To try it, create a Tracy interview for this exact role from these questions and read the recording and report on three real applicants.
Questions
What are good AI engineer interview questions?
Where can candidates find AI engineer interview questions on GitHub?
How do you test an AI engineer without a trivia quiz?
Can an AI interviewer interview AI engineers?
Should candidates be allowed to use AI tools in the interview?
Create a Tracy interview for this exact role from these questions, then read the recording and report; the button opens the Connect page. Or send us one role and we will build the interview plan, with a recording and report you can check. Try it free · Book a demo
Sources
- Tracy AI Interviewer home page, customer proof, retrieved 9 Nov 2026.
- Tracy AI Interviewer pricing, retrieved 9 Nov 2026.
- Tracy AI Interviewer Connect page, retrieved 9 Nov 2026.