Skip to content

Using ChatGPT during coding interviews: test judgement, not memory

In short

Candidates told to use ChatGPT passed 73% of verbatim LeetCode questions in interviewing.io's experiment and 25% of custom ones, and no interviewer raised cheating. Tracy AI Interviewer runs a coding task inside the interview, asks follow-ups on approach and trade-offs, and shows integrity signals with their evidence beside a recording and transcript. You set the AI rule, and an engineer reviews and decides.

Using ChatGPT during coding interviews: where it helps and where it stops

Using ChatGPT during coding interviews works on familiar questions and stops on custom ones. This page is for interviewers. What helps is a rule you state up front, questions the model has not seen, and follow-ups on the candidate's own approach. Candidates using ChatGPT in interviews covers the signs that apply to any interview.

A rule sets what is allowed, and a custom question tests it. The step neither covers is the follow-up on the person's own choice, asked of every candidate the same way, and that is the step Tracy runs for each applicant.

One public experiment is from interviewing.io. It ran 37 mock interviews, and 32 were usable. Professional interviewers were told the study was about how predictable their decisions are. The candidates had 4 or more years of experience, and each was told to use ChatGPT instead of their own skills. The interviews were audio only, and the post carries no publication date.

Not one interviewer raised a concern about cheating. The pass rate depended on the question: 73% for verbatim LeetCode questions, 67% for modified ones and 25% for custom ones. The control group, interviewing.io's other mock interviews, passed 53%, and the author says that figure is probably higher than real-world pass rates. The sample is small and audio only, so read the figures as a direction.

Bar chart of pass rates: control 53%, verbatim 73%, modified 67%, custom 25%.
Pass rates by question type in interviewing.io's experiment, where candidates were told to use ChatGPT: 11 verbatim, 9 modified and 12 custom interviews. The control bar is its other mock interviews. interviewing.io experiment, 32 usable audio-only interviews, publication date not stated, read 14 Nov 2026.

A model handles standard questions well and custom ones poorly.

Should you allow ChatGPT in a coding interview?

Pick one rule and write it in the invitation.

  • If you ban AI help, you are testing what the person can do alone. You cannot stop a second screen, so you also need questions and follow-ups that ask for the candidate's own reasoning.
  • If you allow AI help, you are testing how the person works with it. Grade the prompt they wrote, what they changed in the output and whether they can explain the result.

Teams may prefer the second rule when their engineers use these tools daily, and the first for a screen of fundamentals. Under either rule, the follow-up chain below is the same.

What can the invitation say about AI help?

Write the rule in plain words before the interview, so no candidate has to guess. These are samples we wrote, to adapt to your own policy.

  • Sample, AI help not allowed: "This interview tests how you work on your own. Please do not use AI tools, search or a second screen. We will ask you to explain your choices as you go."
  • Sample, AI help allowed: "You may use an AI assistant. We will ask how you used it, what you changed in its output and why you trust the result."

Add one line on what the candidate will see, such as a recording notice, so the rule and the recording are in the same message. A rule that was never written down is hard to enforce afterwards.

How do you keep the signal when ChatGPT can solve the problem?

The pass rates above set the target: 73% on a verbatim question and 25% on a custom one, so the work is in moving the question away from what the model has seen, then asking about the person's own choices. The interviewing.io author found that a new story around the same inputs and outputs was still easy for ChatGPT, and recommended custom questions and one or two unique twists.

  1. Change the inputs and outputs, not only the story, because a familiar problem with a new name is still familiar to the model.
  2. Ask for the approach before the code. A person who understands can say it in plain words first.
  3. Add a twist after the first solution: "The data no longer fits in memory." "Requests now arrive out of order."
  4. Ask about their own choice: "Why this data structure, and what would make you swap it?"
  5. Ask what they would test. Edge cases come from understanding the problem, so the answer shows how the person thinks about it.

Listen for reasoning that changes when the constraint changes. For a longer set of questions by problem, see coding interview follow-up questions.

Sample transcript: Tracy asks about data too big for memory, the answer is vague, Tracy asks which structure.
Tracy asks what changes when the data no longer fits in memory, then asks where it would live. Illustration with sample data

A twist on the candidate's own choice asks for reasoning the model has not been handed.

What Tracy adds to a coding interview

  1. Every applicant starts on click, at any hour, with no scheduling and no engineer on the call, including candidates your team had no time to interview.
  2. Tracy asks the next question on approach and trade-offs and waits for the end of each thought, so a twist on the candidate's own choice reaches everyone.
  3. Every result keeps a recording, a timestamped transcript and what the person said to each question, with integrity signals and their evidence. A signal is a reason to look, not proof.
Four steps: write the rule, coding task, ask the why, review evidence.
Tracy runs the coding task and the follow-ups for each applicant, and the engineer reviews the evidence. Illustration, no data.

You write the AI rule once; Tracy runs the task and the follow-ups for every applicant.

How does Tracy AI Interviewer run a coding interview with this in mind?

Tracy puts a coding task inside the interview, then asks follow-ups on approach and trade-offs and waits for the end of each thought. Every applicant starts on click, at any hour, so the same follow-ups reach every candidate, including those your engineers had no time to interview. See AI technical interviews and AI interviews for software engineers.

On video interviews she also captures tab changes, abnormal eye tracking (gaze), live face detection, delay before answers, leaving fullscreen and external screens, unless you turn them off. An engineer reviews each signal in the report with its times, beside a recording and a timestamped transcript, along with what the person said to each question. A signal is a reason to look, not proof. The candidate is told about the recording on the join screen, a person decides, and Tracy gives no detection rate.

What should you do when an answer or a signal looks off?

Read the transcript at that minute, compare the answer with the candidate's earlier answers, and ask the person to walk through the code in a short live call. Use the same questions you would ask anyone, and record the decision and the evidence for a second reviewer.

For what a signal can and cannot show, see AI interview cheating detection. For the wider picture of identity and proxy risks, see candidate fraud detection.

Common questions

Is using ChatGPT in a coding interview cheating?
It depends on the rule the hirer set. If the invitation says no AI help, reading answers from a tool breaks it. If AI help is allowed, the question becomes how well the candidate uses it. Say which rule applies before the interview.
Can ChatGPT pass a coding interview?
In interviewing.io's experiment of 32 audio-only interviews, candidates told to use it passed 73% of verbatim LeetCode questions and 67% of modified ones, but 25% of custom questions. Custom questions gave it the most trouble. The experiment compared question types only and did not test follow-ups. Follow-ups on the candidate's own choices are a separate technique, covered in the section on keeping the signal.
Can interviewers tell when a candidate uses ChatGPT?
Often not by watching, in that experiment: no interviewer raised a concern about cheating. Signals such as delay before answers or gaze are reasons to ask a question, never proof.
Can you stop candidates from using ChatGPT in a remote coding interview?
Not fully. Video makes it harder but does not remove it, per the interviewing.io author. Build the interview so a pasted answer is not enough, and let a person review any signal.
Should we let candidates use ChatGPT in coding interviews?
Either rule works if it is stated. Ban it to test fundamentals, or allow it and grade the prompts, the edits and the explanation. Follow-ups on approach and trade-offs work under both.

Run one coding task as an interview

Create an interview with a coding task and read the follow-ups on approach, the signals and the transcript yourself. The first 3 results are free.

Try it free (opens in a new tab) Book a demo (opens in a new tab)

Your first 3 results are free. No card needed.

Sources