Guides · carbon.talent
Data scientist interview questions to ask candidates
Last updated 18 Oct 2026
Good data scientist interviews test judgement more than algorithms: framing a vague business question, choosing a simple baseline, designing a fair experiment, spotting leakage and explaining the result to a non-technical person. The questions below are grouped by those skills, with what strong and weak answers sound like, a data scientist assessment plan and a scoring guide.
1. Framing the problem
"Our churn went up last quarter. The CEO wants a model. What do you do first?"
- Strong: Asks what decision the model will drive and who acts on it, defines churn precisely, checks whether a simple analysis (cohorts, segments) answers it before modelling, and asks how success will be measured.
- Weak: Jumps to "I'd train XGBoost".
"When should we not use machine learning for this?"
- Strong: When a rule or a dashboard is good enough, data is too thin, or no one can act on the prediction.
2. Experimentation and statistics
"Design an A/B test for a new onboarding flow."
- Strong: One primary metric chosen in advance, guardrail metrics, randomisation unit, sample size from a minimum detectable effect, fixed run time, no peeking (or a sequential method), checks for novelty effects and sample-ratio mismatch.
- Weak: "Run it until it's significant."
"The test shows a significant lift but revenue didn't move. Why might that be?"
- Strong: Wrong metric, novelty effect, multiple testing, segment cannibalisation, the effect is real but small relative to noise in revenue.
"Explain a p-value to our head of sales."
- Strong: Plain language, no "probability the hypothesis is true", gives a practical takeaway.
3. Modelling
"Your model has excellent validation accuracy but fails in production. What happened?"
- Strong: Leakage (future information in features), train/serve skew, distribution shift, wrong split (random instead of time-based), class imbalance hiding behind accuracy.
"How do you choose an evaluation metric for a fraud model?"
- Strong: Ties it to cost: precision/recall trade-off at an operating threshold, PR-AUC over ROC-AUC on rare events, the cost of a false positive vs a missed fraud.
"What's your baseline?"
- Strong: Always starts with one (last value, simple rule, logistic regression) and beats it meaningfully before adding complexity.
4. Data and SQL
"Build a feature table of customer behaviour for a churn model, as of each month-end."
- Strong: Point-in-time correctness (only data available at that date), window functions, handles customers with gaps, documents the definition.
5. LLMs and AI features (if relevant)
"We want an LLM to answer support tickets. How would you evaluate it before launch?"
- Strong: Builds a labelled evaluation set from real tickets, defines failure types (wrong, unsafe, unhelpful), measures against a baseline, human review on a sample, monitors after launch.
6. Communication and impact
"Tell me about an analysis or model that changed a decision. What would you do differently?"
- Strong: Specific decision, measured outcome, honest about what went wrong.
Data scientist assessment test: a fair plan (SK4)
- Scenario interview on framing and experimentation (sections 1–2).
- Short practical, under two hours: a small, realistic dataset with one planted problem (leakage or a biased sample). Ask for a short written recommendation, not a notebook dump. Discuss it live.
- Presentation to a non-technical interviewer (ten minutes) on that recommendation.
- Team conversation about how they work with engineers and product. On carbon.talent, vetted profiles show a vetting report from Carbon's real-world data experience test, which has a DS/ML judgement track (scenario-based, not a coding test), plus a working-style profile and a short video intro.
Scoring guide
| Area | 1 | 3 | 5 |
|---|---|---|---|
| Framing | Jumps to a model | Clarifies the decision | Reframes to the highest-value question |
| Experimentation | Peeks, one metric | Sound design | Anticipates pitfalls, sizes correctly |
| Modelling | Complex first | Baseline and right metric | Catches leakage and shift unprompted |
| Communication | Jargon | Clear | Changes the listener's decision |
Questions companies ask
- What questions should I ask a data scientist in an interview?
- Ask them to frame a vague business problem, design an A/B test, explain why a model that validated well failed in production, choose a metric tied to business cost, and explain a result to a non-technical person.
- What is a good data scientist assessment test?
- A scenario interview plus a short, realistic practical with a planted problem such as leakage, followed by a written recommendation discussed live. Keep it under two hours.
- How do I interview a data scientist if I'm not technical?
- Focus on framing and communication: ask what decision their past work changed and how they measured it, and have a technical peer join for the modelling questions.
- What's the difference between interviewing a data scientist and a data analyst?
- Analyst interviews centre on SQL, metrics and clear recommendations. Data scientist interviews add experiment design, modelling judgement and evaluation.
- How many rounds should a data scientist interview have?
- Three or four: intro, scenario interview, practical with live discussion, and a team conversation.