Interviewing

How to Build a Better Technical Interview Scorecard

Most technical interviews measure who interviews well, not who will do the job well. A structured scorecard fixes that — turning gut feel into consistent, comparable signal across every candidate and every interviewer.

Ask five engineers who just interviewed the same candidate how it went, and you will often get five different answers — not because they disagree on the facts, but because each was measuring something different. One was assessing system design, another coding fluency, a third whether they would enjoy working with the person. Without a shared instrument, a technical interview is five private impressions stitched together by whoever speaks most confidently in the debrief. A scorecard is that shared instrument. Done well, it turns scattered opinion into structured, comparable evidence.

Why Unstructured Interviews Fail

Decades of hiring research point to the same conclusion: unstructured interviews are among the weakest predictors of job performance, while structured, criteria-based evaluations are among the strongest. The reason is simple. Unstructured interviews are vulnerable to affinity bias, recency bias, and the halo effect — a candidate who is articulate and likeable reads as competent, while a strong engineer who is quiet or nervous reads as weak. A scorecard does not remove judgment; it disciplines it, forcing every interviewer to evaluate the same dimensions against the same standard.

Start With the Role, Not the Questions

A good scorecard is derived from the role definition, not from a generic question bank. Before writing a single question, articulate what this person must actually be able to do: the systems they will own, the problems they will solve, and the level of independence the job demands. Every dimension on the scorecard should trace back to a real requirement of the role. If a criterion does not map to something the person will do on the job, it does not belong on the card — it is just an interviewer's personal preference dressed up as a standard.

Define the Dimensions That Matter

Most technical roles can be evaluated across four to six dimensions: technical depth in the relevant domain, problem-solving and reasoning, code or design quality, communication and collaboration, and ownership or judgment. Resist the urge to add more. A scorecard with fifteen dimensions is not more rigorous — it is unusable, and interviewers will quietly collapse it back into an overall gut feel. Pick the handful of dimensions that genuinely separate a strong hire from a weak one for this specific role, and define what each one means in plain language.

Use a Consistent Rating Scale

Adopt a single rating scale and use it everywhere — a four-point scale works well because it forces a decision and removes the safe middle. Define each point concretely: what does a 1 look like versus a 4 on technical depth? Anchored scales, where each rating has a written description of the behavior it represents, dramatically improve consistency between interviewers. Avoid five- or ten-point scales; the extra granularity is false precision that interviewers cannot apply reliably, and it invites everyone to cluster around the noncommittal middle.

Separate Observation From Judgment

The single most valuable habit a scorecard can enforce is separating what the interviewer observed from what they concluded. Require interviewers to write down specific evidence — what the candidate actually said or did — before assigning a rating. "Designed a clean caching layer but did not consider cache invalidation until prompted" is evidence. "Seemed senior" is not. Evidence-based notes make the debrief faster, make ratings defensible, and surface disagreements that are real rather than stylistic.

Calibrate Before You Interview

A scorecard is only as consistent as the people using it. Before a search begins, get the interview panel together and align on what each rating means, ideally by scoring a past candidate or a worked example together. This calibration step takes thirty minutes and prevents the most common failure mode: two interviewers giving the same performance a 2 and a 4 because they hold silently different bars. Calibration is also where you catch criteria that are ambiguous or redundant, before they cost you a real candidate.

What a Good Scorecard Produces

A well-built scorecard changes the debrief entirely. Instead of a conversation driven by whoever feels most strongly, you have structured evidence across the same dimensions for every candidate — easy to compare, hard to game, and resistant to the biases that quietly drive bad hires. It will not make the decision for you, and it should not. But it ensures the decision is based on what the job actually requires, evaluated the same way for everyone. That is the difference between hiring the person who interviews best and hiring the person who will do the job best.