What Share of Interview Scorecards Use Unsupported Ratings?

Research question and decision supported

The question is what share of interview scorecard ratings lack evidence that supports the score. The decision supported is whether the scorecard criteria, the interviewer calibration, or the recording process needs repair before the next hiring round. A rating can be honest and still be unsupported if the interviewer did not record what they observed. The method measures the written evidence, not the interviewer's private impression.

State whether the audit covers one role, one interviewer group, or all open roles. A technical role and a customer support role may use different evidence types. Report the scope beside the rate so the result is not applied to roles the audit did not cover. The decision should follow the scope that matters to the hiring manager.

Define a supported rating

A supported rating points to observable evidence such as a work sample result, a specific answer, or a documented example from the candidate's history. The evidence should be sufficient for another reviewer to understand why the score was given. A comment such as strong communication is a conclusion, not evidence. A rating can be low and still be supported if the interviewer records the missing skill or the failed example.

Write examples for each rating level. Include a high score with evidence, a low score with evidence, and a rating that sounds specific but only restates the criterion. Review the examples with the interviewers before the audit. A shared definition prevents the audit from measuring whether the reviewer likes the writing style.

Draw the sample

Select scorecards from a defined hiring period and include every interviewer and role in scope. Include candidates who advanced and candidates who were rejected, because a rejected candidate may receive less evidence. Exclude scorecards that were never submitted or that belong to a withdrawn candidate. Record the role, interviewer, rating, and whether a work sample was used. If the volume is low, review every scorecard and report the count.

Stratify by role and interviewer so one large hiring lane does not dominate the result. Check that the sample includes scorecards completed early and late in the process. A scorecard completed after the hiring decision may be influenced by the outcome. Note the timing and treat it as a separate factor in the analysis.

Measure unsupported ratings by criterion

Report the share of ratings without evidence for each criterion and rating level. A criterion that consistently lacks evidence may be poorly defined or impossible to observe in the interview. A high rate at the top and bottom of the scale may indicate that interviewers use the extremes without recording the reason. Show the number of scorecards and ratings behind each percentage. A single aggregate rate can hide a serious problem in one criterion.

Separate a missing note from a contradictory note. A missing note is blank, while a contradictory note records evidence that points to a different score. The two need different fixes. Report both and read a few examples in each category. A contradictory note may reveal a criterion that interviewers interpret differently.

Compare by interviewer and role

Group the results by interviewer and role, but check the number of scorecards before comparing. An interviewer with two interviews may look very different from one with twenty. Look for a pattern across roles or criteria rather than a single outlier. Ask interviewers to explain one supported and one unsupported rating in their own words. Their explanation may reveal a training gap or a criterion that needs an example.

Do not use the rate to rank interviewers publicly. The purpose is to improve the evidence standard and the calibration. Share the aggregate pattern and the examples, then give each interviewer a chance to correct the record. A blame-oriented review will produce cleaner-looking notes rather than better evidence.

Test the scoring guide

Take ten unsupported ratings and ask two interviewers to rewrite the evidence using the current guide. If they cannot produce a specific observation, the criterion may be too abstract for the interview format. Add a prompt, a work sample, or a rating anchor that describes what evidence looks like. Test the revised guide on a new set of interviews and compare the same criteria.

Check that the guide does not encourage invented evidence. A prompt should help the interviewer record what happened, not fill a blank with a plausible story. Review a sample of revised scorecards with the hiring manager. If the evidence is still vague, change the interview process rather than asking for more writing.

Use the result to change one control

Choose the most consequential gap and change one control. If a criterion lacks evidence, add a structured question or a work sample. If interviewers are unsure what counts as evidence, add a calibration round. If scorecards are completed after the decision, require submission before the debrief. Assign an owner and a review date, then repeat the audit with the same definitions and roles.

Confirm that the change was delivered and used. Check the interview guide, the scorecard version, and the debrief record. A new field that interviewers skip will not improve the evidence. Compare the next sample by role and criterion so the result is not driven by a change in the candidate pool.

What the result does not tell you

The audit measures the written evidence, not the quality of the hiring decision. A candidate can be a strong fit while an interviewer records a weak rationale, and a weak candidate can receive a well-written but positive score. The result also does not show whether the criteria predict performance in the role. A scorecard can be internally consistent and still measure the wrong things.

Use the audit with hiring outcomes, work sample results, and manager feedback. If a criterion never distinguishes candidates, it may not belong on the scorecard. If a criterion predicts later performance but is rarely documented, the interview process may not give interviewers a chance to observe it. The next review should test one criterion against actual work rather than adding another rating field. Record the result beside the scorecard version so the hiring team can see which criteria earned their place.

One short reporting note should accompany the audit. State the roles, the number of scorecards, the criteria, and the rating levels that were reviewed. Show the most common evidence gap and the interviewer group that will receive the first calibration session. Avoid ranking individual interviewers from a small sample. The report should give the hiring manager a specific next step and a date for the follow-up review. Keep the coding examples with the report so the next audit uses the same evidence standard.

Limitations

The method depends on scorecards that are complete enough to audit and on a definition of evidence that reviewers apply consistently. An interviewer may hold relevant evidence in memory that was never written down, which the audit will count as unsupported. Reviewers may disagree about whether a comment is specific enough. Small samples and different interview formats make the rate unstable.

An unsupported rating is not proof that the hiring decision was wrong. It means the written record does not show why the rating was given. Use the result to improve the scorecard and the interview process, not to judge an individual from one hiring round. Report the roles, the definitions, and any guide change so the next audit is comparable.

Sources

1. U.S. Small Business Administration, Manage your business
2. NIST Cybersecurity Framework 2.0
3. U.S. Bureau of Labor Statistics, Occupational Employment and Wage Statistics
4. Federal Trade Commission, Data Security
5. U.S. Department of Labor, Employer Responsibilities
6. NIST Privacy Framework
7. U.S. Department of Labor, Recordkeeping
8. NIST SP 800-53, Security and Privacy Controls
9. Internal Revenue Service, Recordkeeping
10. U.S. Small Business Administration, Manage your finances