Psychometrics · 7 min read

Validity vs Reliability: The Difference Explained With Examples

What is the difference between validity and reliability, can a test be valid but not reliable, and what makes a test reliable? Explained with examples from hiring.

Door Ingmar van Maurik · Founder & CEO, Making Moves

Short answer

Reliability is about consistency: does a test give roughly the same result when repeated and across different assessors? Validity is about accuracy: does the test measure what it is supposed to measure, and in hiring, does it predict job performance? A test can be reliable without being valid, but not the other way round: without reliability there is no validity. Reliability is the precondition, validity is the goal.

Want to know how to evaluate an assessment on both, with benchmarks per test type and a checklist? Read what makes an assessment test valid and reliable.

The difference in one example

Picture a bathroom scale.

  • You step on it three times in a row and see 70, 76 and 64 kilos. The scale is not reliable. You have no idea which reading is right.
  • You step on another scale three times and see exactly 75 kilos each time, while you actually weigh 70. That scale is reliable but not valid: consistent, but consistently wrong.
  • A third scale shows 70 kilos every time. It is reliable and valid.
  • And if you want to know someone's height, even the best scale is not valid. It measures the wrong thing very precisely.

    • That last point matters most in hiring. A test can measure extremely well, but measure something that does not matter for the role.

    The four combinations in hiring

    Not validValid

    |---|---|---|

    Not reliableAn unstructured conversation where every interviewer asks and scores differentlyDoes not exist: without consistent measurement you cannot predict anything ReliableA typing speed test for an account manager: measured consistently, but barely predicts the roleA structured interview with fixed questions, focused on the role and scored with a rubric

    Can a test be valid but not reliable?

    Strictly speaking, no. Reliability caps validity: the correlation between a test and job performance can never exceed the square root of the test's reliability multiplied by the reliability of the performance rating. A test that shows something different every time cannot predict anything consistently.

    Yet in practice it sometimes looks that way. A situational judgement test often has a low Cronbach's alpha (0.50 to 0.70) and still predicts reasonably well. That is because alpha only measures whether the items measure the same thing. An SJT deliberately measures different things, so alpha underestimates its reliability. Test-retest reliability, whether the same person gets the same score later, is a better measure in that case. So check which type of reliability a vendor quotes.

    What makes a test reliable?

  • Enough items or tasks. One question can go well or badly by chance. The more questions about the same thing, the smaller the role of chance.
  • Standardised administration. The same instructions, the same time and the same conditions for every candidate.
  • Objective or fixed scoring. For interviews and exercises: fixed questions, behavioural anchors, trained assessors and preferably two assessors.
  • Clear questions. Ambiguous questions are understood differently each time.

    • What makes a test valid?

  • A good job analysis. You first need to know which behaviour and skills make the difference in the role.
  • A fitting format. Persuasiveness is better measured in a role play than in a questionnaire. Read more in [competency tests in hiring](/en/articles/competency-test-hiring).
  • Evidence that scores relate to performance. Preferably research in comparable roles, and ideally on [your own data](/en/articles/build-own-norm-group).
  • A fitting norm group. A score only means something when compared with people who resemble your candidates.

    • Three questions to ask right away

    1. How reliable is the test, and which measure is that? For selection decisions you usually want 0.80 or higher. Ask whether it is alpha or test-retest.

    2. What does the test predict, and in which roles was that studied? "Scientifically validated" without figures and a target group says little.

    3. Which norm group are scores compared with? Does it fit your role and candidates?

    Frequently asked questions

    What is the difference between validity and reliability?

    Reliability is consistency: do you get the same result when you repeat it? Validity is accuracy: does the test measure what it should and does it predict job performance? A test must be reliable to be valid.

    Can a test be reliable but not valid?

    Yes. A test can very consistently measure something that does not matter for the role. That is why high reliability alone is never enough.

    Can a test be valid but not reliable?

    No. Without consistent measurement a test cannot predict anything consistently. If it seems to, a measure such as Cronbach's alpha has usually been used that underestimates that test's reliability.

    When is a test reliable enough?

    For decisions about individual candidates, 0.80 or higher is the usual standard. Between 0.70 and 0.80 is useful as additional information, not as the only criterion.

    Summary

  • Reliability is consistency, validity is accuracy.
  • A test can be reliable but not valid, but not valid without being reliable.
  • Check which reliability measure is quoted: alpha underestimates tests that deliberately measure different things.
  • Validity starts with a job analysis and is proven with performance data.

    • Want an assessment that demonstrably predicts performance in your roles? See how we develop custom assessments or get in touch.

    Book intake call · View our AI Hiring System