Psychometrics · 13 min read

What Makes an Assessment Test Valid and Reliable?

When is an assessment valid and reliable? Validity benchmarks per test type, how to read reliability and measurement error, and a vendor checklist.

Door Ingmar van Maurik · Founder & CEO, Making Moves


Short answer

An assessment is valid when it measures what it claims to measure and demonstrably predicts job performance. It is reliable when it produces consistent scores across repeated administrations. For hiring decisions, look for reliability of at least 0.80 and demonstrated validity of 0.30 or higher for a comparable role.

This article is part of our complete guide to assessments in hiring.

Why it matters

An assessment is only valuable if it measures what it promises and produces consistent results. That sounds obvious, yet many organizations use assessments without knowing whether they predict job performance at all.

The result: decisions based on noise. You believe you are hiring on data, but you are using an instrument that predicts little better than chance. Worse, it gives you a false sense of certainty.

In this article you will learn what validity and reliability actually mean, how well common test types perform, how to read a test score correctly, and which questions to ask your vendor. We close with how to validate an assessment on your own data.

Validity: are you measuring what you want to measure?

Validity answers one question: does this test actually predict job performance? There are four forms you should know.

Predictive validity

This is the gold standard. You compare test scores with later performance on the job:

  • Do high scorers also score high in performance reviews after 6 and 12 months?
  • Which test components are related to success in the role?
  • Do scores predict retention: do high scorers stay longer?
  • Predictive validity is expressed as a correlation coefficient (r). Common guidelines in personnel selection:

    Correlation (r)Interpretation

    |---|---|

    Below 0.10Negligible 0.10 to 0.20Weak 0.20 to 0.30Useful 0.30 to 0.40Strong Above 0.40Very strong, rare in selection

    That may look modest, but in hiring r = 0.30 is already valuable: across hundreds of decisions it shifts your success rate considerably.

    Construct validity: convergent and divergent validity

    Does the test measure the right construct? A test of analytical ability should measure analytical ability, not reading skill or working memory. You check this in two ways:

  • Convergent validity: the test correlates strongly with other established tests of the same construct. A new cognitive ability test should correlate highly with an existing, validated one.
  • Divergent validity (also called discriminant validity): the test does not correlate strongly with tests of something else. A stress tolerance scale that correlates highly with social desirability mostly measures who likes to present themselves favorably.
  • Factor analysis then shows whether the items cluster as intended.

    Criterion validity

    What kind of success do you want to predict? Productivity, retention, customer satisfaction, growth rate and team contribution are different criteria, and different criteria require different predictors. A test that predicts productivity does not automatically predict turnover. Define what success means in the role first, then choose the test.

    Content validity

    Does the test cover what the job actually requires? You establish this through a job analysis: which tasks, situations and competencies determine success? An assessment for a security officer, for example, tests attentiveness, integrity and judgment during incidents, not abstract pattern recognition unrelated to the work.

    Reliability: is it consistent?

    Reliability answers the question: does the test produce similar results when repeated?

    Test-retest reliability

    Does the same person score similarly at two points in time? Guideline: above 0.80 is good, 0.70 to 0.80 is acceptable, and below 0.70 the test contains too much noise for individual selection decisions.

    Internal consistency (Cronbach's alpha)

    Do all items within a scale measure the same thing? Guideline: α above 0.80 is good, 0.70 to 0.80 acceptable, below 0.70 questionable. Note: for a situational judgment test (SJT), alpha is often lower because each scenario taps something different. Test-retest says more there.

    Inter-rater reliability

    For interviews, assessment centers and work samples: do different assessors reach the same judgment? If agreement is low, structured scoring rubrics with behavioral anchors and assessor training help.

    Why an unreliable test can never be valid

    Reliability caps validity. The correlation between a test and job performance can never exceed the square root of the test's reliability multiplied by the reliability of the performance rating. A test with reliability 0.60 and a rating with 0.50 can reach r = 0.55 at most, and in practice far less.

    The reverse does not hold: a reliable test is not automatically valid. It can measure the wrong thing very consistently. More examples in validity vs reliability explained.

    Standard error of measurement: when is a difference a real difference?

    Every score contains measurement error. The standard error of measurement shows how much: standard deviation × √(1 − reliability).

    Example: a cognitive ability test with a standard deviation of 15 and reliability of 0.90 has a measurement error of about 4.7 points. The 95% interval around a score is then more than 9 points up and down. Candidate A with 108 and candidate B with 103 are therefore not demonstrably different.

    Practical consequence: work with score bands instead of a hard cut-off at a single point, and never let small differences decide.

    How valid and reliable are common test types?

    The most recent large meta-analysis (Sackett, Zhang, Berry and Lievens, 2022) recalculated the predictive value of selection methods. Average corrected validity, with an indication of typical reliability:

    MethodAvg. validity (r)Typical reliability

    |---|---|---|

    Structured interview0.420.70 to 0.85 (raters) Job knowledge test0.400.80 to 0.90 Work sample0.330.70 to 0.85 (raters) Cognitive ability test0.310.85 to 0.95 Integrity test0.310.75 to 0.85 Situational judgment test0.260.50 to 0.70 (alpha) Personality: conscientiousness0.190.75 to 0.85 Unstructured interview0.19Low and variable

    Reliability figures are indicative and vary widely by specific test. Two lessons from these numbers:

  • Older figures overestimate. The well-known study by Schmidt and Hunter (1998) reported 0.51 for cognitive ability tests. According to the reanalysis, the statistical corrections at the time were too generous. If a vendor promises "r = 0.50 or higher" without their own research, ask critical questions.
  • Combining pays off. No single method predicts everything. A cognitive test plus a structured interview or an SJT predicts better than either alone, because they measure different things.
  • How reliable is a cognitive ability test?

    Cognitive ability tests are among the most reliable instruments (often 0.85 or higher) and predict learning ability and performance in complex roles well. Points of attention: they can show adverse impact on certain groups, and they say nothing about motivation or behavior. Always check group differences in your own candidate pool.

    How reliable is a competency test?

    "Competency test" is an umbrella term, ranging from self-report questionnaires to SJTs and assessment centers. Reliability depends entirely on the format. Self-report is sensitive to socially desirable answering; behavior-based formats (SJT, structured interview, work sample) predict better. So always ask: how is the competency measured, and where has that been studied? More in competency tests in hiring.

    What about personality tests?

    Personality questionnaires are reasonably reliable, but on their own they predict performance modestly; conscientiousness is the strongest factor. Their value lies in appropriate use: as a complement, focused on traits that are demonstrably relevant to the role, and never as a knock-out criterion by themselves.

    What about integrity and employee reliability tests?

    Integrity tests, sometimes sold as employee reliability tests, predict counterproductive behavior (such as theft, absenteeism and rule-breaking) and performance fairly well, with an average validity of 0.31. There are two variants: overt tests that ask directly about attitudes and behavior, and personality-based tests that measure traits such as conscientiousness and dependability. They are especially valuable in roles responsible for money, goods or safety. Always combine them with other information and be transparent with candidates about their purpose.

    Checklist: 8 questions for your assessment vendor

  • Validation research: is there research for comparable roles, and which correlations were found?
  • Reliability per scale: alpha and/or test-retest per component, not only for the total score.
  • Norm group: how large, how recent and how relevant (country, language, job level)?
  • Language version: has the version in your candidates' language been studied separately, or only translated?
  • Independent review: has the test been reviewed independently, for example under the EFPA test review model or by the BPS in the UK?
  • Adverse impact: how are group differences monitored (in the US often with the four-fifths rule), and what happens if a component disadvantages certain groups?
  • Updates: how often is the test updated and revalidated?
  • Data: who owns the results, and can you link them to your own performance data in a privacy-compliant way?
  • If a vendor cannot answer the first three questions concretely, you do not really know what you are measuring.

    Why generic tests often fall short

    Commercial assessments are usually validated on broad populations. That creates three problems:

  • The norm group does not fit. What predicts success at a bank differs from a tech startup or a security company. A norm group of thousands of random professionals says little about your role.
  • The model is static. Generic tests are updated every few years; your roles change continuously.
  • One size fits all. The same questionnaire for developers, account managers and customer service, while different competencies predict success in each role.
  • Also read why generic assessments don't work and how to design company-specific assessments.

    Validating it yourself: how to approach it

    With your own assessment system you can demonstrate what works in your organization.

    Step 1: start with a job analysis

    Document which tasks and behaviors determine success. That gives you content validity from day one, before you have any data.

    Step 2: build your own norm groups

    Compare candidates with your successful employees per role, not with the market. More on this in how to build your own norm group.

    Step 3: link scores to performance

    Correlate assessment scores with reviews after 6 and 12 months, retention and other criteria. You will see which components predict and which are noise, and you can adjust the weighting.

    Step 4: be honest about sample sizes

    Local validation requires data. With 60 hires, an observed correlation of 0.35 is still statistically uncertain: the 95% interval runs from roughly 0.11 to 0.55. Plan for at least 100 to 150 hires per job family for stable conclusions. Until then, combine your own data with existing research and a strong job analysis.

    Step 5: keep calibrating and monitor bias

    Repeat the analysis after every hiring cohort and check per subgroup whether the test predicts equally well and disadvantages no one unfairly. That way your system gets smarter over time, something a standard test by definition cannot do.

    The difference in practice

    AspectGeneric assessmentCompany-specific assessment

    |---|---|---|

    Norm groupBroad, cross-industryYour employees per role Evidence of validityResearch at other organizationsDemonstrable on your own data UpdatesEvery few yearsContinuous Bias monitoringGeneric, by the vendorOn your candidate population CostLicense per candidatePart of your own system Data ownershipVendorYou

    Frequently asked questions

    What is the difference between validity and reliability?

    Reliability is about consistency: do you get the same score when you repeat the test? Validity is about accuracy: does the test measure what it should, and does it predict job performance? A test must be reliable before it can be valid.

    When is an assessment reliable enough?

    For individual selection decisions, a reliability of 0.80 or higher is the usual standard. Between 0.70 and 0.80 is usable as supporting information, not as the sole criterion.

    What is good validity for an assessment?

    An average validity of 0.30 or higher is strong in personnel selection. Values above 0.40 are rare; be critical of higher claims without underlying research.

    What is divergent validity?

    Divergent (or discriminant) validity shows that a test does not correlate strongly with tests that measure something else. It demonstrates that the test measures its own, well-defined construct.

    How many candidates do you need to validate a test?

    For stable local validation, plan for at least 100 to 150 hires per job family. With smaller numbers, rely on a thorough job analysis and existing research on comparable roles.

    Can I simply use a translated test?

    Preferably not without research. Translation can shift the meaning of items. Check whether the language version has been studied separately and whether there are norms for your country and target group.

    Summary

  • Validity: does the test measure what it should, and does it predict job performance?
  • Reliability: does the test produce consistent scores? Aim for 0.80 or higher.
  • Read scores with their measurement error: small differences between candidates are often not real.
  • Combine methods: cognitive ability, behavior and structured interviewing complement each other.
  • Validate on your own data: your own norm groups and ongoing calibration make an assessment truly predictive.
  • Want to know how valid your current assessments are, or develop an assessment that demonstrably predicts who will succeed? See how we develop custom assessments, read how our AI hiring system builds validation into the process, or get in touch.


    Book intake call · View our AI Hiring System