AI in Hiring · 9 min read

How to Assess AI Skills: Measuring What People Can Really Do With AI

How to assess AI skills in candidates and employees: why a knowledge quiz is not enough, four methods with examples, a scoring model and the legal points.

Door Ingmar van Maurik · Founder & CEO, Making Moves


Short answer

You cannot assess AI skills with a knowledge quiz alone. A good AI skills assessment combines four methods: knowledge questions, situational judgement scenarios, a practical task in which AI use is allowed, and a verification interview in which the candidate explains their choices. Give the practical part the most weight, set a safety gate for dangerous actions and always let a person make the decision.

What do we mean by AI skills?

What someone needs to do with AI varies widely by role. We distinguish three levels:

LevelTypical roleWhat someone needs to be able to do

|---|---|---|

AI userEmployee, recruiter, marketerChoose the right tool, prompt well, check output, keep sensitive data out Builder (automation)Developer, operations, data analystBuild workflows and agents, set up RAG, connect APIs, handle errors, test Consultant or architectCTO, solution architect, AI leadChoose use cases, weigh cost and risk, set up evaluation and governance, design architecture

Below these levels sit concrete domains: model choice, prompting, output verification, RAG (letting AI work with your own documents), AI agents, integrations, security, evaluation, cost, governance and business architecture. Not every role needs all of them.

Why self-reports and CV keywords fall short

"ChatGPT", "prompt engineering" and "AI agents" now appear on many CVs. That says something about interest and experience, but little about level. There is often a gap between what someone believes they can do (claimed skill) and what they show in a task (demonstrated skill).

That gap does not mean anyone is bluffing. There are ordinary explanations:

  • Self-assessment is hard. In a new field you often do not yet know what you do not know.
  • Test conditions differ. At work you have time, colleagues and your own tools. In a test you often do not.
  • Every measurement contains error. Even a good test gives an estimate, not an exact number.
  • The lesson: do not rely on a single source. Measure behaviour and ask follow-up questions.

    Four methods to assess AI skills

    1. Knowledge questions

    Knowledge questions check whether someone knows the basic concepts and principles. They are quick and objective to score, but do not show whether someone applies that knowledge.

    Example: "A team wants an AI assistant to answer questions based on 2,000 internal policy documents. Which approach fits best? A) Paste all documents into every prompt. B) A RAG setup that retrieves relevant passages and shows the source. C) Ask the model to memorise the policy. D) Choose a larger model."

    2. Situational judgement scenarios

    A situational judgement test presents a realistic work situation and asks which response is best. This measures judgement: does the person know what to do when AI gets something wrong?

    Example: "You ask an AI assistant for figures for a board presentation. The answer contains a convincing percentage about customer preferences, with no source. What is the best thing to do?" The strongest response: do not use the figure until you have found a reliable primary source, and otherwise leave it out. Weak responses are copying the figure because it sounds plausible, or asking the AI "is this correct?" and treating the answer as confirmation.

    3. Practical task where AI is allowed

    You do not ban AI, you expect it to be used, because it is allowed at work too. You score two things: the result (does it work, is it correct, is it safe) and the process (which choices does the person make, is the output checked, what happens when something fails).

    Example: The candidate receives the logs of an AI support agent. The agent calls an order API that keeps returning HTTP 429 (too many requests). The agent retries immediately, dozens of times, and then tells the customer: "Your order is confirmed." Task: what is going wrong and how would you fix it?

    A strong answer covers:

  • The cause: 429 means a rate limit has been reached. Retrying immediately makes it worse.
  • Retries with backoff: waiting with increasing intervals and some randomness, a maximum number of attempts, respecting the Retry-After header and making sure a repeated request cannot create a duplicate order.
  • A safe fallback: the agent only confirms when the API actually returns success. Otherwise it tells the customer honestly that the order has not been processed yet, or hands over to a human.
  • Monitoring: alerts on high numbers of 429 errors and on confirmations without a successful API response.
  • Regression tests: a test that simulates a 429 and checks that the agent never confirms in that case.
  • 4. Structured verification interview

    Finally, the candidate defends their choices in a short interview with fixed questions and a scoring rubric. This shows whether the person understands what was built, even if AI helped.

    Example: "Your agent can pay invoices. It is about to transfer EUR 4,800 to a bank account that was changed by email yesterday. What would you have built in, and why?" Look for human approval above a threshold amount, verification of changed payment details through a second channel, limited permissions for the agent, and logging.

    A simple scoring model

    ComponentWeightWhat it shows

    |---|---|---|

    Knowledge questions25%Concepts and principles Scenarios25%Judgement in realistic situations Practical (task plus verification interview)50%What someone can really do and understands

    Add a safety gate. If someone does something dangerous in the task, such as sending personal data to an unapproved external tool or letting an agent make payments without checks, their level is capped regardless of the total score. A high knowledge score does not compensate for risky behaviour in practice.

    Work with levels or score bands rather than exact points. Small differences between candidates fall within measurement error. Read more in what makes an assessment valid and reliable.

    Legal points to watch

  • GDPR Article 22. A rejection must not be based solely on automated processing where it significantly affects the candidate. Let a person review the outcome and make the decision.
  • EU AI Act. AI systems used to evaluate or select candidates are high-risk under Annex III. After the Digital Omnibus (Regulation (EU) 2026/1744), those obligations apply from 2 December 2027. If you use AI to score answers, this may apply to your setup too. Which duties apply exactly depends on whether you are the provider or the deployer of the system.
  • Validation and fairness. Show that the test measures what the job requires and monitor outcomes by group. A task that mainly rewards fast typing or English jargon measures something other than AI skill. Explain in advance what you measure and when AI use is allowed.
  • How to start

    1. Run a job analysis per role. What does a recruiter need to do with AI, and what does a developer need?

    2. Pick four to six domains. For an AI user, for example: prompting, verification, security and cost.

    3. Run a pilot. Test with employees whose level you know. Check whether the tasks are clear and whether assessors agree with each other.

    4. Compare with performance. After a few months, check whether scores relate to how people actually perform at work.

    For more on measuring behaviour, read competency tests in hiring.

    About the Making Moves AI Skills Assessment

    We are developing the Making Moves AI Skills Assessment along these lines: knowledge, scenarios and practice across the domains above. The assessment is in a pilot phase. It is not yet a validated, normed selection test. Use the outcome as input for a conversation or development plan, not as the sole basis for a selection decision.

    Frequently asked questions

    Is an AI knowledge test enough to assess AI skills?

    No. Knowledge is a precondition, but it says little about application. Combine knowledge questions with scenarios, a practical task and a verification interview.

    Can a candidate use AI during the assessment?

    In the practical part, yes. You are measuring how someone uses, checks and corrects AI. For knowledge questions it is better to exclude AI, so you measure the person's own knowledge. Explain clearly in advance what is allowed.

    Can you also test existing employees?

    Yes. For employees the goal is usually development: seeing where someone stands and which training fits. Be open about the purpose and about what happens with the results. For an organisation-wide assessment, involve your works council or employee representatives.

    Does an AI skills assessment fall under the AI Act?

    A test about AI is not automatically an AI system. If you do use AI to evaluate or select candidates, for example for automated scoring, that is high-risk under Annex III, with obligations from 2 December 2027.

    Summary

  • A knowledge quiz is not enough: measure what someone does with AI, not just what they know.
  • A gap between claimed and demonstrated skill is normal and does not mean anyone is bluffing.
  • Combine knowledge questions, scenarios, a practical task with AI and a verification interview.
  • Give practice the most weight and use a safety gate for dangerous actions.
  • Keep GDPR Article 22 and the AI Act in mind: a person decides.
  • Curious where you or your team stand? Take the free AI Skills Check or get in touch if you want to assess AI skills in your organisation.


    Book intake call · View our AI Hiring System