How to Assess AI Skills: Measuring What People Can Really Do With AI
How to assess AI skills in candidates and employees: why a knowledge quiz is not enough, four methods with examples, a scoring model and the legal points.
Door Ingmar van Maurik · Founder & CEO, Making Moves
Short answer
You cannot assess AI skills with a knowledge quiz alone. A good AI skills assessment combines four methods: knowledge questions, situational judgement scenarios, a practical task in which AI use is allowed, and a verification interview in which the candidate explains their choices. Give the practical part the most weight, set a safety gate for dangerous actions and always let a person make the decision.
What do we mean by AI skills?
What someone needs to do with AI varies widely by role. We distinguish three levels:
|---|---|---|
Below these levels sit concrete domains: model choice, prompting, output verification, RAG (letting AI work with your own documents), AI agents, integrations, security, evaluation, cost, governance and business architecture. Not every role needs all of them.
Why self-reports and CV keywords fall short
"ChatGPT", "prompt engineering" and "AI agents" now appear on many CVs. That says something about interest and experience, but little about level. There is often a gap between what someone believes they can do (claimed skill) and what they show in a task (demonstrated skill).
That gap does not mean anyone is bluffing. There are ordinary explanations:
The lesson: do not rely on a single source. Measure behaviour and ask follow-up questions.
Four methods to assess AI skills
1. Knowledge questions
Knowledge questions check whether someone knows the basic concepts and principles. They are quick and objective to score, but do not show whether someone applies that knowledge.
Example: "A team wants an AI assistant to answer questions based on 2,000 internal policy documents. Which approach fits best? A) Paste all documents into every prompt. B) A RAG setup that retrieves relevant passages and shows the source. C) Ask the model to memorise the policy. D) Choose a larger model."
2. Situational judgement scenarios
A situational judgement test presents a realistic work situation and asks which response is best. This measures judgement: does the person know what to do when AI gets something wrong?
Example: "You ask an AI assistant for figures for a board presentation. The answer contains a convincing percentage about customer preferences, with no source. What is the best thing to do?" The strongest response: do not use the figure until you have found a reliable primary source, and otherwise leave it out. Weak responses are copying the figure because it sounds plausible, or asking the AI "is this correct?" and treating the answer as confirmation.
3. Practical task where AI is allowed
You do not ban AI, you expect it to be used, because it is allowed at work too. You score two things: the result (does it work, is it correct, is it safe) and the process (which choices does the person make, is the output checked, what happens when something fails).
Example: The candidate receives the logs of an AI support agent. The agent calls an order API that keeps returning HTTP 429 (too many requests). The agent retries immediately, dozens of times, and then tells the customer: "Your order is confirmed." Task: what is going wrong and how would you fix it?
A strong answer covers:
4. Structured verification interview
Finally, the candidate defends their choices in a short interview with fixed questions and a scoring rubric. This shows whether the person understands what was built, even if AI helped.
Example: "Your agent can pay invoices. It is about to transfer EUR 4,800 to a bank account that was changed by email yesterday. What would you have built in, and why?" Look for human approval above a threshold amount, verification of changed payment details through a second channel, limited permissions for the agent, and logging.
A simple scoring model
|---|---|---|
Add a safety gate. If someone does something dangerous in the task, such as sending personal data to an unapproved external tool or letting an agent make payments without checks, their level is capped regardless of the total score. A high knowledge score does not compensate for risky behaviour in practice.
Work with levels or score bands rather than exact points. Small differences between candidates fall within measurement error. Read more in what makes an assessment valid and reliable.
Legal points to watch
How to start
1. Run a job analysis per role. What does a recruiter need to do with AI, and what does a developer need?
2. Pick four to six domains. For an AI user, for example: prompting, verification, security and cost.
3. Run a pilot. Test with employees whose level you know. Check whether the tasks are clear and whether assessors agree with each other.
4. Compare with performance. After a few months, check whether scores relate to how people actually perform at work.
For more on measuring behaviour, read competency tests in hiring.
About the Making Moves AI Skills Assessment
We are developing the Making Moves AI Skills Assessment along these lines: knowledge, scenarios and practice across the domains above. The assessment is in a pilot phase. It is not yet a validated, normed selection test. Use the outcome as input for a conversation or development plan, not as the sole basis for a selection decision.
Frequently asked questions
Is an AI knowledge test enough to assess AI skills?
No. Knowledge is a precondition, but it says little about application. Combine knowledge questions with scenarios, a practical task and a verification interview.
Can a candidate use AI during the assessment?
In the practical part, yes. You are measuring how someone uses, checks and corrects AI. For knowledge questions it is better to exclude AI, so you measure the person's own knowledge. Explain clearly in advance what is allowed.
Can you also test existing employees?
Yes. For employees the goal is usually development: seeing where someone stands and which training fits. Be open about the purpose and about what happens with the results. For an organisation-wide assessment, involve your works council or employee representatives.
Does an AI skills assessment fall under the AI Act?
A test about AI is not automatically an AI system. If you do use AI to evaluate or select candidates, for example for automated scoring, that is high-risk under Annex III, with obligations from 2 December 2027.
Summary
Curious where you or your team stand? Take the free AI Skills Check or get in touch if you want to assess AI skills in your organisation.