How to Evaluate AI Consultants: 10 Questions That Separate Skill From Jargon
How to evaluate AI consultants before you hire one: 8 capabilities to verify, red flags, 10 interview questions and a practical test with a rubric.
Door Ingmar van Maurik · Founder & CEO, Making Moves
Short answer
You do not recognise a good AI consultant by a demo or by jargon, but by what they can demonstrably do: start from the business process, back up choices with measurements, name the risks and deliver a system you can run yourself. The best way to test that is with targeted questions about what goes wrong and a short practical case in which the consultant has to defend their design.
Why it is hard right now
Many people have recently started calling themselves AI consultants. On a CV or LinkedIn profile you can barely tell who has years of experience and who has a few months.
On top of that: a demo is easy, production is hard. A chatbot that answers five questions in a presentation takes an afternoon. A system that handles thousands of real questions, with customer data, outages at external services and a cost per conversation, is a different craft. Those skills do not show up in a demo. You have to ask for them.
What a good AI consultant can demonstrably do
|---|---|
Red flags
10 interview questions that separate understanding from jargon
Ask them in a fixed order and note what each answer contains. More on structuring interviews in competency tests in hiring.
1. "Which process would you tackle first, and how will you know after three months whether it paid off?" Strong answer: asks about volumes and the cost of errors, and mentions a baseline and a measurable target.
2. "Our order API returns HTTP 429. Our support agent keeps retrying and then tells the customer the order is confirmed. What went wrong and how do you fix it?" Strong answer: 429 means too many requests. You need a limited number of retries with backoff, a timeout, and above all: the agent may only confirm based on a real success response from the system. Otherwise it says honestly that it failed and hands over to a person.
3. "How do you prevent a CV containing hidden instructions from making an agent send candidate data elsewhere?" Strong answer: treat content from documents as data, not as instructions. The agent that reads CVs has no tool to email or upload externally. Outgoing actions go through an allowlist and human approval.
4. "How would you measure whether the new model version is better than the old one?" Strong answer: run the same evaluation set of real, labelled examples on both versions, including cost and latency, with criteria defined in advance.
5. "An employee asks the internal assistant about salaries and gets an answer from an HR document. How would you have prevented that?" Strong answer: filter permissions at retrieval time, based on who is asking, not afterwards in the prompt.
6. "How do you choose between a large and a small model for this task?" Strong answer: measure both on the same set, on quality, cost per task, latency and where the data goes.
7. "What does this system cost per month at 20,000 conversations?" Strong answer: calculates out loud with tokens per conversation, price per token and caching, and names the biggest uncertainty.
8. "Which actions may the agent take on its own, and which not?" Strong answer: reading and preparing is fine. Irreversible actions such as paying, deleting or sending externally require human approval.
9. "How would you notice that the system has got worse after a month?" Strong answer: tracing, monitoring of quality and cost, and regression tests on every change.
10. "What do we get at handover, and what should we be able to do ourselves?" Strong answer: code, prompts, evaluation sets and documentation in your environment, plus a handover that lets your team make changes. Also mentions GDPR and whether the EU AI Act applies.
Note: an answer that sounds good but contains no concrete detail is a reason to dig deeper. "Can you give an example?" quickly separates experience from jargon.
A practical test you can run yourself
Score with a rubric of five criteria, each 0 to 4:
|---|---|---|---|
An important caveat: a gap between what someone claims and what they demonstrate is a reason for follow-up questions, not proof of bluffing. Discuss the gap openly.
For measuring AI skills more broadly, read how to assess AI skills.
The Making Moves AI Consultant & Architect Assessment
The AI Consultant & Architect Assessment combines knowledge questions, decision scenarios, a practical case in which AI use is allowed and a structured verification interview in which the consultant defends their architecture. To be clear: the assessment is in a pilot phase. It is not yet a validated, normed test. So use the outcome as structured input for your decision, alongside references and previous projects.
Frequently asked questions
What should I look for when hiring an AI consultant?
Demonstrable experience with systems in production, an evaluation plan, attention to security and permissions, a monthly cost estimate and agreements on ownership of code and data.
Is a certificate enough evidence?
No. A certificate shows knowledge. Whether someone applies it to your situation, you test with scenario questions and a practical case.
May the consultant use AI during the test?
Yes, if that is also the case in real work. You are then not assessing whether someone knows everything by heart, but whether they make good choices and can defend them.
Does this approach also work for AI engineers and implementation partners?
Yes. Adapt the case to the role: more building for engineers, more process and governance for partners who lead an entire programme.
Summary
Want to evaluate AI consultants or AI engineers in a structured way? Take a look at the AI Skills Assessment or get in touch.