AI in Hiring · 9 min read

How to Evaluate AI Consultants: 10 Questions That Separate Skill From Jargon

How to evaluate AI consultants before you hire one: 8 capabilities to verify, red flags, 10 interview questions and a practical test with a rubric.

Door Ingmar van Maurik · Founder & CEO, Making Moves


Short answer

You do not recognise a good AI consultant by a demo or by jargon, but by what they can demonstrably do: start from the business process, back up choices with measurements, name the risks and deliver a system you can run yourself. The best way to test that is with targeted questions about what goes wrong and a short practical case in which the consultant has to defend their design.

Why it is hard right now

Many people have recently started calling themselves AI consultants. On a CV or LinkedIn profile you can barely tell who has years of experience and who has a few months.

On top of that: a demo is easy, production is hard. A chatbot that answers five questions in a presentation takes an afternoon. A system that handles thousands of real questions, with customer data, outages at external services and a cost per conversation, is a different craft. Those skills do not show up in a demo. You have to ask for them.

What a good AI consultant can demonstrably do

CapabilityWhat to ask for as evidence

|---|---|

Starts from the business process and the return, not the modelA previous project with a measured baseline and a target in time, money or quality Chooses models on quality, cost, latency and privacyAn evaluation set with real examples and the comparison that led to the choice Designs retrieval (RAG) with permission filteringAn explanation of how an employee only sees documents they have access to Designs agents safelyLeast privilege per tool, human approval for irreversible actions, timeouts and a fallback Understands prompt injection and data protectionConcrete measures against hidden instructions in documents, emails or web pages Builds evaluations, tracing and regression testsAn example of a test set that reruns on every change, and logs that let you trace an error Estimates running costsA calculation of tokens per task, monthly volumes and the effect of caching Plans governance and handoverAttention to GDPR, the EU AI Act, documentation and a plan that makes you the owner of code, data and prompts

Red flags

  • Only a chatbot demo. No example of something that has been in use for months.
  • No evaluation plan. "We will just test it" without a test set, criteria or a moment of measurement.
  • Promising "100% accurate". Language models make mistakes. A good consultant tells you how to catch them.
  • No answer on failures. Cannot explain what happens when an API does not respond or hits a rate limit.
  • Lock-in without data ownership. Everything runs on the consultant's own platform, with no export of data, prompts and configuration.
  • No idea of monthly costs. No estimate of what the system costs per month at your volume.
  • 10 interview questions that separate understanding from jargon

    Ask them in a fixed order and note what each answer contains. More on structuring interviews in competency tests in hiring.

    1. "Which process would you tackle first, and how will you know after three months whether it paid off?" Strong answer: asks about volumes and the cost of errors, and mentions a baseline and a measurable target.

    2. "Our order API returns HTTP 429. Our support agent keeps retrying and then tells the customer the order is confirmed. What went wrong and how do you fix it?" Strong answer: 429 means too many requests. You need a limited number of retries with backoff, a timeout, and above all: the agent may only confirm based on a real success response from the system. Otherwise it says honestly that it failed and hands over to a person.

    3. "How do you prevent a CV containing hidden instructions from making an agent send candidate data elsewhere?" Strong answer: treat content from documents as data, not as instructions. The agent that reads CVs has no tool to email or upload externally. Outgoing actions go through an allowlist and human approval.

    4. "How would you measure whether the new model version is better than the old one?" Strong answer: run the same evaluation set of real, labelled examples on both versions, including cost and latency, with criteria defined in advance.

    5. "An employee asks the internal assistant about salaries and gets an answer from an HR document. How would you have prevented that?" Strong answer: filter permissions at retrieval time, based on who is asking, not afterwards in the prompt.

    6. "How do you choose between a large and a small model for this task?" Strong answer: measure both on the same set, on quality, cost per task, latency and where the data goes.

    7. "What does this system cost per month at 20,000 conversations?" Strong answer: calculates out loud with tokens per conversation, price per token and caching, and names the biggest uncertainty.

    8. "Which actions may the agent take on its own, and which not?" Strong answer: reading and preparing is fine. Irreversible actions such as paying, deleting or sending externally require human approval.

    9. "How would you notice that the system has got worse after a month?" Strong answer: tracing, monitoring of quality and cost, and regression tests on every change.

    10. "What do we get at handover, and what should we be able to do ourselves?" Strong answer: code, prompts, evaluation sets and documentation in your environment, plus a handover that lets your team make changes. Also mentions GDPR and whether the EU AI Act applies.

    Note: an answer that sounds good but contains no concrete detail is a reason to dig deeper. "Can you give an example?" quickly separates experience from jargon.

    A practical test you can run yourself

  • Assignment (60 to 90 minutes). Give a realistic situation from your own organisation, for example a support process that needs partial automation. Ask for an architecture sketch, model and retrieval choices, an evaluation plan, risks and a cost estimate. AI use is allowed, just as in real work.
  • Defence (20 minutes). The consultant walks you through the design. You ask follow-up questions: what happens on a timeout, why this model, who may do what, how do you measure success?
  • Score with a rubric of five criteria, each 0 to 4:

    Criterion024

    |---|---|---|---|

    Problem and returnStarts from the technologyNames a goal, but not measurableMeasurable goal and baseline Architecture and choicesNo justificationJustified on one factorWeighs quality, cost, latency and privacy Security and permissionsNot mentionedMentioned in general termsConcrete measures per tool and data source Evaluation and operationsNo planMentions testing, no test setEvaluation set, tracing and regression tests DefenceCannot explain own designExplains, but wavers under follow-up questionsExplains, acknowledges weaknesses and adjusts

    An important caveat: a gap between what someone claims and what they demonstrate is a reason for follow-up questions, not proof of bluffing. Discuss the gap openly.

    For measuring AI skills more broadly, read how to assess AI skills.

    The Making Moves AI Consultant & Architect Assessment

    The AI Consultant & Architect Assessment combines knowledge questions, decision scenarios, a practical case in which AI use is allowed and a structured verification interview in which the consultant defends their architecture. To be clear: the assessment is in a pilot phase. It is not yet a validated, normed test. So use the outcome as structured input for your decision, alongside references and previous projects.

    Frequently asked questions

    What should I look for when hiring an AI consultant?

    Demonstrable experience with systems in production, an evaluation plan, attention to security and permissions, a monthly cost estimate and agreements on ownership of code and data.

    Is a certificate enough evidence?

    No. A certificate shows knowledge. Whether someone applies it to your situation, you test with scenario questions and a practical case.

    May the consultant use AI during the test?

    Yes, if that is also the case in real work. You are then not assessing whether someone knows everything by heart, but whether they make good choices and can defend them.

    Does this approach also work for AI engineers and implementation partners?

    Yes. Adapt the case to the role: more building for engineers, more process and governance for partners who lead an entire programme.

    Summary

  • Evaluate an AI consultant on what is demonstrable, not on demos or jargon.
  • Ask about evaluation, permissions, failures, costs and handover.
  • Use fixed questions and note what each answer contains.
  • Run a practical case with a defence and score it with a rubric.
  • Treat gaps between claim and performance as a reason for follow-up questions, not as proof.
  • Want to evaluate AI consultants or AI engineers in a structured way? Take a look at the AI Skills Assessment or get in touch.


    Book intake call · View our AI Hiring System