AI in Hiring · 8 min read

What Is RAG? Retrieval-Augmented Generation Explained for Business and HR

What is RAG (retrieval-augmented generation)? How it works step by step, when it beats fine-tuning, common failure modes and how to evaluate a RAG system.

Door Ingmar van Maurik · Founder & CEO, Making Moves


Short answer

RAG (retrieval-augmented generation) is a way to make a language model answer questions based on your own documents. For every question, the system first looks up the relevant passages, for example from your employee handbook, and passes them to the model together with the question. The model then writes an answer with citations. You get current, verifiable answers without retraining the model. Whether a RAG system works well depends mostly on the retrieval part: does it fetch the right passages, and only the passages the person asking is allowed to see?

Why RAG exists

A language model only knows what was in its training data. It does not know your leave policy, it has not seen last month's update to the travel policy and it cannot read your internal wiki. Ask it anyway and you often get a plausible but invented answer.

RAG solves this by having the model look things up instead of "remembering" them. The term comes from a 2020 research paper and is now the standard approach for internal assistants and knowledge bases.

How RAG works, step by step

1. Collect documents. Policies, handbooks, job profiles, FAQs. Include metadata: who owns it, when it was last updated, who may read it.

2. Chunking. Documents are split into smaller pieces, for instance per paragraph or section. Chunks that are too small lose context; chunks that are too large add noise. Good chunking follows the structure of the document.

3. Embeddings. Each chunk is converted into an embedding: a long list of numbers that captures the meaning of the text. Texts with similar meaning get similar numbers, even when they use different words.

4. Vector search. The user's question is also turned into an embedding. A vector database finds the chunks whose meaning is closest to the question. This is often combined with classic keyword search (hybrid search), because vector search can miss codes, names or exact terms.

5. Reranking. A second, more precise model scores the candidates again and moves the most relevant ones to the top. Only the best few go through.

6. Prompt with sources. The selected chunks are placed in the prompt with the question, plus an instruction such as: answer only from these sources and say so honestly if the answer is not there.

7. Answer with citations. The model answers and points to the chunks it used, so the user can check where it came from.

When to choose RAG, fine-tuning or a long context window

ApproachWhat it doesGood forLess suited to

|---|---|---|---|

RAGLooks up relevant chunks for every questionMany documents, knowledge that changes, citations and per-user permissionsTasks that mainly need a different writing style or fixed output format Fine-tuningFurther trains the model on examplesConsistent tone, fixed format, specific classification tasksFacts that change often; you cannot easily remove knowledge again and you get no sources Long context windowPuts all documents in a single promptA handful of documents, one-off analysisLarge knowledge bases; cost and latency per question go up, and models do not always use information buried in the middle of a long text equally well

In practice they are often combined. But for "let employees query our policies", RAG is almost always the starting point.

Common failure modes

  • Wrong chunks. The system retrieves a passage that looks similar to the question but is about something else, such as the leave rules from a different collective agreement. The answer sounds confident but is wrong.
  • Stale documents. The old and the new version of a policy both sit in the index. Without dates and version control, the system picks one at random.
  • No permission filtering. Anyone can pull passages from salary overviews or sickness records through the assistant. This is the biggest risk, and it has to be handled at retrieval time, based on who is asking. An instruction in the prompt like "do not share confidential information" is not security.
  • Hallucinated citations. The model cites a source that does not exist or says something else. Check automatically that cited chunks were actually retrieved and support the claim.
  • No "I don't know". If no good chunk was found, the system should say so rather than guess.
  • How to evaluate a RAG system

    A demo with five questions tells you little. Measure it like this:

    What you measureHow

    |---|---|

    Test setCollect real questions from employees, with the correct answer and the document it is in. Include hard questions and questions that have no answer. Retrieval hit rateHow often is the right chunk among the retrieved results? If this is low, a better model will not help much. FaithfulnessDoes the answer match the retrieved sources, without additions? Have people review a sample, optionally supported by an automated grader that you check yourself first. PermissionsTest with accounts from different roles that nobody ever sees chunks they have no access to. Cost and latencyCost per question and waiting time for the user, at the volume you expect.

    Rerun this set on every change to chunking, model or prompt. That is how you see whether a change is really an improvement.

    HR example: a policy assistant

    An organisation with a few thousand employees gets the same questions at HR every week: how much leave for moving house, how the home working allowance works, what applies if you fall ill on holiday. A RAG assistant answers these from the current handbook and collective agreement, with a link to the exact passage.

    What makes it good: only the latest version of each document is indexed, every document has an owner, managers see manager policies and employees do not, and when in doubt the assistant refers to an HR adviser. Personal records do not belong in it.

    For recruiters the same principle works as a knowledge base: make job profiles, interview guides and selection criteria searchable. How such a system fits into hiring more broadly is described on our AI hiring system page.

    What to ask a builder or a candidate

    To find out whether a vendor or candidate really understands RAG, ask these questions:

    1. "How do you choose the chunk size, and how do you test that choice?" Strong answer: it depends on the documents, and is measured with retrieval hit rate on a test set.

    2. "Why combine vector search with keyword search?" Strong answer: vector search can miss exact terms such as codes or names.

    3. "How do you make sure an employee never sees salary documents?" Strong answer: permission filtering at retrieval time, tied to the user's identity, not via the prompt.

    4. "What happens when a policy is updated?" Strong answer: automatic reindexing, removal of old versions, date as metadata.

    5. "How do you know the system got better after a change?" Strong answer: a fixed test set with retrieval and faithfulness measurements, plus cost and latency.

    Vague answers like "the model is very smart" are a reason to dig deeper. Read more in how to evaluate AI consultants and how to assess AI skills.

    Testing AI knowledge in a structured way

    The Making Moves AI Skills Assessment tests, among other things, whether someone can design and evaluate RAG systems: knowledge questions, decision scenarios on retrieval and permissions, and a practical case in which AI use is allowed. The assessment is in a pilot phase and is not yet a normed test. Use it as structured input alongside interviews and references.

    Frequently asked questions

    What does RAG stand for?

    RAG stands for retrieval-augmented generation: generating text (generation) supported by information that has been looked up (retrieval). The model searches your documents first and answers afterwards.

    Does RAG prevent hallucinations?

    It reduces them but does not eliminate them. If the wrong chunk is retrieved or the model summarises the source incorrectly, you still get errors. That is why citations and evaluation are needed.

    Is RAG safe for confidential HR documents?

    Only if permissions are filtered at retrieval time and you know where the data is processed. Mind the GDPR: do not index personnel files without a clear legal basis and access rules.

    Do we need to train our own model for this?

    No. RAG works with existing models. Your investment goes mainly into good documents, retrieval quality, permissions and evaluation.

    Summary

  • RAG lets a language model answer from your own, current documents, with citations.
  • Quality depends mostly on chunking, search and reranking.
  • For changing knowledge, RAG usually beats fine-tuning or one long prompt.
  • The biggest risks: wrong or stale chunks, missing permission filters and hallucinated citations.
  • Evaluate with a test set of real questions, retrieval hit rate, faithfulness, cost and latency.
  • Want to check whether candidates or vendors really master RAG? Take a look at the AI Skills Assessment or get in touch.


    Book intake call · View our AI Hiring System