RAG (retrieval-augmented generation) is a method in which a language model first finds and reads the relevant passages in your company's documents, then bases its answer on those sources. You do not need to retrain the model; as your documents change, the answers change with them, and every answer can point to its source.
This article is written for the people who make the decision, not for engineering teams. We explain in plain terms how the method works, what drives its quality, how permissions are protected, how success is measured and when another approach fits better. If you want to place it within your company's wider AI plan, see our AI consulting page.
Why doesn't a language model know your company?
Large language models (LLMs) are trained on broad text from the internet and licensed sources up to a certain date. They do not know your contracts, internal procedures, product catalog or a rule that changed last week. When they face a question they cannot answer, they sometimes produce a plausible but wrong answer instead of saying "I don't know." This is called hallucination.
There are three basic ways to give a model company knowledge:
- Paste the document into the prompt: this works for a few pages, but with hundreds of documents it hits the model's limits and raises the cost of every question.
- Fine-tuning: further training the model on your own examples. It changes behavior and style, but it is a poor way to carry information that changes often.
- RAG: keeping the knowledge next to the model rather than inside it. For each question, the model reads only the relevant passages.
For most uses like these, RAG is the right starting point:
- Internal knowledge assistants where employees ask questions about procedures, policies and technical documents.
- Draft replies with sources for customer support teams.
- Search across contracts, tenders and proposal files.
- Customer-facing assistants that work from product and service documentation.
How does RAG work, step by step?
A RAG system has two stages: preparing the documents and answering the questions.
- Collection: sources such as PDFs, internal wiki pages, support tickets, emails or database records are identified and pulled in regularly.
- Chunking: long documents are split into smaller passages, each covering a single topic.
- Embedding: each chunk is converted into a list of numbers, a vector, that represents its meaning. Texts with similar meanings end up with vectors close to each other.
- Storage: the vectors are saved in a vector database along with details such as the document name, date and access permissions.
- Search: when a user asks a question, the question is also turned into a vector and the closest chunks are found. This is called vector search, or semantic search.
- Answer: the retrieved chunks are passed to the language model together with the question. The model writes the answer from those chunks and shows which documents it used.
This design has a big advantage: when a document is updated, only that document's chunks are processed again. The knowledge stays fresh without touching the model. Running cost is driven mainly by document volume, question volume and how much text is sent to the model with each question.
What determines answer quality?
In RAG projects, problems usually come from search rather than from the model. If the model never sees the right passage, no amount of model power will produce the right answer. The main decisions that drive quality:
- Chunk size: chunks that are too small lose context; chunks that are too large drag in irrelevant text. The right size depends on the document type and is found by testing.
- Keeping structure: storing headings, section names and dates with each chunk makes both search and citation easier.
- Hybrid search: semantic search alone can be weak for product codes, contract numbers or proper names. Combining it with keyword search closes that gap.
- Reranking: a second model reorders the candidate chunks by how well they fit the question.
- Document quality: scanned PDFs need text recognition (OCR), tables need separate handling and old or conflicting versions need to go.
- Freshness: when a document changes or is deleted, the index must be updated promptly; otherwise the assistant serves up a rule that no longer applies.
That last point is often underestimated. When three versions of the same procedure sit side by side, even the best system has to guess which one is current.
How are permissions and data security protected?
The first rule for an assistant that works with company data: a user must not be able to see, through the assistant, a document they are not allowed to open. That is why permission checks happen at search time. Each chunk carries the access information of its source document, and the search runs only across chunks the user is allowed to see. When permissions change in the source system, for example when an employee moves to another department, the assistant must reflect that change right away.
Telling the model "do not share this" is not a security control. Information that should not reach the model must never be put in front of it. Beyond that, settle these points early:
- Personal data: if documents contain personal data, the purpose of processing, whether it will be masked and your obligations under KVKK and GDPR.
- Where data is processed: a cloud model provider, servers in a specific region or a model running on your own infrastructure?
- Logging and retention: how long questions and answers are stored, and who can access them.
- Prompt injection: attempts to steer the model with instructions hidden inside documents. Documents from outside sources need extra scrutiny for this.
We have gathered the checks to run before working with personal data in Data protection and AI.
How do you measure whether a RAG system works?
"We tried it and it looks fine" is not a measurement. Before the system goes live, build an evaluation set:
- Collect real questions. Gather the questions employees or customers ask today by email, phone or support ticket.
- Write the correct answer and source. For each question, someone who knows the subject records the expected answer and the document it comes from.
- Measure search separately. Did the system find the right chunk among the top results?
- Measure the answer separately. Is it faithful to the sources, complete and free of anything the sources do not say?
- Test the "I don't know" behavior. For questions the documents cannot answer, the system should say so plainly instead of making something up.
In production, add a simple feedback button under each answer and log the questions that went unanswered. Every time the model, the chunking or the search settings change, run the same evaluation set again.
When is fine-tuning a better choice than RAG?
RAG and fine-tuning are not rivals; they solve different problems. In short:
- When information changes often, sources must be cited and permissions differ by user, RAG is the right choice.
- When the model must behave consistently in a set format, tone or classification logic, fine-tuning fits better. Sorting incoming requests into fixed categories or writing reports in the same structure every time are typical examples.
- For a narrow, repetitive task, a small fine-tuned model can be faster and more economical than a large one.
- The two can work together: fine-tuning shapes the behavior, RAG supplies the knowledge.
Sometimes you need neither. If the question depends on structured data, such as "how many orders came from each region last month?", the right tool is a report or a database query. RAG also serves as the knowledge layer for AI agents, a topic we cover in Autonomous AI agents.
An AI assistant is only as trustworthy as the documents it stands on.
Frequently asked questions
Do we have to send our data to an AI provider to use RAG?
It depends on the architecture. With a cloud model, only the chunks relevant to the question are sent to the model, and data processing terms and server region should be settled in the contract. For sensitive data, an open-source model running on your own servers is also an option.
Does RAG eliminate hallucinations completely?
No, but it can reduce them significantly. Requiring answers to cite sources, making the system say "I don't know" when nothing is found and evaluating regularly keep the risk manageable.
Which documents are a good fit for RAG?
Text-heavy, current documents with a clear owner work best: procedures, policies, product documentation, contracts and support records. Old, conflicting or duplicate documents lower quality, so start with a cleanup.
Do we need a separate vector database?
Not always. Some relational databases and search engines support vector search. If the document set is limited, starting with your existing infrastructure can be simpler than adding a new product.
Where to start?
Pick one area that gets a lot of questions, such as HR policies, technical support or sales documentation. Collect the real questions asked there recently, mark which documents hold the answers and check whether those documents are current. That list defines both the scope of your pilot and your evaluation set.
If you want to plan an AI assistant that works with your company data, get in touch.