Enterprise RAG Architecture: What It Takes to Answer From Your Own Data

Retrieval augmented generation, or RAG, is the usual pattern for an AI assistant that answers from an organisation’s own documents. It retrieves relevant content first, then asks a language model to answer using that content. A demo can be built quickly. Most of the real work is in the parts a demo skips: permissions, stale content, poor retrieval, and knowing whether the answers are right.
The parts of the system
| Layer | What it does | How it commonly fails |
|---|---|---|
| Ingestion | Brings in documents and data, and keeps them current | Outdated or duplicate content is treated as true |
| Chunking | Splits content into retrievable pieces | Pieces lose the context needed to make sense of them |
| Indexing | Stores content for keyword and vector search | Exact terms, such as policy numbers, are missed by vector search alone |
| Access control | Limits retrieval to what the user may see | The assistant exposes content the user could not open directly |
| Retrieval | Finds and ranks the most relevant content | The right document exists but is not retrieved |
| Generation | Writes an answer from the retrieved content | The answer goes beyond what the sources say |
| Evaluation | Measures quality before and after release | Nobody can tell when a change makes things worse |
Getting retrieval right
Most poor answers from an enterprise assistant are retrieval failures, not model failures. If the right passage never reaches the model, no prompt will fix it. Three techniques do most of the work.
Hybrid search
Combine keyword search with vector search. Vector search finds passages with similar meaning, while keyword search reliably finds exact terms such as product codes, clause numbers and names. Most mature search platforms support both in a single query.
Reranking
Retrieve a wider set of candidates, then use a reranking model to put the most relevant first. This often improves answers more than switching to a larger language model.
Metadata filters
Store fields such as document type, business unit, effective date and version, then filter on them. A question about current policy should not retrieve a version that was replaced last year.
Chunking without losing meaning
There is no single correct chunk size. Short chunks are precise but lose context; long chunks keep context but dilute relevance. Splitting along the structure of a document, such as its headings and sections, usually works better than splitting every fixed number of characters. Attach the document title and section heading to each chunk, so that a passage like “this applies from 1 July” still makes sense on its own. Tables, scanned PDFs and slide decks need their own handling, and they are often where quality quietly falls apart.
Design decisions that matter most
Enforce permissions at retrieval
Filter content by what the current user is allowed to see before it reaches the model. Instructing a model not to reveal something is not an access control.
Treat documents as untrusted input
Retrieved text can contain instructions, whether by accident or by design. Keep the actions an assistant can take narrow, particularly once it can do more than answer questions.
Show your sources
Citations let people check an answer and make errors easier to spot. Let the assistant say it does not know when the content does not support an answer.
Manage content as a product
Assign owners, remove superseded versions and record when each source last changed. Many apparent AI errors are really content errors.
Evaluate before you trust it
Build a test set of real questions with known good answers, including questions that should be refused and questions the content cannot answer. Measure retrieval separately from generation, so that you know whether a wrong answer came from finding the wrong content or from misusing the right content. Run the set every time you change a model, a prompt or the way content is chunked.
| What to measure | The question it answers |
|---|---|
| Retrieval recall | Did the right passages appear in what was retrieved? |
| Groundedness | Is every claim in the answer supported by the retrieved sources? |
| Answer relevance | Does the answer address the question that was actually asked? |
| Citation accuracy | Do the cited sources say what the answer claims they say? |
| Refusal accuracy | Does the assistant decline when it should, and only when it should? |
Automated scoring, with a language model acting as the judge, lets you run these checks at scale, but calibrate it against human review on a sample first. After release, collect feedback in the interface and review a sample of real conversations regularly. Users will ask questions your test set never imagined.
Running it in production
Keep the index current
Sync on a schedule or when content changes, and handle deletions as carefully as additions. A deleted document that still answers questions is a governance problem.
Log for audit and improvement
Record the question, the sources retrieved and the answer given, with retention rules that match how sensitive the content is.
Watch cost and latency
Retrieval, reranking and generation each add time and cost. Measure them separately so you know where to optimise.
Version everything
Track the model, prompt, embedding model and chunking settings for every release, so you can explain why an answer changed. Changing the embedding model means re-embedding the whole collection.
A sensible path from pilot to production
- Pick one narrow use case with a clear business owner and a measurable outcome, such as time saved answering policy questions
- Clean up the content for that use case first: remove duplicates, retire superseded versions and assign an owner to each source
- Build an evaluation set of real questions, including ones that should be refused, before writing the first prompt
- Pilot with a small group who will give honest feedback, and review their conversations every week
- Widen access only when retrieval, groundedness and permission tests hold steady across releases
- Add new content sources one at a time, re-running the evaluation set after each
Moving slowly at the start is usually what lets a project move quickly later. An assistant that gives one wrong answer about leave entitlements or a security policy in front of the whole organisation can lose trust that takes months to rebuild.
When RAG is the wrong tool
RAG suits questions whose answers sit in text spread across many documents. It is a poor fit for questions about structured data, such as totals, trends or anything that needs a calculation; there, letting the assistant query the database or reporting layer directly gives more reliable answers. And if the content is small and rarely changes, it may fit straight into the model’s context with no retrieval layer at all.
Privacy obligations still apply
If personal information is involved, the Privacy Act and the Australian Privacy Principles apply. The Office of the Australian Information Commissioner has published guidance on using commercially available AI products, and notes that incorrect or artificially generated information about an identifiable person can itself be personal information. Consider this during design, not just before launch.
Frequently asked questions
What is the difference between RAG and fine tuning?
RAG gives a model access to your content at the moment it answers, so updates take effect as soon as the index changes and answers can cite their sources. Fine tuning changes the model itself, which suits teaching a style, format or specialised task but is a poor way to keep a model current with changing facts. Many enterprise systems use RAG alone, and some combine the two.
Does RAG stop AI hallucinations?
It reduces them but does not eliminate them. A model can still misread its sources or go beyond them. Citations, groundedness checks and an assistant that is allowed to say it does not know are what make errors rare and visible.
Which vector database should we use?
Start with what your platform already offers. Search services such as Azure AI Search, and the vector features now built into mainstream databases, cover most enterprise needs, including hybrid search and security filtering. A specialist vector database is worth considering when testing shows you need its scale or performance.
How do we stop the assistant showing people documents they should not see?
Carry each source document’s access permissions into the index, and filter results by the signed in user’s identity before anything reaches the model. Test with accounts that have different permissions, including one that should see nothing at all.
How long does it take to build an enterprise RAG assistant?
A working prototype on a small, clean set of documents can come together quickly. Most of the effort goes into connecting real sources, enforcing permissions, cleaning up content and building an evaluation set, so plan for that work rather than for the demo.
Can RAG work with content in SharePoint and Teams?
Yes. SharePoint sites, Teams files and shared drives are some of the most common sources. The important part is carrying across each item’s permissions and keeping the index in step as files change, so the assistant reflects what people can see today.
Explore our solution and enterprise architecture services →
OAIC: Privacy and the use of commercially available AI products →
