# When Does Enterprise AI Actually Need RAG?

> When should enterprise AI use RAG, direct context, a database query, or an API? A practical way to choose the right source for each question.

> 💡 **TL;DR: Key Takeaways**
> - **Not every information question needs RAG.** A small, stable source can go directly into context, while live transaction data should come from an API or database.
> - **RAG solves the problem of finding the right passage in a large document collection.** If the relevant source is already known, another retrieval layer may add complexity without value.
> - **Finding a document does not authorize its disclosure.** Apply access controls before content reaches the model, and test retrieval quality, freshness, and cost with real questions.

Imagine a customer support system that needs to answer three questions:

1. Which filter does this device need for its annual service?
2. Where is my order right now?
3. Does my contract include a special discount for this product?

They all look like questions about information. But the system should not get that information in the same way.

The answer to the first question may sit on a single page of a maintenance manual. You can give that page to the model with the question. The second answer changes constantly and should come from the order system or its API. The third may require both a contract and a current customer record. The system must also make sure that the person asking is allowed to see them.

You could try to solve all three by putting documents into a search system and adding RAG. That does not make it the right architecture.

## What problem does RAG solve?

RAG, or *retrieval-augmented generation*, searches a body of information before the model answers. It finds the relevant passages and adds them to the model's context.

Context is the information available to the model when it generates a response. It can include the user's question, instructions, retrieved documents, earlier messages, and results returned by tools.

If the source material is small, you may not need a retrieval layer at all. In its [Contextual Retrieval guide](https://www.anthropic.com/engineering/contextual-retrieval){.dofollow target="_blank" rel="noopener"}, Anthropic says that including the entire knowledge base can be the simplest option when it fits comfortably in the model's context. Its example of 200,000 tokens is not a universal threshold. The model, cost, latency, and consequences of an error all affect that decision.

Search becomes useful when the source collection grows or when you cannot know in advance which section will matter. Instead of sending thousands of pages to the model, RAG can select the passages most relevant to the question.

That is the core problem RAG solves: **where, inside a large collection, is the information the model needs?**

It is an important problem. It is not the whole of enterprise context.

## Live operational data is not a document

When a customer asks, “Where is my order?”, the answer does not live in a document prepared yesterday. The latest order status, carrier event, and delivery estimate live in operational systems.

You could regularly export those records into documents and copy them into a search index. The copy would begin to age as soon as it was created. The system might retrieve the correct document and still give the wrong answer.

A more reliable approach is to validate the order number and read the current state through an authorized API or database query. The language model can turn that result into a clear response for the customer. The order system remains the source of the information.

An enterprise system therefore needs to ask more than “Which document should I retrieve?” It must first identify where the information belongs:

- Direct context for a small and stable set of sources,
- Enterprise search or RAG for a large collection of text documents,
- A database query or API for current records,
- Deterministic software for explicit, stable rules.

One response may use several of these paths. The data does not have to be copied into one store first.

## Finding the right document does not grant access to it

The most relevant search result might be a compensation table, an employee record, or a customer contract that the user is not allowed to see.

Retrieving the correct document does not make that access legitimate. Authentication, user and group permissions, tenant boundaries, and document-level access rules still need to be enforced.

AWS explicitly warns that document permission filters in Bedrock Knowledge Bases [do not constitute an authorization boundary on their own](https://docs.aws.amazon.com/bedrock/latest/userguide/kb-managed-ds-custom-acl.html){.dofollow target="_blank" rel="noopener"}. The service cannot verify whether the identity supplied by the application is genuine. The application remains responsible for authenticating the user.

Azure AI Search can store document permissions in its index and compare them with the user's identity at query time. But [permission changes in the source system only affect search results](https://learn.microsoft.com/en-us/azure/search/search-document-level-access-overview){.dofollow target="_blank" rel="noopener"} after that metadata has been synchronized. The permission model matters. So does its freshness.

The boundary is straightforward:

> Search finds a document. Authorization decides whether that document may be found for this user.

An unauthorized document should not be handed to the model and filtered after the answer has been generated. It should be removed before it can enter the candidate results.

## More context does not always produce a better answer

As context windows grow, sending every document at once becomes tempting. For a small, stable body of information, that can be the simpler and better option.

But context capacity and context quality are different things. As irrelevant material grows, the model may have more difficulty identifying what matters. In [a study comparing RAG with long-context approaches](https://aclanthology.org/2024.emnlp-industry.66/){.dofollow target="_blank" rel="noopener"}, long context performed better across many of the tests when enough resources were available, while RAG retained a clear cost advantage. The result is not that RAG is obsolete or that long context does not work. Performance depends on the question, the source material, the model, and the economic constraint.

Quality also depends on how documents are divided, whether exact keyword and semantic search are combined, and whether the first results are ranked again. Microsoft's [RAG retrieval guidance](https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-information-retrieval){.dofollow target="_blank" rel="noopener"} notes that reranking can improve relevance but adds latency to every query.

Keyword search, semantic search, and an additional reranking model are not maturity levels. Each is an intervention for a particular retrieval failure. If simpler search works, adding more machinery does not make the system more enterprise-ready.

## The decision to make before building RAG

A company does not need to begin with “Should we build RAG?” A more useful question is:

**How should the right, authorized information reach this system at the moment it is needed?**

If the information is small and stable, direct context may be enough. If the system must search a large document collection, RAG or enterprise search may make sense. If it needs current operational state, it should query the source system. If access varies by user, permissions must be enforced before retrieval.

RAG is one of those options. It is powerful in the right place. When it becomes a catch-all name for every data problem, it starts to hide the source, access, and freshness decisions that matter more.

[I covered the boundary between fine-tuning, direct context, and RAG through information, rules, and learned patterns in the previous article](/does-enterprise-ai-really-need-fine-tuning).

You can find the other decisions about models, information, authority, and evaluation in the [guide to designing an AI system for your business](/designing-ai-systems-for-business).

---

Language: English
License: CC BY 4.0
License URL: https://creativecommons.org/licenses/by/4.0/
Scope: Evren Bal-authored text, unless this article expressly states otherwise.
Excluded: Third-party material, quoted excerpts, logos, and separately marked images retain their own rights.
Attribution: Credit Evren Bal, link to the canonical source and license, and indicate changes.
Source: https://evrenbal.com/when-does-enterprise-ai-need-rag