Does Enterprise AI Really Need Fine-Tuning?

💡 TL;DR: Key Takeaways

• Do not put changing facts into model weights. Use direct context for a few documents, RAG for a large and changing source collection, and an API or database for live records.
• Keep explicit business rules in software. Training a model to reproduce a refund deadline or permission boundary makes an auditable rule unnecessarily uncertain.
• Fine-tuning makes sense when the model must learn a durable pattern from examples. First show that simpler methods fall short, then test the adapted model on held-out data and measure the complete business outcome.

A customer asks whether a device you sell works at minus 10 degrees. The answer is clearly stated on its one-page technical data sheet.

Would you fine-tune a model to answer that question?

You should not. Give the model the relevant data sheet, or the relevant passage, together with the question. If there are two or three source documents, they can go into the same context.

The situation changes when the document set runs into the thousands, the content changes often, or each user has different access rights. The system must then find the right source before it gives the relevant passage to the model. That search-and-retrieval job is what RAG, or retrieval-augmented generation, is for.

Neither direct context nor RAG changes the model's training. The model uses a current source while answering. When the data sheet changes, you update the document instead of retraining the model.

So when does fine-tuning make sense?

The answer depends on what is missing from the system: information, an explicit rule, or a pattern that has to be learned from examples.

Do not train a model on information that is already in a document

Many enterprise AI systems need access to information. The model may need to know product specifications, a current price, a return policy, or the status of a customer's order.

When the information sits in one or a few short sources, it can be provided directly. As the number of sources and users grows, RAG, enterprise search, a database query, or an API may become the better option. The right choice depends on where the information lives, how often it changes, and who is allowed to see it.

Fine-tuning is usually an unnecessary detour here. Information a model saw during training is not guaranteed to be current or used correctly in every answer. Providing the source at the time of the request also makes it easier to see what the answer is based on.

The same distinction applies to company rules. “Returns are accepted within 30 days” may be information. “If the deadline has passed, route the request to an agent” is an explicit business rule. It may be more reliable and auditable to define that rule in software than to teach it to a model through examples.

Sometimes the answer is not in a document

Consider a retailer trying to forecast next month's demand. It may have historical sales, promotions, seasonality, holidays, and stock levels. But no document contains next month's sales figure.

The system has to learn relationships in past examples and produce a forecast for a new period. Training a model makes sense here. That model may be a time-series model, a classical machine-learning method, or another purpose-built forecasting system. It does not have to be a fine-tuned large language model.

This distinction matters. “Training a model” and “fine-tuning an LLM” are not the same decision. The first covers any model that learns a pattern from historical data. The second adapts a pretrained model to a narrower task using specific examples.

Fine-tuning becomes worth considering when a model has the right information and clear instructions, yet keeps making the same behavioural error. It may repeatedly classify incoming text incorrectly, extract required fields inconsistently, or break the requested output structure. Representative input-and-output examples can improve performance on such a narrow task.

Even then, start with the right model, clear instructions, the necessary context, and a few good examples. If those do not reach the required quality, use a test set that can show whether the failure really comes from model behaviour before evaluating fine-tuning.

A real example of where fine-tuning belongs

MEDIC, a system developed to reduce errors in prescription directions at Amazon Pharmacy, draws the boundary well.

Prescription directions written by clinicians can express the same information in many different ways. The system needs to identify parts such as the use verb, dose, route of administration, and frequency. The answer is not waiting in a single technical document. The system needs to recognise the same fields across many forms of free text.

The researchers used a small DistilBERT model adapted with roughly one thousand expert-labelled examples for that task. The model extracts the relevant parts of a prescription. Software then checks those parts against pharmacy rules and drug databases. When information is missing or contradictory, the system stops rather than producing a recommendation. Technicians and pharmacists retain the final review.

During MEDIC's deployment at Amazon Pharmacy, the researchers reported a 33% reduction in prescription-direction near misses: errors caught before an incorrect direction reached the patient. The result is limited to one organisation and task, but it makes an important point. Fine-tuning did not replace a current source of information. It improved the system's ability to extract consistent fields from free-form text.

Another finding from the study is equally useful. A larger T5 model fine-tuned on 1.5 million examples kept confidently generating directions in difficult cases instead of recognising its limits. MEDIC's result did not come from an adapted model alone. A narrow model task, explicit stopping conditions, verification sources, and human review worked together.

RAG and fine-tuning are not competitors

RAG brings relevant information into the system at the time of the answer. Fine-tuning changes how a model behaves for a particular kind of input. A system can use both.

For example, it may retrieve a current product document with RAG and provide the relevant passage to a model. If the model must then extract fields into the company's schema, and still does that inconsistently despite well-designed instructions, fine-tuning may be worth considering for that narrow task.

A comparison by Microsoft researchers on agricultural questions also found that the two approaches could improve results together. The study is limited to one dataset and particular models. Its useful conclusion is not that every system should combine RAG and fine-tuning. It is that the two approaches can complement each other because they solve different problems.

Start with what is actually missing

It is easier to make the decision from the missing component than from a list of method names.

Before moving to fine-tuning, four answers should be clear:

1. On which real inputs does the model repeat which error?
2. Do the right source, clearer instructions, or a few examples solve that error?
3. Can the improvement be measured on a representative test set that was not used in training?
4. Is the quality or cost benefit worth the work of preparing data, tracking model versions, and retraining when needed?

I previously explained why an AI project should not start with model selection. The fine-tuning decision should follow the same order. First identify why the system is falling short. Then decide which component can close that gap.

Building an enterprise AI system does not mean putting everything the company knows into a model. Information can be read from its source. Rules can be applied in software. A separate model can learn patterns in historical data. Fine-tuning becomes useful only when the problem is truly repeated model behaviour.

The first question, then, should not be “RAG or fine-tuning?” Ask instead: is the system missing information, a rule, or a pattern it needs to learn?

Next, I will look at the line between a workflow whose steps are defined by software and an AI agent that chooses its next step.

You can find the other decisions about models, information, authority, and evaluation in the guide to designing an AI system for your business.