Connecting AI to Company Data: RAG, APIs, or Both?

An employee asks an AI assistant, “How many days of leave do I have left?”

The assistant finds the company’s leave policy and explains the annual allowance. It is a useful explanation, but it has not answered the question. The employee’s remaining balance lives in the HR system, alongside leave already taken and requests still awaiting approval.

Connecting more documents will not necessarily give the assistant that balance. It needs a way to read the right record.

This is one of the first decisions I make when connecting AI to business software: which questions need documents, which need current records, and which need both?

I use document retrieval to find and explain guidance, a defined integration to check current records, and both when a question needs a rule applied to someone's current situation.

In my work on the Internal Multi Agent Platform at Comoto, I connected an internal application to enterprise information and services. That work involved more than finding relevant text. The application also needed to identify the user, call the appropriate integration and give them a useful result.

I’ll use a leave assistant as a running example here. The same decisions apply to customer support, finance, inventory and other internal applications.

Start with the questions people will ask

“Connect the AI to our company data” is too broad to design around. I want a few actual tasks first.

Consider these three questions:

These can appear in the same conversation. The assistant should choose its information sources according to the task.

Two terms help describe the implementation. Retrieval-augmented generation, or RAG, retrieves relevant information and gives it to a model to help produce an answer. An API is an interface through which software reads information or requests an operation from another system.

They describe different layers. A document search service can itself be called through an API. Information read from an operational API can also ground a generated answer.

The practical comparison in this article is between searching an indexed document collection and reading a business system for the current task.

Use document retrieval for explanations and guidance

RAG is useful when the answer is contained in material people would otherwise need to find and read: policies, product documentation, operating procedures or support articles.

For the leave-policy question, I would prepare the approved documents for search, retrieve the relevant passages and give those passages to the model with their source information. The response can explain the policy and link the employee to the section it used.

A typical preparation process extracts text, divides longer documents into useful sections and makes those sections searchable. Search might combine keywords with embeddings, which represent similarities in meaning. Microsoft’s RAG overview describes this retrieval-and-grounding pattern.

An embedding model is one part of that system. I covered how I approach choosing one in Similar Text Isn’t Always the Right Answer. For this application, I also need to preserve the document’s title, policy version, applicable audience and source location.

Those details affect whether the answer applies. A relevant paragraph from an old policy, or a policy for a different group of employees, can produce a convincing explanation of the wrong rule.

Before adding more documents, I would test whether the assistant can find and use the right passage for a small set of real questions. Include questions whose answers are absent from the collection. The interface should explain that limitation rather than fill the gap with an unsupported policy.

Read operational records through a defined integration

A leave balance is a different kind of information. It changes as people take leave, cancel requests and receive adjustments.

I would usually read that balance from the HR system when the person asks, through an API or another supported integration. An approved data service or reporting replica may also fit if its refresh delay is acceptable for the task.

The application should know whose balance it is requesting. For a “my balance” operation, I would derive the employee identity from the authenticated session and the application’s account mapping. A name or employee identifier written into a chat message does not establish permission to read that record.

I would give the model a narrow operation, such as get_my_leave_balance, backed by application code. That code handles authentication, access checks, the upstream request and the response format. The model does not need an unrestricted connection to the HR database.

The result should retain the meaning of the fields. Eight days available now, eight days before pending requests and eight calendar days are different answers. I want the integration to return the units and status information needed to explain the value correctly.

Some questions need very little generation. The application can display the balance, its unit and the time it was checked directly. A short explanation can accompany that display without asking a model to recalculate the authoritative record.

Combine the sources when the task needs both

Now consider the employee asking whether they can take a particular set of dates.

The assistant may need a policy passage, the current balance, the working calendar and any pending requests. It also needs to understand what the application is allowed to conclude. Enough available leave does not necessarily mean a manager has approved those dates.

I would separate the work into explicit steps:

1. Resolve the signed-in employee and validate the requested dates.
2. Retrieve policy information that applies to that employee.
3. Read the permitted balance and relevant request records.
4. Apply defined calculations and business rules in application code.
5. Explain the result, its sources and any remaining step.

That could produce an answer such as: “The dates use three working days under your calendar. You currently have eight days available. You can submit the request for approval.”

The application should preserve the distinction between checking a request and submitting one. Creating the request introduces a separate operation, with its own validation, permissions and confirmation of the result.

The information flow can stay small:

Employee question
        |
Authenticated application and access checks
        |
        +-- Policy search --> Relevant passages and sources
        |
        +-- HR integration -> Permitted records and timestamps
        |
Application rules and calculations
        |
Answer, supporting sources and next step

For this task, a defined flow keeps the source selection and business rules easy to test and explain.

Decide how current the answer needs to be

“Live data” needs a more precise definition than “we called an API.”

An API may return a cached value or read from a reporting system that refreshes overnight. A document index may refresh frequently enough for a policy-search task. The right question is how much delay the business can accept for this particular answer.

For each source, I would record:

• What system owns the information.
• When the source record last changed, where that timestamp is available.
• When the application fetched or indexed it.
• What should happen if the information is too old.

The time a response arrived does not prove when its underlying data became current. I would keep those timestamps distinct.

For a policy collection, changes and deletions need to reach the search index. For a cached balance, the cache needs an appropriate scope and expiry, with changes invalidating it where the integration supports that.

The interface should make important limits visible. “Balance last checked yesterday” gives the employee different information from “Current balance checked just now.” Neither message should imply that an unavailable source was successfully read.

Carry permissions through both paths

The employee’s access should apply to document search as well as operational records.

I would filter document retrieval to material the person is authorised to access before that material enters the model’s context. Asking the model to hide restricted passages after supplying them does not provide that boundary. Microsoft’s security-filter pattern gives one example of applying access filters during retrieval.

Operational calls need the same care. A connector using a shared service account may have broader access than the employee. The application still needs to enforce the person’s permitted operations and records. OWASP’s authorisation guidance recommends checking permissions on every request.

I treat retrieved documents and API responses as information to use, rather than instructions that can change the workflow. The application decides which operations are available and checks access before each call.

I also check what happens when access changes. Indexed permission metadata, saved answers and cached results can outlive the access that originally allowed them to be created. Their handling needs to follow the application’s sharing and retention rules.

These checks belong in the software around the model. They let the business control which information each workflow can use.

Give partial failures a useful response

The policy search might work while the HR integration is unavailable.

In that situation, the assistant can still explain the policy. It should also make clear that it could not check the employee’s balance. It should not turn a failed lookup into a zero balance, or reuse an older value without identifying it as such.

For example:

I found the policy explaining how leave is counted, but I couldn’t retrieve your balance. You can open the HR system to check it, or try the balance lookup again.

A missing record, an access restriction and a temporary connection failure may need different next steps. I would preserve those distinctions in the integration result so the interface can respond appropriately.

The same principle applies to a combined answer. If the application has only the policy, it can explain the rule. It cannot confirm that the employee has enough leave for the requested dates.

Test the questions and the boundaries

Before expanding the assistant’s scope, I would test the complete journey with a few representative users and questions.

I would review the retrieved passages, integration results and displayed answer together. That makes it possible to locate an error: the wrong document was selected, a source was stale, a field was misinterpreted or the explanation went beyond the evidence.

Choose the data path before expanding the assistant

Here is the decision table I would use at the start of an engagement:

These choices can support an enterprise workspace, a configured agent or a custom application. My comparison of ChatGPT Enterprise, Claude Enterprise, Copilot Studio and custom AI applications covers that wider delivery decision.

A custom integration can keep the systems the team already relies on while giving employees a simpler way to use them. The application can own the access rules, data paths and operating visibility, with the model helping people ask questions and understand the results.

I would begin with one useful task and identify the information needed to finish it. Then connect the appropriate sources, make the outcome inspectable and expand from that working flow.