What Happens If AI Gives a Wrong Answer?

Let's start with an honest answer: the AI will make mistakes. So does a new operator, and so does an experienced specialist. The right question is not "will it make mistakes?" but "what happens when it does, and who will notice?"

The typical fear goes like this: the bot tells a customer delivery is free when it isn't. Or it promises a discount that doesn't exist. Let's look at why this happens and how we prevent it.

Why the model gets things wrong

A language model is not a database of facts. It generates the most likely continuation of a text. When the information it needs isn't at hand, it fills the gap with plausible-sounding text. This is called a "hallucination", and the dangerous part is that a wrong answer sounds just as confident as a right one.

The main causes of errors:

• the model doesn't know your rules and answers from general knowledge;
• information in the knowledge base is outdated, or documents contradict each other;
• the question is ambiguous and the model read it differently;
• the model was given more freedom than it needs.

Rule one: answers only from your materials

The assistant should answer from your documents, not from general knowledge. We use an approach called RAG: when a question comes in, the system first finds the fragments in the knowledge base that are closest in meaning, then passes them to the model with the instruction "answer only based on these texts". We show how meaning-based search works in our article on embeddings.

The answer comes with its source: which document, which section. An employee or customer can check it in one click. That builds trust and makes errors quick to find.

Being able to say "I don't know"

One of the most valuable traits of a good assistant is saying "I don't know" at the right moment. If the search found no suitable fragments, or what it found is only loosely related to the question, the bot doesn't make things up. It answers honestly: "I don't have information on that, let me connect you with a colleague." For the customer, that is far better than a confident wrong answer.

Talking and doing are different things

There is a big difference between answering a question and taking an action. A wrong answer is unpleasant; a wrong action (a refund, a cancelled order, a contract sent out) causes real damage. So the rule is simple:

• the model prepares a draft of the action, and an employee confirms it;
• the model's permissions are minimal: it reads only the data the scenario needs;
• any step involving money, a customer or a legal commitment goes through a person.

This principle matters for security too. We explain the risk of commands hidden inside documents in our article on data security.

Test questions before launch

"It seems to work fine" is not enough. Before launch we collect 100–200 real questions from customers and staff and write down the correct answer for each. The set always includes:

• common everyday questions;
• questions whose answer is not in the knowledge base (the bot should say "I don't know");
• ambiguous questions;
• off-topic requests and provocations.

After every change (a new model, new documents, an edited instruction) the set is run again. That way quality is measured in numbers, not impressions.

Monitoring after launch

The work doesn't end at launch. Every conversation is logged, customers can rate answers, and failed conversations are reviewed weekly. In our experience most errors are not in the model itself but in gaps in the knowledge base: a rule changed and the document wasn't updated. In those cases, fixing the document is enough.

Handing over to a person

The bot shouldn't try to solve everything on its own. The conversation goes to an employee when:

• the customer explicitly asks to talk to a person;
• the issue involves a payment or a complaint;
• the bot fails to understand the question twice in a row;
• the customer is clearly unhappy.

The employee sees the whole conversation history, so the customer doesn't have to explain everything again.

The bottom line

You can't bring the risk of error down to zero, but you can manage it. Answers only from your materials, cited sources, a bot that can say "I don't know", important actions confirmed by a person, quality measured with test questions. These five rules turn AI from an unpredictable tool into a predictable assistant.

We apply these rules from day one in our AI integration projects. We explain how an assistant works safely with a CRM or accounting system in a separate article, and cover project budgets in our article on costs. Got questions? Write to us.