How do we know when an AI's search is correct?

Everyone asks an AI questions. Almost nobody asks the follow-up: when the AI went looking for the answer, how do we know its search was right?

Because here's the uncomfortable bit — when an AI answers you, it usually searched something first. Documents, a knowledge base, a codebase. And if that search quietly failed, the AI doesn't look less confident. It looks exactly as confident, about a wrong answer.

The background, in plain English

A keyword search can fail two very different ways, and they look identical.

Say you search a company's records for "staff discounts" and get nothing back. Which world are you in? Maybe there genuinely is no staff discount. Or maybe it exists and the paperwork calls it an "employee purchase programme". The search returns the same empty screen either way. It has no way to say "wrong word, try again" — silence is its only vocabulary.

Humans handle this instinctively. An empty result makes you shrug, rephrase, try a synonym, ask a colleague. You treat the silence as a shrug, because that's what it is.

An AI in a hurry treats the silence as evidence. "I found nothing" becomes "there is nothing" — and then it says so, fluently, in a full sentence, with the calm authority of something that has never once doubted itself. The search failure has been laundered into a fact. That's the actual danger with AI-assisted answers: not that the machine can't find things, but that it can't tell you when its own not-finding means nothing at all.

You can write rules against this — "always check the knowledge base first", "never assert absence from one empty search". We did. Rules like that lose to habit, reliably. What works is changing the tool, so the honest behaviour is the easy behaviour.

How the system works now

At Btab, our AI agents answer questions against an internal knowledge base all day. Here's what happens now when one of them searches it:

1. They ask in plain language. Not carefully chosen keywords — the whole question, as you'd say it. The tool breaks it into words, ranks every page by how many different words from the question it carries, and weighs rare words heaviest. That one ranking rule, measured on a test set of 27 real questions, took us from 86% recall (plain keyword search) to 100% — with zero silent misses. No AI inside, no synonym dictionaries (we built one, measured it, and it added exactly nothing). Just careful counting, in about 110 lines of code, answering in 56 milliseconds.

2. The answer is never a blank screen. This is the heart of it. Every response comes back with three honesty devices attached:

• A verdict. Strong match means "the answer is here, read it." Weak match comes with an instruction printed in the output: do not conclude absence from this — go verify in the source. The tool literally tells its reader when it isn't sure.
• A confession. If any word from your question appears nowhere at all in the knowledge base, the tool says so, by name. "The word 'discounts' appears nowhere here" is real information — it's the difference between "no" and "wrong word".
• A map of its own edges. Every answer ends by listing which topics the knowledge base covers and which it doesn't. So "no results" about an uncovered topic can never masquerade as "this doesn't exist".

3. Every wrong answer makes it stronger. When an AI still gets something wrong, that question goes into the test set, and the whole suite re-scores in about a second. The bar only moves up. The tool earns trust the way anything does — by keeping a record of its mistakes and being checkable against it.

So — how do we know when an AI's search is correct?

Honestly: we often can't, not from the answer alone. Confidence is free; the wrong answer and the right one arrive in the same tone of voice.

What we can do is refuse to let a search stay silent about its own blind spots. Make it state its confidence. Make it confess the words it never found. Make it show the edges of what it was searching in the first place. None of that requires clever technology — ours is a hundred-odd lines of ordinary code — it requires deciding that an empty result is a question, not an answer.

An empty search is not a fact. Build your tools so they say so out loud.