Retrieval-passing is not answer-passing

Under every ask box on this site there are three suggested questions. They exist because an empty text field is the worst interface in software: nobody knows what a thing can answer until they see one answer.

I wrote them the way anyone would. "How is Halyard architected?" "What was the hardest part to ship?" "Why this stack and not something managed?" Then, before shipping, I ran each one through the retriever and counted the passages that came back.

Twenty-eight of the thirty-nine returned zero. Not a bad answer. Nothing at all.

The reason is obvious in hindsight. The corpus is written in each project's own vocabulary, and the retriever requires a query to match at least two terms in a passage before that passage counts. "How is X architected?" contributes "architected", and the documents say "Architecture" as a heading and then talk about workers and queues. One matched term is not two.

`verify:ask` walks every scope on the site, runs every suggested question through the real retriever, and fails the build if any of them returns nothing or leaks a passage from a different project.

Then I proved it, by replacing one suggestion with "What is the airspeed velocity of an unladen swallow?" and running it. It failed, named the question, and exited 1. A check that has never failed has never been tested.

I rewrote all thirty-nine questions in the projects' own vocabulary. "How does sentinel prove a bug is real?" "What is the licensing problem with Discord music bots?" "How is a chapter theme generated?" All thirty-nine retrieved. The check was green.

Because it was cheap, I also asked the live endpoint each question by hand, spaced out past the rate limit, and read what came back.

Four of them answered `NO_ANSWER`. "What does a chapter cost to generate?" "Why was Alibi never launched?" "What does Pitchless cost per post?" "How does OCR handle a handwritten page?"

Each of those retrieved three passages, all on-scope, all about the right project. The retriever had done its job perfectly. The passages simply did not contain the answer, because the underlying documents state no cost figures and no launch decisions, correctly, since I never measured those things.

The model refusing was the right behaviour. The suggestion was still wrong.

Retrieval-passing is not answer-passing. A retrieval check can prove that a question finds relevant material. It cannot prove that the material contains the answer, and the two failures look identical from the outside: the user gets nothing either way.

Closing that gap automatically means putting a model call in the build step, which makes every build cost money and depend on a provider being up. That is a bad trade for a personal site, so the check stays where it is and the gap is documented at the point where somebody would fall into it: in the file where suggestions are written, in the check itself, and in the project's always-loaded context file.

The four questions were replaced with ones the corpus can actually answer, and all thirty-nine were asked of the live endpoint before shipping. That is the process now: the automated check catches the cheap failure, and a human asks the question once.

There is a broader version of this that I keep running into. Every check has a shape, and the shape decides which failures it can see. A check that passes tells you about the failures it was designed to catch and nothing whatsoever about the ones it was not. Green is not evidence. Green plus a plant is evidence, and green plus a plant plus somebody actually using the thing is better.