A system that never learns whether it was right

I wrote it into a build specification two years ago and it has decided more architecture than anything else I have thought. A recommender that never learns whether it was right is a toy.

The reason it costs something is that most systems are built so that finding out is impossible. A recommendation is shown, the user does or does not act on it, and the loop ends there because nobody wrote down what the system predicted. There is no error to measure because there was never a commitment to be wrong about.

So the design rule that falls out of it is uncomfortable: the system has to commit, in public, to something checkable, before it finds out. Everything after that is bookkeeping.

KinoLog predicts what you will rate a film, before you watch it, on the card. The number is arithmetic rather than generation: a nearest-neighbour model over your own rated diary. That matters here for one reason, which is that it is reproducible, so the same inputs give the same claim and an error means something.

When you log the film the prediction settles. The signed error goes into a running mean, and that mean becomes a bias correction on future predictions. So far, ordinary.

The interesting part is how hard the correction is held back. It is shrunk toward zero by a factor of n over n plus eight, capped at four tenths of a star, and does nothing at all below six settled films. Six films move the prediction almost not at all. Forty move it by at most four tenths.

That is deliberate underpowering. One scalar can learn "we run half a star low for you" and cannot learn "we run low on horror and high on comedy", and I would rather have a correction that can never do much damage than one that chases noise from five data points and calls it personalisation. The stats page shows the hit rate whether or not it flatters the model, and says "still calibrating" below the threshold rather than printing a percentage of nothing.

Under every ask box here there are three suggested questions. They are the first thing anyone clicks, so one that dead-ends teaches the visitor the feature is broken before they have typed a word.

verify:ask runs all thirty-nine of them through the real retriever on every build and fails if any returns nothing, or if any leaks a passage from a different project. It was proved by planting a nonsense question and watching the build exit 1.

Then it found the harder case, which is the one worth writing down. The check went green and four questions still answered NO_ANSWER. Each had retrieved three passages, all on-scope, all about the right project. The passages simply did not contain the answer, because the source documents state no cost figures and no launch decisions. The model refusing was correct. The suggestion was still wrong.

Retrieval-passing is not answer-passing. An automated check can prove a question finds relevant material and cannot prove the material answers it, and the two failures are identical from the outside. Closing that gap automatically means a model call in the build step, which makes every build cost money and depend on a provider being up. So the check stays where it is, the gap is documented at the three points where somebody would fall into it, and a human asks each new question once.

Bodivo is adaptive training. Every metric it holds is classified as measured, calculated or unknown, and the class travels with the value rather than being inferred later.

The consequence is that unknown is never rendered as zero. A trend below its sample threshold comes back marked insufficient, with the threshold stated. A view with no evidence returns the reason it is empty and what would fill it. Those screens are the product: the app is a claim about a person, and a claim about a person built on four data points is the thing I spent four years being taught not to make.

It is also the least demonstrable feature I have built, because it looks like an empty app. That is the trade, and I would make it again.

None of these is an evaluation framework in the sense a job description means. There is no benchmark suite, no held-out set, no leaderboard.

What they share is smaller and I think more portable. Each one commits to something falsifiable before it finds out, records what it committed to, compares that against what actually happened, and bounds how much the comparison is allowed to change its behaviour. The last clause is the one people drop, and it is the one that separates a correction loop from a feedback loop that eats itself.

The other thing they share is that the honest outcome is usually silence. KinoLog says still calibrating. Bodivo says insufficient. The ask endpoint says it does not know. A system that can only ever produce an answer has no way to be measured, because it has removed the outcome that would have told you it was wrong.