Give it 5% of the training set. Watch the other model fall over.
With only 5% of the training data, Olivet still hits 82% accuracy. A leading neural net falls to 35%. Same task.
HANDWRITTEN-DIGIT ACCURACY AS YOU THROW DATA AWAY
Horizontal axis is how much of the training set you keep.
People pitch AI as a data problem. More rows, better model. That story is convenient if you sell storage. It is a bad story if you sit on a desk with eighteen months of a series that matters and a decision on Tuesday.
So we starve the models on purpose. Handwritten digits, the oldest public test in the cabinet, because everybody already knows the ceiling. Full data: both methods look fine (97% versus 96%). Cut to half: the leading neural net wobbles. Cut to a fifth: it is already in the 70s. At 5%, it is guessing with extra steps. Olivet decays. It does not collapse.
Why? A leading neural net is an excellent memorizer. Memorization is a luxury. When the history is short, you need the shape of the thing, not a scrapbook of the examples you happened to see. That is the same reason a volume model still works after the market shifts, and a price model still has a pulse after a break: it fitted a mechanism, not a playlist.
Then make it argue
Accuracy without a ranking is a coin flip you cannot debug. On a public wine-quality task the order is boring in the best way: alcohol, volatile acidity, sulphates. A chemist can hate those weights. That is the point. If they cannot argue with the model, you do not have a model. You have a mood.
FEATURE WEIGHTS (WINE QUALITY, PUBLIC SET)
We run this kind of test because client data looks like the right-hand side of the first chart, not the left. Thin. Proprietary. No giant public photo set hiding in the cupboard. If a method only works with a million tidy rows, it will embarrass you the first week it meets a real dataset.