What counts as proof
An in-sample backtest is not proof. What is: out-of-sample and walk-forward results, deflated Sharpe, costs and survivorship taken seriously, a paper record over time, scored calls, and a readiness scorecard.
- Who it is for
- Everyone on the book
- Reading time
- 4 min read
- Updated
Bellwether shows many numbers that look like evidence. Most are weaker than they look, and the console says so next to each one. This is what to trust.
Not proof
An backtest. A rule tuned on twenty years of history and then scored on the same twenty years has been shown the answers. The Best Sharpe in the grid, when the Lab tries a grid of settings, is the most flattered number on the by construction; the Lab's own explanation of the figure says so, and What luck alone would show sits beside it: the Sharpe the luckiest skill-less configuration would have reached.
A raw return on today's mega-caps. The dataset is today's survivors. A backtest over it buys the winners of the last 20 years because they won. Read a strategy by its excess over SPY and the 60/40 mix, never by its raw return, and never treat the raw number as an estimate of live performance.
A backtest without costs, taxes or halts. Every Lab run charges half-spread plus impact on every fill and passes every simulated order through the same the live runs, including the halts. Turnover tells you how much those costs, and taxes, would matter.
A Claude backtest inside its training window. A model asked to "forecast" a date it has already read about may be remembering, not reasoning. Historical walk-forwards here use deterministic only.
Closer to proof
Out-of-sample. First part against last part on a Lab run (the first 70% of sessions against the last 30%) asks whether the second part looks like the first. Nothing was fitted on one part and tested on the other, so it is not a hold-out test; it shows whether the result leans on one era. On years it had not seen, the line of a grid of settings, chooses each year's setting on the prior years only and stitches the unseen years into one curve. That is the only number from a grid that deserves a benchmark comparison, and it still only chooses among the settings you submitted.
. The probability that the true Sharpe beats what the best of N skill-less tries would show, after skew, fat tails and sample length. Below about 0.95 the result cannot be told apart from selection luck. A Lab run prints it as the chance the edge is real, under the question Could it be luck?. A grid's probability of overfitting (the Lab asks Did the search pick noise?) answers the same question from the other side: above 0.5 the search is picking noise.
A paper track record over time. Performance is the book's real behavior: chained daily returns against SPY and 60/40, drawdowns, the trade journal, win rate and expectancy. It cannot be tuned after the fact.
Scored calls. Every 's checkable statements are scored at their horizon from daily closes, with no model grading, and Research shows hits over scored calls on each brief (a combined hit rate across briefs is not computed yet). A Researcher with a hit rate on stated calls has a record; one that only writes prose does not.
The readiness scorecard. Go live (owner only; the Readiness for link at the foot of the Evidence tab of Performance opens it) is a table of gates read from what the system already records. Among them: at least 20 funded paper days, at least 15 closed round trips, a max of 8% or less, a Sharpe of at least 0, no recent incidents, an agent-hygiene audit score of 80 or more, no ignored in the last sessions, AI cost of $0.80 or less per decision and 1% or less of equity a year, and green tests. A gate that cannot be measured yet not passing, and thresholds can only be tightened. The verdict reads All gates green only on a day every gate passes, the screen counts consecutive green trading days against the 5 it needs, and even then it only says the paper record is far enough along to start the cut-over runbook; nothing on that screen flips paper to live.
Reading a backtest card honestly
Open an algorithm's Backtest or a run in Lab › Backtests and go in this order:
- The window and the universe. Survivors? One regime? The card covers 2006 to 2026 on 20 mega-caps; the by-regime table shows what a long bull market hides.
- The benchmark line. Compare Sharpe and drawdown to SPY over the same years before admiring the CAGR.
- The production profile, not research. Production applies today's caps and halts; research is what the rules do with room to run.
- Reasons to doubt this result on a Lab run, or the Read with care list on an algorithm's card. If deflated Sharpe is low, stop there.
- Trades per year and hit rate. Even good strategies sit near 50–55% on daily hit rate; a card claiming 80% is telling you something is wrong.
A card that survives all five earns a weight in the , and a small one first. More weight is earned in paper.