In-Sample vs Out-of-Sample Testing, Explained
When you tweak a bot's settings until the backtest looks good, you are fitting it to that stretch of history. That stretch is the in-sample period. With enough knobs you can make almost anything look profitable in-sample.
Out-of-sample data is history the bot was never tuned on. If a strategy only works in-sample, you probably found noise rather than an edge.
How Bob's Bots splits the data
We test each bot on the whole history, then report the most recent portion separately as out-of-sample. A bot that makes money in-sample but not out-of-sample gets the badge Profitable in-sample only. That is a warning, not a pass.
Walk-forward testing
Walk-forward testing cuts history into consecutive windows and checks each one. A real edge should show up in most windows. One that comes from a single lucky stretch won't.
Luck checks
If you try 50 variations, one of them will look good by chance. The deflated Sharpe ratio corrects for the number of variations tried. A random-entry test compares your bot with bots that enter at random times under the same exits and costs. If random entries do as well, the entry rule isn't adding anything.
The rule we follow
A strategy only counts as validated once it has passed on data it never saw.