Skip to content
Log inRegister

Learn trading

Overfitting and optimization

Trading on margin can lose you more than you put in. Past results, including backtests, do not predict future results. Cloud9Trader provides tools and education, not financial advice. Read the full risk warning.

Optimization means running many backtests with different settings to find the ones that did best. It is useful, and it is the easiest way to fool yourself.

Overfitting (or curve-fitting) is when settings match the quirks of the past so closely that they fail on new prices. The more settings you tune, and the finer you tune them, the more likely it is.

The defence is to keep some data back:

  • In-sample data is what you optimize on.
  • Out-of-sample data is a later period you do not touch until the end. Run the chosen settings on it once. If the result collapses, the settings were fitted to noise.
  • Walk-forward testing repeats this in steps: optimize on one period, test on the next, move both forward, and look at the out-of-sample results joined together.

A worked example

Section titled “A worked example”

You optimize a stop loss and a take profit on January to June. The best pair made 18% with a 6% drawdown. Neighbouring settings made between −4% and 3%.

That lone spike is a warning: a small change in the market is likely to push you off it. A setting surrounded by others that also did reasonably well, a broad plateau, is more likely to hold up.

You then run the chosen settings once on July to September. They make 1% with a 7% drawdown. That out-of-sample result, not the 18%, is your best estimate of what to expect.

In Cloud9Trader

Section titled “In Cloud9Trader”

The Optimize tab runs many backtests over ranges of an algorithm’s externals. Exhaustive tries every combination on a grid you define, with up to ten values per external. Genetic Algorithm starts from random settings within your ranges and keeps breeding the best ones, which covers ranges too large to try in full. Every run is plotted on one chart, so you can see whether the best result sits on a plateau or a spike.

Optimizations run on minute data. Re-run your chosen settings as a backtest on every tick before you trust them, and check them on a later period than the one you optimized on. Cloud9Trader does not yet split in-sample and out-of-sample periods for you, so choose the dates yourself.

Section titled “Read next”