From the January 01, 2012 issue of Futures Magazine • Subscribe!

Back- vs. forward testing: Test twice (or more), trade once


While it can be exciting to optimize a system and watch the theoretical results improve with a few simple tweaks, over-optimization, or curve-fitting or over-fitting, is a threat to a trading system’s success in actual trading.

Curve-fitting is the excessive use of optimization to create the most profitable system possible based on the historical data. It could involve the use of too many technical indicators, limiting the days of the week that a system will take trades, restricting the times of day the system can trigger a trade and using other filters and triggers that create the best possible scenario on a given data set by eliminating all or most of the losing trades.

The problem with curve-fitting is that virtually any system can be optimized to show close to 100% profitability (the percentage of winning trades) — on historical data. That’s because traders can manipulate the system to take advantage of every price change, creating a system that is custom-designed only for that particular time period.

Often, this leads to a limiting system that performs exceptionally well on the selected historical data, that quickly turns into a train wreck on any other data set, including live trading. "Reality bites" (below) shows the equity curve of a strategy that performed well during backtesting, but once forward testing began the results were mediocre, indicating that the strategy was likely curve-fit.

Traders can avoid curve-fitting by paying attention to key performance metrics, such as percent profitable and average trade net profit, and aiming for realistic thresholds. Many profitable trading systems, for instance, are 40% to 60% profitable — nowhere near the 100% that is so enticing to some traders. All trading systems will have losing trades. A high percent profitable number probably indicates that the system has been over-optimized.

The rule on optimization is to use broad logical principles. If you find eliminating long entries on Tuesdays improves your performance, that is probably coincidental regardless of how it improves your testing results. You also must save clean data to test after you optimize. Traders also can avoid curve-fitting by continuing to test a system beyond backtesting using in- and out-of-sample data, and by taking advantage of forward performance testing.

<< Page 2 of 4 >>
comments powered by Disqus