To judge how well a given trading system should work in the future, we backtest it on past market data. Backtesting applies a set of trading rules to historical data to estimate how those rules would have performed if we actually had traded them. Good hypothetical historical results do not guarantee that a set of rules will work well in the future. However, poor hypothetical historical results almost certainly mean a system should not be traded in real time.
The perceived value of backtesting is rooted in the belief that historical tendencies repeat. Traders have been testing strategies on historical data for generations. However, the practice became popular with the advent of personal computers and purpose-built system-testing software, such as System Writer, which evolved into TradeStation. This software and a database of historical data allowed those without a code-writing background to test trading system ideas. The broader understanding and acceptance of trading systems, as well as the frustration many encountered when trying to build trading systems on their own, helped the market of third-party systems flourish throughout the 1990s.
Futures Truth is an independent company that has tracked commercially available trading systems since the 1980s. Currently, it tracks more than 500 systems. Futures Truth tests trading systems in real time, not on historical data. This prevents the modification of rules over time and better simulates rule execution in actual market conditions, such as periods of high volatility. According to Futures Truth, only about 45% of the tracked systems are profitable in the long-term, while only 20% have exhibited a good risk/reward ratio. However, these numbers likely are better than the broader population’s because only those vendors truly confident in their logic turn it over to Futures Truth for real-time analysis and public critique.
So many systems fail because they lack a valid premise. Instead, the entry and exit parameters are derived from data mining. Data mining simply scans historical data for rules that would have worked in the past. Often, such rules are fit precisely to the past and have no hope of working any better than random on unseen data. Instead, system development should start with a theory that can be tested, analyzed and fine-tuned for application. This concept also implies a different perspective on system testing itself: The goal of backtesting is not to produce a collection of hypothetical profit and loss statistics. It is to test the validity of the theory and the accuracy of the rules in capturing the premise.
System testing is a multifaceted process from the data, to the time scale, to order entry assumptions, to contract specifics and risk control. Failing at any of these can ruin an otherwise valid test — or, manipulating them can generate results that are far superior than what we would achieve in real time. You need to do it right if you hope to validate — or when appropriate, invalidate — your system.