Tools of the trade
There are two elements to backtesting: The proper tools — software and data — and a scientific method to develop systems using those tools. Let’s start by looking at the tools of the trade.
Many options are available for testing your ideas. They differ in the ease of turning ideas into code and in how they handle the details, which can have a major impact on the results. For example, if a system enters on a limit order, some software records a fill if that price is touched. However, there is hardly a guarantee such an order would have been filled in real trading, nor is there a guarantee it won’t be. Entering on stops guarantees an entry, but not a price.
Another issue is recording real prices. While most professionally developed software no longer has this issue, it is still a concern for those who manually test systems in spreadsheets, such as Microsoft Excel. For example, if a system buys on a stop equal to the close plus one-third of the average range over the last three periods, and if the average range is 10, then we are buying at the close plus 3.333. If we are trading the E-mini S&P 500, it trades in 0.25 tick sizes. This means the entry differential must round up to 3.50. A beginning trader may not realize this if manually crunching numbers, and it wasn’t too long ago that many professional programs made the same mistake. Over time, such an error could add up to a sizable discrepancy.
In the big picture, however, such procedural details are minor. The big issue is data.