From the April 2014 issue of Futures Magazine • Subscribe!

Top 3 mistakes of trading system development

No. 2: Friction-free trading

Take a look at most trading systems for sale and you’ll usually notice a small disclaimer at the bottom: commissions and slippage not included. Likewise, many do-it-yourself developers also ignore commissions and slippage during development. If they do include these costs, they usually underestimate the amount.

You’ll hear various reasons for not including commissions and slippage. “Different brokers charge different commissions” is a common excuse. “My system uses only limit orders” is another. The real reason, though, is to make a system appear better than it really is. Profitable systems become a lot more difficult to discover when real-world trading costs are added in.

For example, take a simple scalping system for the E-mini S&P (CME:ESM14) futures. Without commissions and slippage, such a system might take 20 trades per day, and average $15 profit per trade. The trader might see the result of $300 per day and think that is a nice payday. But add in $5 round-turn commissions, and one tick round-turn slippage (which is probably optimistic), and the $300 per day actually turns into -$50 per day.

One hidden effect of evaluating a strategy without commission and slippage costs is that it leads the trader to take more trades than necessary. Here’s an example. Suppose System A is the scalping system mentioned above. It earns $300 per day before commissions and slippage. Compare that to System B, which only trades one time per day, and averages $50 per trade before trading costs. Anyone evaluating both systems side by side, would surely pick System A over System B. Yet, when commissions and slippage are added in, the exact opposite is true; System B is the only worthwhile approach. It trades less, and commissions and slippage are a much smaller percentage of the gross profits.

It is critical, then, to include adequate amounts for commissions and slippage at the start of development. For the E-mini S&P 500 futures, $5 per round turn for commissions, and one to two ticks round turn for slippage, is a reasonable place to begin. 

No. 3: All data used

A third mistake many traders make during development is using all the available historical data for their testing. Most inexperienced traders will run optimizations and analysis up until the present day. The thinking usually is that the trader wants to be sure the strategy is “tuned” to the most recent data.

Of course, if the first test is a failure, the trader will go back, add a rule or a filter (thereby getting closer and closer to committing the complication mistake discussed earlier), and run through all the data again. 

Eventually, the trader finds something that works, and then moves to live trading. This approach, though, is almost always a recipe for real-time disaster.

A much preferred, and much more difficult, path is to use an out-of-sample test to verify a trading system. For instance, the trader might develop a strategy over the past 10 years of data, but leave the most recent year of data untouched and untested. Once development is complete, the strategy is run on the unseen (out-of-sample) data. If the system performs acceptably on the out-of-sample data, then the system can be considered for live trading.

Taken one step further, walk-forward testing can be used, which combines multiple out-of-sample periods. Such an approach is more likely to lead to  success, because any resulting equity curve is entirely composed of out-of-sample results.

The pitfall to walk-forward or out-of-sample testing is that once the test is run once, any further tests will not truly be “out-of-sample.” So, it is easy to inadvertently make an out-of-sample test an in-sample test, if you perform it multiple times. But performing out-of-sample tests is much better than optimizing on all data. 

<< Page 2 of 3 >>
Comments
comments powered by Disqus