Building a trading strategy: After testing and optimization

Traders know how hard it is to come up with systems that have a sustainable edge. Many people can look at a chart and determine trading rules that fit a particular chart perfectly. However, when they test their rules over a large period of market history, they learn that their system just doesn’t work.

Other traders will develop a system neglecting commissions and slippage. The trading system looks great, and the trader is very enthusiastic—until these real-life frictional costs are added in, and a winning system turns into a breakeven system or worse.

With a proper test and development method, it is common for traders to test dozens, hundreds or even thousands of trading ideas before they find one that works. A sound development process eliminates a lot of worthless trading systems. Yet, once a trading system passes initial development and testing, work is far from done. There are more steps necessary before committing real funds to a trading system. Following these “after development” rules can save a trader thousands of dollars, and eliminate a lot of heartache.


Monte Carlo simulation

One mistake traders make is to base their trading decisions on the shape of the historical backtest equity curve. Many will look at the equity curve and think “because the historical maximum drawdown is $X, I’ll probably never have a larger drawdown than that.” Or they will conclude “this system has made money every year, so it is safe to trade.”

Unfortunately, both of these assumptions are naïve. This is because the historical equity curve is just one of many possible paths the trading system could have taken over time. This is depicted in “Which way?” (below). Rearrange a few trades here and there, and the maximum drawdown could have been much worse, and profitable periods easily could have become losing periods. On the flip side, the performance could have been better than what it was, too.
 

Is there a way to analyze this phenomenon—to see the various different paths a trading system could have taken? Yes, there is, with a method called Monte Carlo analysis.

Monte Carlo analysis is based on the idea that trades in the past will occur in the future, just in some different, and unknown, order. By scrambling up the trades in a different order, different equity curves will be generated.

Although there are circumstances where Monte Carlo testing is not appropriate, for most trading systems, the analysis is valid and can provide insight into the trading system. You simply record each trade result and then pick trades out in various orders to create an equity curve. Computers obviously do this sampling process a lot faster, and create thousands of simulated equity curves.

When you analyze thousands of Monte Carlo simulations, you obtain statistics of what the biggest drawdown could be, what annual return is possible and your risk of ruin. If you have performance objectives for simulation results, you then can easily see if your system meets your goals. There is no way to obtain such information from just looking at the historical equity curve.

Incubation

When most traders finish developing a strategy, they typically are so excited  that they rush into real money trading. This can be a big mistake, especially for newer traders. A lot of this has to do with trading software. Most modern trading software makes it easy to create systems, modify rules and perform multiple optimizations. In fact, most software is set up to encourage optimization—the software makes it easy to do. 

So, with the software promoting a lot of these bad habits, the historical equity curve invariably will look better with each new test. But, most of the time a better looking backtest has zero (or even negative) correlation to future performance. Why is this? Simply put, adding rules, introducing filters and running more optimizations have the effect of fitting the trading strategy to the historical data, a huge risk when optimizing. 

Many times, watching the strategy after development yields important clues. This process is called incubation. In its simplest form, you let the strategy run live, but without actually trading, for a period of three to six months. Once a month, you look at the results and add those new results to your historical backtest equity curve. An example of this is shown in “System watch” (below), where the incubation results are added on to the walkforward historical results. In this case the incubation results look similar to the historical ones, a sign that strategy development was performed correctly. 
 

After a few months of incubation, you’ll likely see one of two things: The performance of the system “looks” the same as the historical testing, or it does not. One trick is to print the equity curve (historical and incubated ) as large as possible, and tape it to a wall about 10 feet away. If you can see where the historical testing ended and where incubation began, you have a problem. Ideally, there should be no change in performance.

A more statistically oriented test you can run to determine if performance has changed is a histogram of daily results. Again, the point is to see whether or not the system performance changed after you developed the system.

What could cause a performance change? Certainly, if the market is behaving differently than anytime during the tested history, the performance could change. But assuming you have tested over years of data, though, it should  not be dramatic. More likely is that there was a mistake made during development. For example, the strategy might have been over-optimized.

System correlation

Another area traders need to watch is correlation to other systems. If you are trading only one system, you have nothing to worry about. But, if you are adding a system to your portfolio, you have to ensure that your new system complements the others. You don’t want all your systems buying and selling at the same time.

One way to achieve non-correlation is to trade different markets and different time frames. If you like mathematical proof of non-correlation, you can run a linear regression of trading system results. The best way to do this is to compare results of the two strategies, and see the correlation coefficient. If the results are between –0.5 and +0.5, you have a pair of systems that are not correlated with each other. “Trading twins” (below) shows what correlated and non-correlated systems might look like.
 

Correlation of systems is important because a key trading factor is proper risk control. When two or more systems are highly correlated, your risk level increases dramatically. Non-correlated systems create a smoother equity curve and are key to proper diversification.

After you have run Monte Carlo simulations, incubated a system and checked correlation to your other systems, it may be time to trade a small amount of contracts. Why only a small number? There are a few reasons. First, there is a chance that systems passing all previous steps still might fail. Second, trading small is a great way to check assumptions you made regarding slippage. You may find that slippage is a lot more or a lot less than you have anticipated and therefore your position-sizing approach might change.

Finally, you might decide you do not like the system, even though it produces profits. If you are not comfortable with a system, you likely won’t follow it properly. It is much better to find out you don’t like trading a system while you are trading small.


Take your time

After a system is developed, real money trading is not the next step. Instead, proceeding with caution and carrying on with more analysis is the best approach. From Monte Carlo simulations to incubation to correlation analysis and limited real money testing, performing these extra steps can save you money and heartache. In the end, you’ll have a more robust strategy.


Kevin J. Davey has been trading for more than 20 years. Kevin is the author of the Wiley Finance book “Building Winning Algorithmic Trading Systems.” You can reach him via his website  at www.kjtradingsystems.com.

About the Author

Kevin J. Davey has been trading for more than 25 years. Kevin is the author of “Building Winning Algorithmic Trading Systems.”