There are many ways to build a bad trading system, but two of the more common are the lack of a valid premise and the selection of non-robust parameters. To overcome these issues, developers often split their data into two sets: the development, or in-sample, set and the test, or out-of-sample, set. A common split is using 80% of the data for development and 20% for validation.
However, this split often leaves much to be desired, particularly for systems that trade infrequently, such as trend-following systems. Often, 20% of the data provides too few trades for a statistically significant test. Also, even with a development set and a test set, the process can become contaminated after just a few modifications and out-of-sample tests. Each time the testing process is repeated with a different system, degrees of freedom are reduced. If enough testing is done using the data, the out-of-sample data effectively becomes in-sample data, and the advantage of this method of testing is lost.
A better approach is called walk-forward analysis, originally discussed by trading system developer Robert Pardo, author of “Design, Testing and Optimization of Trading Systems” (John Wiley & Sons, 1992). This technique also performs optimization on a training set, but it uses a much smaller portion of the in-sample data. It then tests the system on the following block of data. After the test, it shifts the development set forward by just a portion — say, one month for a test period of a year. The optimization/test then repeats. Because walk-forward analysis uses multiple training and testing periods, it is less likely to contaminate the data.
However, one issue with walk-forward testing is it can be labor intensive. That’s where trading technology comes into play.
A more detailed example of walk-forward analysis can better demonstrate the challenges and benefits of this approach. Walk-forward testing works like this. Say that you have 12 years of data extending from 1998 to 2009. Assume that your trading strategy needs a minimum of three years of data for testing and optimization.
Start by developing and optimizing the system using only the first three years of data — in this example, 1998 to 2000. On these three years of data, try as many ideas as you like and optimize parameters in as many ways as you can. Under no circumstances do you look at the data after 2000. When you are done with development, record the rules for the system with the optimum parameters. These rules and optimized parameters will be used later for the final testing with new data starting with 2000.
Slide the three-year time window of data forward — say, one month. Now, your dataset runs from the second month of 1998 to the end of the first month of 2000. Repeat the analysis, including optimization, and record the rules and optimized parameters. In the final pass, these parameters will be used for trading the second month of 2000.
Continue walking forward and optimizing the three-year data periods. Record the results for use in the first month following the three-year optimization period. When you reach the end of your data, go back and test the system for the entire period from 2000 to 2009. Switch the rules and parameters each month to the ones that you found and recorded. In effect, you are performing a new out-of-sample test for each month. The system performance for these nine out-of-sample years (108 out-of-sample months) is a better indication of how a system will perform in real time than the performance of a single out-of-sample test. If the results for the out-of-sample months look good, continue the walk-forward process in real time before risking real money on the system.
If we want to do true walk forward testing, a few issues need to be addressed. You must determine how you will change the parameters used in the out-of-sample walk-forward period. If a parameter value changes from 20 to 22, this isn’t an issue; however, if there are two peaks with one at 20 and the other at 50, a decision must be made regarding which value to use.
Another problem is what is a transitional trade, also called boundary trades. If both the old and new parameters would be trading in the same direction at a transition, there is no issue, but this isn’t always the case. We can handle these in three different ways.
One way is to exit the boundary trade of the old period and enter the trade of the new period right away. You cannot exit the trade from the old period if the trade from the new period is in the same direction. You can exit the trade from the old period and wait to the first new entry in the new period if they are in different directions between the periods.
Consider a simple example. We will use natural gas from 1991 to Nov. 25, 2009. We will use a triple moving average crossover system and $100 for slippage and commission. We are using the following parameters in our walk-forward analysis:
Short moving average = 5 to 20, steps of 5
Medium moving average = 20 to 60, steps of 10
Long moving average = 60 to 100, steps of 20
We used a training set of 750 bars and an out-of-sample period (run period) of 175 bars (see “Walk-forward report”).
The report shows each training set and run set. The criterion column shows the net profit/drawdown ratio for the training set. We can assume that sometimes the best set of parameters will stay constant over time. We see the short and medium moving average are the same from July 13, 2001, to Oct. 5, 2007, while the long-term average during this period oscillates from 60 to 100. We can see that 13 of 23 out-of-sample run windows are profitable. We also had three of them in a row lose money in the Oct. 31, 2002, to Dec. 13, 2004, window. Considering this is a trend following system, our results are reasonable.
Next consider the raw trade report (see “Trade by trade”). The raw trade report shows each trade for both the training and run periods from Jan. 3, 1991, to Feb. 2, 1995. The runs with “out” in the last column are in the training set. The “trans” trade is in both the training and out-of-sample set. The last two trades are in the “run” or out-of-sample set.
The next way to handle transition trades is to exit all trades, exit only if direction changes or exit and reenter. Exiting all trades is simple. At the end of a boundary, all open trades are exited and a new signal is used to get back into the market. While this is a simple approach, it can have a significant effect on system performance. Exiting only if the trade direction shifts produces more realistic results and, in the case of longer-term systems, does not disrupt a long-term trade that may be responsible for significant profits.
Our final transition-trade solution is to exit and reenter in the current direction. This option is similar to only exiting if the direction changes with one exception. If the direction changes, a new position is opened on the first day of the new out-of-sample period. This concept is often best for long-term trend-following systems.
CASE STUDY: PORTFOLIO EXAMPLE
Most traders trade a collection, or portfolio, of markets. This complicates the procedure, but it is not necessarily complex. This is the portfolio we’ll review:
• Natural gas
• Japanese yen
• 30-year Treasury bonds
• Crude oil
We will use overlapping dates from Jan. 3, 1991, to Nov. 25, 2009. We will also deduct $100 per trade for slippage and commission. We will use the triple moving average crossover system and again select our parameters based on the net profit/drawdown ratio. Our optimization scan will be as follows:
Short moving average = 5 to 20, steps of 5
Medium moving average = 20 to 60, steps of 10
Long moving average = 60 to 120, steps of 20
Our training period will include 1,250 data points, and the out-of-sample period will have 300.
The walk-forward results are solid (see “Portfolio report,” right). Our data period includes 12 testing windows. Out of these 12 windows, 10 were profitable. We also can see that our parameters are stable. The last three windows use the same set of parameters, and two other times we had the same set of parameters in two adjacent windows. These last three windows all used 5, 60 and 80. Our walk forward analysis produced $321,953.75 using our exit all trades on window boundary mode.
This system is a trend-following system, so exiting trades on each boundary is not optimal. If we try the second approach that exits only if direction changes, we make $408,957.50. Exiting and reentering each trade makes $406,648.75. Walk-forward analysis is a type of adaptive system technology, and most trend following systems did badly in 2009, but our walk-forward analysis here was profitable in 2009 because parameters were allowed to change over time.
While we used the built-in ratio of net profit/drawdown to determine our optimal parameter settings, there are times we may want to program custom functions. For example, if we want to add a simple function that weights recent results more, we could add the below code fragment to the measure:
If BarNumber = LastBar and drawdown <> 0 Then
setoptimizefactor((NetProfit + NetProfit - NetProfit[BarSize - 1]) / Drawdown)
We can then just select “optimize factor” in TradersStudio and use this custom function. Custom functions can be as complex as needed. You could even use a function such as optimal f to evaluate the system or product scoring based on objective rules.
One goal with parameter selection is to select a set of parameters where neighboring sets produce similar results. When developing a system, we can judge this robustness by producing three dimensional charts of parameters instead of just a raw performance score. We can see robust areas between two parameters and performance using charts, but the situation becomes significantly more complicated if we have five parameters.
Recent research has delved into ways to find robust parameter sets across many variables so walk-forward testing can be automated. The solution is in using an n-dimensional map of all the parameters and tying the score to each set of parameters. Next, different parameter neighbors are formed and robust areas are found. Fuzzy logic can be employed to address the issue of oscillation between two sections of the space with similar scores but different trades. This oscillation problem often can make walk-forward optimized perform badly.
Although walk-forward testing has been with us for a couple decades now, the area has fully automated processes, from parameter selection to system modification, have only become possible with recent technological developments. As traders become more familiar and comfortable with these tools, a new era of hands-off trading may emerge where systems are viable for years, not months, and profits can be reliably captured across a decade.
Murray A. Ruggiero Jr. is a consultant. His firm, Ruggiero Associates, develops market timing systems. He is the author of “Cybernetic Trading Strategies” (Wiley). E-mail him at firstname.lastname@example.org.