Can your system do the walk?
There are many ways to build a bad trading system, but two of the more common are the lack of a valid premise and the selection of non-robust parameters. To overcome these issues, developers often split their data into two sets: the development, or in-sample, set and the test, or out-of-sample, set. A common split is using 80% of the data for development and 20% for validation.
However, this split often leaves much to be desired, particularly for systems that trade infrequently, such as trend-following systems. Often, 20% of the data provides too few trades for a statistically significant test. Also, even with a development set and a test set, the process can become contaminated after just a few modifications and out-of-sample tests. Each time the testing process is repeated with a different system, degrees of freedom are reduced. If enough testing is done using the data, the out-of-sample data effectively becomes in-sample data, and the advantage of this method of testing is lost.
A better approach is called walk-forward analysis, originally discussed by trading system developer Robert Pardo, author of “Design, Testing and Optimization of Trading Systems” (John Wiley & Sons, 1992). This technique also performs optimization on a training set, but it uses a much smaller portion of the in-sample data. It then tests the system on the following block of data. After the test, it shifts the development set forward by just a portion — say, one month for a test period of a year. The optimization/test then repeats. Because walk-forward analysis uses multiple training and testing periods, it is less likely to contaminate the data.
However, one issue with walk-forward testing is it can be labor intensive. That’s where trading technology comes into play.
A more detailed example of walk-forward analysis can better demonstrate the challenges and benefits of this approach. Walk-forward testing works like this. Say that you have 12 years of data extending from 1998 to 2009. Assume that your trading strategy needs a minimum of three years of data for testing and optimization.
Start by developing and optimizing the system using only the first three years of data — in this example, 1998 to 2000. On these three years of data, try as many ideas as you like and optimize parameters in as many ways as you can. Under no circumstances do you look at the data after 2000. When you are done with development, record the rules for the system with the optimum parameters. These rules and optimized parameters will be used later for the final testing with new data starting with 2000.
Slide the three-year time window of data forward — say, one month. Now, your dataset runs from the second month of 1998 to the end of the first month of 2000. Repeat the analysis, including optimization, and record the rules and optimized parameters. In the final pass, these parameters will be used for trading the second month of 2000.
Continue walking forward and optimizing the three-year data periods. Record the results for use in the first month following the three-year optimization period. When you reach the end of your data, go back and test the system for the entire period from 2000 to 2009. Switch the rules and parameters each month to the ones that you found and recorded. In effect, you are performing a new out-of-sample test for each month. The system performance for these nine out-of-sample years (108 out-of-sample months) is a better indication of how a system will perform in real time than the performance of a single out-of-sample test. If the results for the out-of-sample months look good, continue the walk-forward process in real time before risking real money on the system.
If we want to do true walk forward testing, a few issues need to be addressed. You must determine how you will change the parameters used in the out-of-sample walk-forward period. If a parameter value changes from 20 to 22, this isn’t an issue; however, if there are two peaks with one at 20 and the other at 50, a decision must be made regarding which value to use.
Another problem is what is a transitional trade, also called boundary trades. If both the old and new parameters would be trading in the same direction at a transition, there is no issue, but this isn’t always the case. We can handle these in three different ways.
One way is to exit the boundary trade of the old period and enter the trade of the new period right away. You cannot exit the trade from the old period if the trade from the new period is in the same direction. You can exit the trade from the old period and wait to the first new entry in the new period if they are in different directions between the periods.
Consider a simple example. We will use natural gas from 1991 to Nov. 25, 2009. We will use a triple moving average crossover system and $100 for slippage and commission. We are using the following parameters in our walk-forward analysis:
Short moving average = 5 to 20, steps of 5
Medium moving average = 20 to 60, steps of 10
Long moving average = 60 to 100, steps of 20
We used a training set of 750 bars and an out-of-sample period (run period) of 175 bars (see “Walk-forward report”).
The report shows each training set and run set. The criterion column shows the net profit/drawdown ratio for the training set. We can assume that sometimes the best set of parameters will stay constant over time. We see the short and medium moving average are the same from July 13, 2001, to Oct. 5, 2007, while the long-term average during this period oscillates from 60 to 100. We can see that 13 of 23 out-of-sample run windows are profitable. We also had three of them in a row lose money in the Oct. 31, 2002, to Dec. 13, 2004, window. Considering this is a trend following system, our results are reasonable.