Trading system optimization has been a hot topic of discussion since the first commercial trading system programs appeared in the 1980s. Optimization has always been a double-edged sword because while it discovers the best inputs for a specific trading system logic, it also defines what’s best by the past, which may or may not repeat itself. However, in light of new technologies, optimization is finding new roles in a system trader’s toolbox.
The basic idea of classic optimization is simple. In developing a trading system the first step is to develop a valid premise. Then, a set of rules is developed to exploit that premise. Most rules, of course, need inputs (look-back periods, range or volatility multipliers, etc.). Optimization is the process of determining the values to use for the inputs.
Early on — and even today for some — a popular method of optimization was to blindly test every conceivable combination of input values within a defined range, then trade in real time using the set that returned the best hypothetical results in the past. This is a recipe for disaster. Even if a system is based on a sound premise, the best set of parameters over the past 20 years is rarely the best set for any given day over the next 20 years.
IS IT ROBUST?
A better way to use optimization, when it comes to individual system performance, is to use it to judge a trading system’s stability. Consider the results of a basic triple moving average crossover system on a diversified basket of 15 markets tested from Jan. 3, 1980, to Aug. 9, 2007, with $100 deducted per trade for slippage and commission.
For this system, the following inputs will be optimized:
• Short average (SA): lengths from five to 30, in steps of five
• Middle average (MA): 30 to 60, steps of five
• Long average (LA): 60 to 100, steps of five
These input ranges create 378 possible combinations. Let’s look for the valid cases where SA < MA < LA and use a statistical test to judge the robustness of the optimization. This test is simple. All combinations are tested and the average and standard deviation of the summary statistics are calculated:
Average profit: $864,377.95
Drawdown (DD): ($151,278.47)
Standard deviation (SD): $33,175.56
DD + 1 x SD: $750,791.11 ($184,454.03)
DD + 2 x SD: $637,204.27 ($217,629.59)
DD + 3 x SD: $523,617.42 ($250,805.16)
Then, as a further test of the system’s validity, one, two and three times the standard deviation of the different profit values is subtracted from the average profit.
At a minimum, the system should be profitable at twice the standard deviation. Ideally, it should be profitable at three times. This triple moving average crossover system still makes more than $520,000 at 3 x SD, which represents the 99% confidence area.
One way to examine the performance of different input values is to use a surface chart. “On the surface” (below) shows what happens to net profit as we move across different input values for the moving averages. The first chart shows the net profit figures for different values of the short and medium moving averages and the second chart shows the net profit figures for different values of the medium and long moving averages.
The parameter set that produced the most money was SA = 25, MA = 55 and LA = 75. This combination made $1,088,191 with a drawdown of $105,888 in the backtest. There were 28 combinations that produced more than $1 million in the backtest and 165 combinations produced more than $900,000, representing about 43% of the combinations. As a summary, an SA between 20 and 30, an MA between 40 and 50 and an LA between 80 and 95 produces a stable area of the curve based on net profit.
If you were considering this trading system, another set worth a closer look would be 25, 45 and 90. This combination produced $1,013,964 with a drawdown of $124,764. There are other combinations with better profit/drawdown ratios, but the optimization surface is not as smooth. Further, this set of parameters is in the center of our proposed range.
CUSTOM CRITERIA
When optimizing a trading system, you usually can sort the results by net profit, drawdown or any of the standard fields in the summary results. However, although useful, these figures are not always the best measure of performance. Ratios such as net profit/drawdown give a risk-adjusted measure, but what we really want is a measure that predicts system robustness and future performance.
This is why your backtesting platform needs to be able to optimize using custom criteria. A system can be biased by results based on the number of trades. With custom criteria, solutions with only a few trades do not win out. An example of a relative measure that you could use would be “optimal f,” which captures risk, performance and distribution in one number. Here is what the code would look like, using a basic channel breakout logic, for TradersStudio (Ed. Note: a software Murray Ruggiero played a key role in designing):
Sub OptimizeFactorExample(SLen)
Dim MinMove
Dim Hi, Lo
Dim OFactor As Double
MinMove=GetActiveMinMove()
Hi = Highest(High,SLen,0) + MinMove
Lo = Lowest(Low,SLen,0) – MinMove
Buy(“ChanBuy”, 1, Hi ,Stop, Day)
Sell(“ChanSell”, 1, Lo , Stop, Day)
If barnumber = lastbar then
OFactor = CalcOptimalf_Sess
(0.05, 20)
SetOptimizeFactor(OFactor)
End if
End Sub
This is an example where “optimal f” for the system is calculated on the last bar. It then can be used as a custom optimization factor.
Another solution, coded by private trader Wes Landen, calculates a measure that weights performance over the period used for testing the system. The most recent performance is weighted the most and the weighting is reduced as results move back in time. This favors systems that are doing the best most recently but still have good overall performance.
The calculation for this measure sets the value using several different measures. First, there is a linear factor that establishes a discount value for the oldest data. For example, a value it can be set to start with a weight of 0.7 for the first year and scale up to a weight of 1.0 in the final year. Next, the equity curve is weighted using a Fibonacci sequence — that is, a weight of one for the first year, two for the next year, three for the third year, five for the fourth year, and so on. Finally, the weighted sum of annual values is divided by the maximum drawdown.
Because this methodology weights most recent performance, this type of analysis can produce custom criteria for optimization that is predictive of future performance.
WALK-FORWARD ANALYSIS
Walk-forward testing is a way to validate an optimized system on data it has never seen without risking it in the actual markets. Here’s how it works. Say you have 12 years of data from 1995 to 2006. Let’s assume that the trading strategy in play needs a minimum of three years of data for testing and optimization.
Start by developing and optimizing the system using only the first three years of data (in this example, 1995-97). On this three years of data, try as many ideas as you like, but do not look at any data after 1997. When you have discovered what you consider a good set of parameters, take those inputs and use them to trade the data starting with 1998. After a short period — say, one month — record the results.
This is your first block of out-of-sample performance.
Now, slide the three-year optimization window forward by one month as well so that you are now optimizing from the start of the second month of 1995 through the first month of 1998. Regenerate new optimal parameters and use those inputs to trade the next block of out-of-sample data. Repeat the analysis until you have walked forward through all of your data.
When your data finally runs out in 2006, go back, and test the system for the entire period from 1998 to 2006. The system performance for the 108 out-of-sample months is a much better indication of how well the system will perform in real time.
There is nothing magic about the assumed periods (here, three years for system development and one month for the walk-forward interval). Picking these two time parameters is a trade-off between optimization time and statistical validity of the results. In practice, using about 20% of the optimization period for the walk-forward window works fairly well.
If the out-of-sample results look good, continue the walk-forward process in real time. If the system holds up, it’s time to consider trading it with real money. However, keep in mind that markets do change fundamentally with time. Even the most robust trading systems, no matter how frequently they are re-optimized eventually break down for extended periods.
There are additional concerns with this approach, however. It’s not always straightforward. To better understand the walk-forward process in general, as well as the potential pitfalls, it’s helpful to look at an actual example. The following walk-forward simulation uses the Custom Report features of TradersStudio, but it could also be replicated manually if necessary.
One problem is when the optimal parameter value changes as the testing moves to a different time window. If a parameter value changes from 20 to 22, it is not much of a problem. But if there are two peaks with one at 20 and the other at 50, and the optimal value changes from 20 to 50 on a single walk-forward step, it’s a potential problem because it sets up the possibility that the trades are different between the two windows. These are called transitional or boundary trades in walk-forward analysis.
There are different solutions to this issue. One is to exit the open trade at the end of the out-of-sample period and enter the open trade that bridges both the next optimization period and the next run period. If we choose to exit trades only when the direction changes, we would create a trade that was entered on an old period, which isn’t workable. Alternatively, we can exit all trades and wait until the first new entry. Another option is to exit and re-enter; this method exits the trade from the run period and enters in whatever direction (the same or different) that the system is in on the first day of the new run period.
We can demonstrate this using the triple moving average crossover system. Instead of optimizing using net profit or drawdown, we will use the weighted equity function. We will weight starting at 0.7 for the oldest data and 1.0 for the most recent. We will use TradersStudio’s walk-forward optimizer on a basket that includes the markets shown in “Our markets” (below). This optimizer can only use overlapping date ranges. Because natural gas started trading in 1991 that is where we will start. We will end on Aug. 9, 2007. The training window will be 2,000 bars and a re-optimization will be performed every 500 bars, using the following optimization ranges:
• SA: 15 to 30, steps of five
• MA: 35 to 60, steps of five
• LA: 65 to 90, steps of five
Our first out-of-sample period starts on June 3, 1999. Let’s look at the parameters and how they change over time walking forward (see “C-C-C-Changes,” below). The two out-of-sample periods that lost money have been tough on trend-following methodologies. Using the classic net profit function for walk-forward optimization often yields poor walk-forward results because it is really out-of-sample testing.
Here, the results are good for our triple moving average crossover system out-of-sample more than eight years of out-of-sample results from June 3, 1999, to Aug. 9, 2007.
This performed better than most of the fully in-sample optimizations performed during this period and shows that this weighted equity curve as a criterion for optimization shows great promise.
As computers get faster and backtesting software advances, we will see a new evolution from static trading systems and simple optimization to more advanced methods of optimization that try to predict future performance based on the in-sample period. Walk-forward testing with equity curve analysis is a starting point in that journey.
Murray A. Ruggiero Jr. is a consultant in East Haven, Conn. His firm, Ruggiero Associates develops market-timing systems. He is vice president of research and development for TradersStudio and author of Cybernetic Trading Strategies (John Wiley & Sons). E-mail him at ruggieroassoc@aol.com.
