Optimization: Teaching an old dog new tricks
When optimizing a trading system, you usually can sort the results by net profit, drawdown or any of the standard fields in the summary results. However, although useful, these figures are not always the best measure of performance. Ratios such as net profit/drawdown give a risk-adjusted measure, but what we really want is a measure that predicts system robustness and future performance.
This is why your backtesting platform needs to be able to optimize using custom criteria. A system can be biased by results based on the number of trades. With custom criteria, solutions with only a few trades do not win out. An example of a relative measure that you could use would be “optimal f,” which captures risk, performance and distribution in one number. Here is what the code would look like, using a basic channel breakout logic, for TradersStudio (Ed. Note: a software Murray Ruggiero played a key role in designing):
Dim Hi, Lo
Dim OFactor As Double
Hi = Highest(High,SLen,0) + MinMove
Lo = Lowest(Low,SLen,0) – MinMove
Buy(“ChanBuy”, 1, Hi ,Stop, Day)
Sell(“ChanSell”, 1, Lo , Stop, Day)
If barnumber = lastbar then
OFactor = CalcOptimalf_Sess
This is an example where “optimal f” for the system is calculated on the last bar. It then can be used as a custom optimization factor.
Another solution, coded by private trader Wes Landen, calculates a measure that weights performance over the period used for testing the system. The most recent performance is weighted the most and the weighting is reduced as results move back in time. This favors systems that are doing the best most recently but still have good overall performance.
The calculation for this measure sets the value using several different measures. First, there is a linear factor that establishes a discount value for the oldest data. For example, a value it can be set to start with a weight of 0.7 for the first year and scale up to a weight of 1.0 in the final year. Next, the equity curve is weighted using a Fibonacci sequence — that is, a weight of one for the first year, two for the next year, three for the third year, five for the fourth year, and so on. Finally, the weighted sum of annual values is divided by the maximum drawdown.
Because this methodology weights most recent performance, this type of analysis can produce custom criteria for optimization that is predictive of future performance.
Walk-forward testing is a way to validate an optimized system on data it has never seen without risking it in the actual markets. Here’s how it works. Say you have 12 years of data from 1995 to 2006. Let’s assume that the trading strategy in play needs a minimum of three years of data for testing and optimization.
Start by developing and optimizing the system using only the first three years of data (in this example, 1995-97). On this three years of data, try as many ideas as you like, but do not look at any data after 1997. When you have discovered what you consider a good set of parameters, take those inputs and use them to trade the data starting with 1998. After a short period — say, one month — record the results.
This is your first block of out-of-sample performance.
Now, slide the three-year optimization window forward by one month as well so that you are now optimizing from the start of the second month of 1995 through the first month of 1998. Regenerate new optimal parameters and use those inputs to trade the next block of out-of-sample data. Repeat the analysis until you have walked forward through all of your data.
When your data finally runs out in 2006, go back, and test the system for the entire period from 1998 to 2006. The system performance for the 108 out-of-sample months is a much better indication of how well the system will perform in real time.
There is nothing magic about the assumed periods (here, three years for system development and one month for the walk-forward interval). Picking these two time parameters is a trade-off between optimization time and statistical validity of the results. In practice, using about 20% of the optimization period for the walk-forward window works fairly well.
If the out-of-sample results look good, continue the walk-forward process in real time. If the system holds up, it’s time to consider trading it with real money. However, keep in mind that markets do change fundamentally with time. Even the most robust trading systems, no matter how frequently they are re-optimized eventually break down for extended periods.
There are additional concerns with this approach, however. It’s not always straightforward. To better understand the walk-forward process in general, as well as the potential pitfalls, it’s helpful to look at an actual example. The following walk-forward simulation uses the Custom Report features of TradersStudio, but it could also be replicated manually if necessary.
One problem is when the optimal parameter value changes as the testing moves to a different time window. If a parameter value changes from 20 to 22, it is not much of a problem. But if there are two peaks with one at 20 and the other at 50, and the optimal value changes from 20 to 50 on a single walk-forward step, it’s a potential problem because it sets up the possibility that the trades are different between the two windows. These are called transitional or boundary trades in walk-forward analysis.
There are different solutions to this issue. One is to exit the open trade at the end of the out-of-sample period and enter the open trade that bridges both the next optimization period and the next run period. If we choose to exit trades only when the direction changes, we would create a trade that was entered on an old period, which isn’t workable. Alternatively, we can exit all trades and wait until the first new entry. Another option is to exit and re-enter; this method exits the trade from the run period and enters in whatever direction (the same or different) that the system is in on the first day of the new run period.