Automated trading systems, especially those based on technical indicators, are all the rage. Such systems are highly dependent on settings, such as moving average periods, which are usually determined in unsystematic fashion. Some are set by tradition, such as only going long when an index is above a 200-day moving average. Others are found by brute-force backtesting, generally by what is called “single-factor design,” in which each setting is tested on its own.
All of these methods fail to capture the interconnectedness of the settings, and are not likely to find something optimal. Indeed, these methods usually sift through many attempts before finding one that generates any profit at all.
One systematic method that avoids these problems is the grid search method. However, this is not generally used, and for good reason. For the five-factor model, discussed here, computer runs covering the parameter space could run into the millions. For some types of trading systems, this might take weeks or even months for a single investigation.
The Design of Experiments (DOE) method solves all of these problems.
APPLYING THE METHOD
To implement the DOE method, first, a set of points in the parameter space is selected based on a statistical criterion — 56 points plus six additional ones for the experiment discussed here. Runs are conducted at each of those set of points and an outcome, profit/loss in this case, is recorded.
Then, a polynomial function is fit to the outcome data; the trial points are the independent variables, and the outcome is the dependent variable. This allows us to express the outcome, the profit/loss, as a simple calculation throughout the entire parameter space called a response surface. Finally, one of several types of hill-climbing methods can be applied to the response surface to find the “sweet spot” — the optimal parameter settings.
For our problem here, a single run on 20 years of historical data takes 0.25 seconds to run on a typical PC. A grid search on the entire parameter space with resolution of one decimal place would require 43 million runs, which at a quarter of a second each comes to 17.7 weeks. On the other hand, for the DOE optimization, the 62 points in the parameter space require 12.84 seconds, and the hill-climbing for the sweet spot is another 45 seconds. This 58-second investment of time is a lot more efficient than one taking 17.7 weeks.
Not only does the DOE approach make it possible to find optimal settings quickly and easily, but it finds those that take into account the relationships among the parameter settings in producing the outcome. This dramatic increase in efficiency also has another important benefit — researchers can test modifications to the structure of the trading system itself. For example, should the system employ a simple average, a moving average or an exponential moving average? A conclusion to that question can be found directly by repeating the trial runs and the hill-climbing for each method. In the case of the problem we’re discussing here, the total would be three minutes.
The polynomial model used for the problem in the paper is a cubic. The cubic response surface has a greater ability to fit observed outcomes, but it also contains a lot of terms relative to the number of factors that require many trial runs. For a problem with run times of a second or two, this isn’t difficult. For trading systems with longer run times — for example, those of an hour or so — a simpler polynomial model, such as a quadratic or an interactive (a polynomial without the power terms), can be used that requires fewer trial runs. A comprehensive grid search for one-hour trial runs is completely out of the question, but a quadratic model with 15 one-hour trial runs is feasible.
THE VIX RSI STRATEGY
On page 96 of “Short Term Trading Strategies That Work,” Connors and Alvarez describe a trading system for the SPY:
1. The S&P 500 (SPY) is above its 200-period moving average.
2. The two-period relative strength index (RSI) of the Volatility Index (Vix) is greater than 90.
3. Today’s Vix open is greater than yesterday’s close.
4. The two-period RSI of the SPY is below 30.
5. Buy on the close.
6. Exit when the two-period RSI of the SPY closes above 65.
The motivation for this system is to use the Vix to detect a weak and oversold market. In a backtest on three-and-a-half years of data, the authors found 92 signals with an average hold under five days and a frequency of success of 79.35%.
DOE ANALYSIS
The trading system above contains five factors:
1. Period of the SPY moving average
2. RSI setting of the Vix
3. RSI setting of the SPY
4. Exit setting of the SPY RSI
5. Period of the RSI
To define the parameter space, we choose minimums and maximums for the first four factors:
1. [50 – 250]
2. [80 – 100]
3. [15 – 45]
4. [35 – 65]
The fifth factor is a discrete integer that will be defined to take on the values of 2, 3, 4, 5 and 6. The first factor is also a discrete integer, but its large interval allows us to treat it as continuous. Any value we find will be truncated to an integer without serious effect to the investigation.
Using the Gosset computer program described by Hardin and Sloane (see “Further reading”), we generate parameter settings for 56 trial runs. Daily data for the SPY and the Vix from Jan. 29, 1993, through Oct. 17, 2003, were selected for the trial runs. A computer program written in the GAUSS programming language (www.aptech.com) was developed for the simulation of the trading system given the parameters.
To produce a more realistic result, each simulation started with an account of $500,000, paid a $20 transaction cost per trade, and a bid/ask spread was also included to simulate slippage.
For hill-climbing, the Sqpsolvemt program in the GAUSS Run-Time Module was used. Sqpsolvemt uses a nonlinear sequential quadratic programming method allowing us to constrain the search to the parameter space defined by the minima and maxima. It doesn’t handle integers, however, and thus a separate hill-climb will be conducted within each value of Factor 5, and the sweet spot will be the best result among those separate runs.
The 56 trials for the cubic polynomial model provide no degrees of freedom, and an initial run found sweet spots with large error deviations. To gain degrees of freedom and a better fit to the observations, a center point was added, and the sweet spots from the initial run were also added for a total of 62 trials. The results for each value of Factor 5 are shown in “By a factor of 5."
Rather than rely on the backtesting for our choice of sweet spot, we will forward test each of them in another data set, the SPY and Vix from Oct. 17, 2003, through Oct. 12, 2009. The results for these runs are shown in “Forward thinker”.
Opinions might reasonably differ here. One the one hand, the RSI period of four clearly wins the profit contest. On the other, the greater theoretical prediction for a period of three might suggest that it would be superior in the long run. None of these sweet spots generates much of an annualized rate of return, but to be fair the forward test includes the 2008 market meltdown.
The forward test for the original settings in Connors and Alvarez (see “Further reading”) are:
Annualized Sharpe Maximum
Profit/Loss Return Ratio Drawdown
28329.22 0.93 0.5819 -20434.68 |
The DOE method has succeeded in finding settings that produce from 3.5-1 to 4.5-1 greater profit, and more than three times the annualized return, compared to the original input values recommended by the trading system authors.
FINDINGS
There aren’t enough degrees of freedom in the cubic polynomial regression model for very much statistical reliability. However, some tentative findings are possible from the t-statistics of the coefficients. First, the results don’t seem to be very sensitive to the period of the SPY moving average. In fact, a period of 200 for the sweet spot for the RSI period of three produces results in the forward test that are comparable:
Annualized Sharpe Maximum
Factor 5 Profit Return Ratio Drawdown
3 131987.39 4% 1.6738 -25140.78
4 130783.56 3.96% 1.2546 -24001.11 |
While the DOE results suggest the longer the period the better, a 250-day moving average is a full year’s worth of data and longer periods may be impractical in some situations. It does appear that the traditional period of 200, while still long, should do quite well in practice.
Connors and Alvarez strongly hold to an RSI period of two in contrast to the traditional period of 14. The DOE results first support their contention that a period of 14 is too long. On the other hand, the results also suggest that a modification to a period of three or four could produce significantly greater profits.
The regression analysis also suggests that the RSI period (Factor 5) and the Vix RSI setting (Factor 2) are interrelated. A future study might produce better results that incorporated separate RSI periods for the Vix vs. the SPY.
Further reading:
• Connors, L. and C. Alvarez, “Short Term Trading Strategies That Work,” TradingMarkets Publishing Group, 2009
• R. H. Hardin and N. J. A. Sloane, “A New Approach to the Construction of Optimal Designs,” Journal of Statistical Planning and Inference
Ronald Schoenberg is a partner and research manager at Trading Desk Strategies LLC. E-mail him at ronschoenberg@optionbots.com or see www.optionbots.com.