Most new traders take up mechanical trading with one thought in mind: to historically test their trading idea or strategy in order to “prove” it works. Most trading software is built around this objective. Simply pull up a chart, insert a strategy, quickly execute hundreds, thousands or millions of optimization runs and review the best results. The best parameter set is then used in live trading.
Alas, if only the process were so easy! The reality is that the simple approach—the method that most software encourages—is also the method least likely to work. The old computer adage “garbage in, garbage out” certainly applies here. Traders who have a “garbage” development process typically will receive “garbage” from the testing software—a strategy that performs poorly in real time.
The solution is to build a strategy with the right process. But what is right, and what is wrong? This article examines some different development decisions that need to be made, and can be a guide to some of the best development practices.
Start with an idea
Regardless of how a strategy is evaluated, they all start with an idea. That is where we will begin. Just as there are good and bad ways to test, there are also good and bad sources of trading ideas as shown in “Strategy source” (below). Here we highlight a few.
Books/magazines/web sites: There are literally thousands of public domain sources for trading ideas. These could be fully defined strategies, or just strategy components, such as entries and exits. The problem is that these public domain sources rarely work well as is. Think about it—why would someone knowingly disclose a superb trading system or idea, instead of keeping it secret and just trading it for themselves? Most sane traders would keep it secret. This means that most of the public domain ideas will not perform well when tested. But, that does not mean they are worthless.
Public sources of ideas can be a terrific place to start a trading strategy; or as a source for a value added overlay to apply to an already viable strategy. Simply taking an idea/code and modifying it to make it unique is a common approach used by experienced developers. They rely on the public domain for the seed idea, but the trader then takes the idea and grows it.
Random ideas: There are a few software products out there that will randomly take trading ideas and test them out, via machine learning, artificial intelligence or genetic optimization. Typically, millions of different trading strategies are created, and the best performing one is selected. While on the surface this seems reasonable, the same issues of over-optimization that plague a multi-variable system are likely to impact a randomly-generated strategy. If a few million trading strategies are generated, chances are at least some will perform well—and that could all be because of chance. While it is possible to randomly discover an edge, it is far easier to randomly discover a fluke strategy that just happens to test well on a specific set of data.
Trader experience: An excellent source of trading ideas comes from a trader’s own experience. What could be better than continually observing some market pattern and then turning it into a successful strategy? While this sounds great in theory, reality is a little tougher. What frequently looks like predictable behavior on a chart or two turns out to be a short-lived anomaly. Of course, rigorous testing will reveal that, but many traders develop an idea based on a very limited sample of charts, quickly evaluate it and then begin to trade it. This almost never works. Good traders, though, will combine their experience with proper testing, and frequently can build good trading strategies because of it.
From idea to strategy
Once the idea is formed, most traders typically will do one of two things with it: Keep the number of rules and conditions small, or add a multitude of filters, conditions and rules to entries and exits to improve the strategy performance. Both approaches have their pros and cons.
Very few rules: Many traders espouse the theory that “simplest works best.” This definitely can be true in real-time trading.
The problem is that simple trading ideas yield imperfect results—large and significant drawdowns and periods of flat and generally unremarkable performance are the norm, such as the results for a simple strategy shown in “Flat times” (below).
Paranoid traders worry about someone else determining their simple rules, and because thousands of people are out there every day building systems, it makes sense that the simpler ideas will be discovered first. Quite a few traders dismiss the simple ideas in favor of strategies with many rules and conditions.
Lots of rules: The flip side to minimal rule strategies is a strategy with dozens of rules, parameters and conditions. Typically, these systems will be adjusted every time a big loss is encountered—in effect, engineering losses out of the backtest. Equity curves produced with complicated systems typically have a very appealing historical backtest. The problem with such strategies, obviously, is that markets change; future losses likely will be slightly different than historical losses, and any attempt to circumvent these losses will only yield a more unrealistic backtest. Future performance usually suffers as more rules are added, which many find hard to accept.
Period to test
Once the core strategy is established, traders have to decide what time period to test it on. While some of this decision will be influenced by the testing method, there are always a few key points to consider:
- More history is better than less history. Given a choice between historical testing for one year and 10 years, the choice should be clear. More history means more trades, and more unique market conditions and phases to evaluate the strategy. (Veteran trader Bill Eckhardt requires a minimum of 1,800 examples of a trade set-up, preferring 15,000, before taking a trade).
- Finding strategies that work over five to 10 years of data is much more difficult than finding a one-month super performer. Traders need to resist the urge to go with the easy testing and instead test on more data.
- Most data providers do not provide unlimited tick data history. Many vendors only provide six months of tick data; this rules out a long backtest. Therefore, tick data should be used with extreme caution, or ideally not at all.
- Other than the limited amount of tick data available, historical tick data is sometimes filtered and aggregated by the exchange or data provider. This could lead to misleading historical results, as the data today might be significantly different than data of a few years ago.
On to testing
The type of testing performed can have a serious impact on creating good strategies. There are roughly four major types of testing methodology:
Traditional backtest: Most trading software promotes a traditional backtest in which all the data is analyzed and rules are optimized based on the whole dataset. The problem is that, just like with randomly generated trading ideas, a few iterations out of a few thousand will always look superb. But that does not mean that the strategy will actually work well going forward.
Out-of-sample testing: This approach takes a traditional backtest, optimizes more than 80% of the data (“in sample”) and evaluates 20% (“out of sample”) based on the best optimization results of 80% of the data. This is much better than traditional optimizing, although the tendency to test and retest still exists, which can easily transform the out-of-sample period into an in-sample period. Also, the parameters would never change with time or market conditions. This can be a problem that walk-forward testing can remedy.
Walk-forward testing: Considered the best test method by many, walk-forward testing combines multiple out-of-sample historical periods to create a longer-term, out-of-sample performance curve. In this respect, it is clearly better than a single out-of-sample test. But this method is more complicated and tougher for most traders to understand. Therefore, many shy away from it. Many experienced traders, though, agree that walk-forward testing is the next best thing to live trading.
Live testing: Every trader should agree that live testing is the ultimate judge of a strategy’s viability. Historical performance is nice to have, but live trading in a real account is the only absolute “truth” in strategy testing. In fact, more than a few traders only test in this manner—they evaluate strategies with real money, in real trading. This can be expensive, of course, especially for novice traders without any market knowledge or experience, as they test multiple bad ideas with real money. Veterans are the group of traders most likely to benefit from this test method.
Although algorithmic and mechanical testing seem straightforward, there are many nuances and pitfalls to this type of trading. Traders would be well advised to research the whole field of algorithmic trading carefully before embarking on strategy development. Mistakes avoided because of a good development process can and will save thousands down the road.