Once you have built your trading system, you must test it before putting money at risk. CTA Joe Krutsinger, who has developed more than 700 trading systems, put it this way, “Before you can go somewhere, you have to know where you want to go.”
First, with any testing platform, you have to learn the scripting language it uses when you’re inputting the code for your methodology. Next comes data. The type of data you will require to test your trading system depends on your system’s time frame. If your system will trade multiple times a day you will need intraday data, but if you programmed your system to only trade once a day — if it’s a moving average system, for example — end-of-day data will suffice.
Traders should pay attention to how much data is supplied. While TradeStation updates all of its data once you’ve paid for it, TradersStudio comes with data up to the time that you purchased it and then you have to purchase additional data from a third-party such as CSI and plug it into the software.
When testing data, take into account that most platforms use continuous data, says George Pruitt, director of research for Futures Truth Magazine. “Futures have rolling contracts, and because of that you have to create a synthetic continuous contract and that can affect your analysis by showing results that may be better or worse than you would have really gotten. And there’s really no way to adjust for this.”
Knowing how your data providers adjusts continuous prices is important as is avoiding overoptimization, says Pruitt. “If you try trading a system that has been curve-fitted, in real-time it will fail,” Pruitt says.
Curve-fitting is when you optimize your trading system on a specific set of data. “Let’s say you have a moving average system and you want to test it on bonds,” Pruitt says. “You test it with the 19-day moving average crossing the 39-day moving average. Then you re-optimize it and figure out that you get the best results using the 9- and 14-day moving averages. And then you keep re-optimizing and find a crossover that works even better than the 9 and 14. So what you’ve done there is fitted the system to that amount of historical data. The future will not replicate the past exactly. And because you’ve curved your system to that data, your system is doomed to fail,” he says.
The example of using only one parameter of moving averages is just a simple one; most traders use as many as seven parameters and optimize them all.
Also, you should make sure you have enough years of data and that you test in enough different markets and that you use the same parameters in all the tests, says Tim Arnold, president and founder of Trading Blox, a trading system software developer.
“One way to limit curve-fitting is to test your system on out-of-sample data — data outside of the data you’re using for your testing,” says Arnold.
Try to pick a time frame of data to test and then test forward or shift the dates around. For example, if you had data as of 2006 going back to 1967, you could test your system on data from 1967 to 1987, and if the system performed well you could then test it with the next 20 years of data to see if the system is profitable. You can also take different pieces of that data, for instance, test from 1977 to 1987 and then 1985 to 2000. This way if you have adjusted your parameters to work best with only one set of data, when you test with another set you will see that your system would not perform well outside of that initial sampling of data, which means the system has been overoptimized.
A downfall of many platforms is that they do not offer portfolio analysis — you can’t test your system against multiple markets at the same time. If you want to test your system against multiple markets, you have to backtest your system in each market individually. And you would have to do extra work to analyze the results.
TradersStudio, Multi Charts and Trading Blox offer portfolio analysis, but TradeStation does not. However, there is third-party software you can add on to TradeStation for that purpose.
Also, you should not have to tune your system all the time. “Tuning it once a quarter would be fine, but if you need to tune your system everyday other day, it’s probably not going to hold together very well,” Krutsinger says.
You also must take into account commissions and slippage when testing a system. High frequency systems are more sensitive to commissions and slippage, and must perform better just to account for those costs. The more you trade, the more commissions you pay and the greater likelihood of losses due to slippage, particularly if you are entering and exiting on stops.
There are many platforms to choose from when testing your trading system. TradeStation 8 is one of the most popular. Traders Studio is a lesser-known platform that also can be used for testing strategies, but it is not currently able to do intra-day testing; it can only do end-of-day testing. TradeStation can do both.
Multi Charts and trading Blox are other system testing software. MultiCharts is a charting package that provides drawing tools and allows you to create your own custom indicators in a TradeStation EasyLanguage-compatible language. Trading Blox’ high-end software lets you use its scripting language to create your own custom systems.