Forecasting with caliber and affinity

July 29, 2016 01:00 PM

The movement of stocks in the New York Stock Exchange (NYSE) is similar to birds flying in one of those spectacular aerial formations that are constantly shifting and evolving in unpredictable ways. As traders, we might look up at this display and wonder if there is some way we could use the past behavior of the whole flock to help us forecast the movements of a particular bird. Approaching this problem with novel mathematics gives us a chance to leverage our past knowledge of all stocks to forecast the behavior of a single stock. The concept behind this method was based on pure mathematics and led to an idea for analyzing data that was so strange and outside the normal range that is was initially rejected. How could it be programmed? 

The idea festered, so I wrote the program and tested it on very small array of numbers.  The results revealed something about how the rows of the array were relating to one another. What would happen if the rows were the daily returns of the stocks in the NYSE? After years of experiments, this new approach boiled down to working with two new indicators. The first one, called “Caliber,” is a way of summarizing the overall behavior of the NYSE using a single number. The second number, “Affinity” of a stock, appears to be connected with the future returns of the stock as well. Let’s first have a look at what Caliber can tell us about the market.

Caliber: How laid back is the market?

As you watch a flock of birds, it’s easy to make a rough estimate of how tightly or loosely the birds are flying in formation. Caliber describes how loosely stocks are connected to one another. The rule of thumb is that the higher the Caliber the looser the formation. More formally, Caliber is a new mathematical measure of how independently stocks are moving relative to one another, which is not affected by the trends of individual stocks. The 3,000 or so stocks in the NYSE are of course highly dependent. Caliber tries to sift through these dependencies to estimate how many truly independent factors are driving the market. For example, a Caliber of 500 would mean that the NYSE is behaving about as freely as a market comprised of 500 completely independent stocks. “Birds of a feather,” (below) shows that the biggest drops in the market often coincided with periods when the Caliber was low, which relates to periods when stocks were moving in a tighter formation. 

The chart is not about forecasting. This graph simply suggests that periods where stocks were moving together tended to coincide with periods when the market was falling. In terms of our stocks-as-birds metaphor, NYSE stocks tend to drop in tight formation and rise in loose formation. This relationship is clear in the spectacular drop in the Caliber of the NYSE in the three-year period leading up to the crash of 2008-2009 (see “Birds on a wing,” below). 

What it is 

Caliber is a measure of the diversity of a market. To calculate the Caliber, all of the daily returns for each of the approximately 3,000 stocks in the NYSE for some time period, usually 60 days, are inputted. The resulting matrix has roughly 3,000 rows and 60 columns, which can be row-reduced to 60 non-zero rows and 2,400 rows of zeros. Since we would obtain the same result no matter how the market is behaving, this approach is not useful. However, using a probabilistic approach for this reduction, the more dissimilar the rows, the more operations it takes to completely reduce this matrix, which then translates into a higher Caliber. The number dubbed the Affinity of a stock simply measures how quickly a stock is cancelled on average during this process. In other words, higher Affinity stocks are cancelled the fastest because their behavior is the most representative of all the stocks in the exchange. Let’s now consider how Affinity might be used as an indicator to forecast returns. 

Affinity: Is your stock in the winner's circle?

If a stock has a high Affinity number, then its movements are more consistent or consonant with other stocks on the exchange. High Affinity stocks can be thought of as the more typical or mainstream stocks and testing suggests they tend to do better on average over long periods of time. Every week we calculate and share a list of the Affinity ranking of all stocks listed on the NYSE. Over the last 25 years, stocks with a higher weekly ranking had better returns on average (see “Flying straight,” below). To see this small effect, I had to average a large number of future returns to smooth out the enormous variation in returns which, of course, are often negative even for the highest ranked stocks.

If we use the actual Affinity numbers of stocks instead of their weekly ranking, we get even stronger evidence that high Affinity stocks have better performance (see “Flying North,” below). Once again, the enormous amount of averaging is hiding the huge variation in returns. 

Further improvement can be achieved in forecasting returns if we use 120 days of prior data instead of the 60 days (see “Spreading our wings,” below). With 120 days of prior data, the relationship between Affinity and future returns is easier to see because we only have to average 1,000 future returns to generate the large peak that appears at the right of the graph. 

A better way to evaluate stocks?

Some of the most highly regarded stocks also happen to be the most expensive so there does appear to be some connection between a stock’s perceived value and how much it costs. It’s natural to ask what exactly these more expensive stocks are offering us since, historically; they haven’t given terrifically better returns than many lower-priced stocks. A partial answer might be that the more expensive stocks might often, but not always, have higher Affinities because higher Affinity stocks tend to be more expensive (see “Big birds,” below).  This observation, coupled with our forecasting evidence, suggests that the Affinity of a stock might be a better indicator of a stock’s value to investors than the actual price of the stock. If you are nervous about the market, you might start looking at a fund holding low Affinity stocks.

Somebody get me a supercomputer

Our research shows that when stocks move in tight formation, there is a higher likelihood the market will turn lower. And when there is greater diversity of returns the market generally performs better. Caliber and Affinity are new ways to define relationships among stocks. Specific trading systems based on Caliber and Affinity have not yet been developed to back up our analysis, but the evidence given above shows these indicators hold value. Additional research on other markets would be useful in confirming and expanding these results but to calculate these indicators weekly on NYSE stocks requires 14 hours of computer time.

The complexity of the algorithm prevents us from sharing all the calculations but the Caliber and Affinity indicators are calculated weekly and posted online open to everyone at

Finally, there are intriguing applications of the Caliber and Affinity algorithms that can be used across various markets. This is just the beginning, smart traders will find new ways to utilize these indicators.

About the Author

Bill Ralph, PhD, is a Professor of Mathematics at Brock University in Canada and an artist. He works on extracting and visualizing meaningful information hidden in noisy data. He can be reached at