*“In July 2002, the [Dow Jones] index recorded three steep falls within several trading days. (Probability: one in four trillion) And on October 19, 1987, the worst day of trading in at least a century, the index fell 29.2%. The probability of that happening, based on the standard reckoning of financial theorists, was less than one in 10 to the 50th –odds so small they have no meaning. It is a number outside the scale of nature….Everyone knows [that] financial markets are risky. But in the careful study of that concept, risk, lies the knowledge of our world and hope of a quantitative control over it. ” *

* — Benoit Mandelbrot*

Understanding risk in the field of finance and investments is of paramount importance. Investors need to know the risk associated with various investments in order to understand the likelihood of future events (losses in particular) and to plan accordingly as well as analyze competing investments with different risk-return profiles on equal grounds.

To this end, financial practitioners and academics have developed statistical approaches to model risk based on an asset’s past price action. Foremost among them is the use of standard deviation as a proxy for an investment’s risk. To be specific, standard deviation of the historical returns of an investment is used to define "risk" in many cases. Consider for example the fact that the Sharpe ratio, which adjusts the expected return of an asset by its risk, uses the standard deviation of returns as the risk metric. Another instance can be found in modern portfolio theory and the efficient frontier problem. The goal is to find the asset combinations (i.e., portfolio weightings) that yield the least amount of risk for given expected returns–risk again being defined as the standard deviation of the total portfolio’s returns.

In a strict statistical sense standard deviation measures how much random outcomes differ from their average. So for instance, a never-ending series of returns of 2%, 2%, 2%, .… 2% has an average of 2% and a standard deviation of 0 (because there is never any variation from the mean). In exactly the same way a never-ending series of returns of -2%, -2%, -2%, .…, -2% also has a standard deviation of 0 because there is never any variation about the mean, which is -2%. On the other hand, the return stream 0%, 4%, 0%, 4%, .…, 0%, 4% also has an average of 2% but a non-zero standard deviation (of 2.0) because there is some variation from this mean. The stream 1%, 3%, 1%, 3%, …., 1%, 3% again has a mean of 2%, but a smaller standard deviation (of 1.0) because the individual outcomes do not vary from the 2% mean as much as the 0%, 4% series. Thus we can see that standard deviation is really just a measure of the amount of "spread" associated with random outcomes."

But is standard deviation really a "good" measure of financial risk? We begin to explore this question, but first we must ask ourselves what is "risk" in the world of finance? For the purposes of this piece, we define financial risk (otherwise “risk”) as follows: Financial risk is defined as the probability of losing a certain amount of money over a given time frame.

With risk defined we can move to a systematic study of how well standard deviation has predicted it in the past. We will describe the experimental procedures and tests conducted, provide the results of the baseline tests and include an extended analysis where the baseline study is broken into separate time frames.

**The study**The experimental approach followed in this study is based on determining how accurately standard deviation has projected risk in the past. Since we have defined risk as a probability of losing a certain amount (say X) over a given time frame (say Y), the test for accuracy is straightforward. We can predict the probability associated with a given loss and time frame based on an asset’s historical standard deviation. Then going forward we can determine if such losses occurred as frequently as predicted.

This is analogous to looking back at all the days the local weatherman predicted a 10% chance of rain with how often it actually rained that next day. Suppose that in giving tomorrow’s forecast he predicted a 10% chance of rain 200 times over the past five years. Now imagine it actually rained 20 of those days -10% of the time. In this case our weatherman has been historically perfectly accurate. If instead it rained 30 days, 15%, then he was inaccurate. We might also say that in this case he was not conservative because the actual number of occurrences of rain over the predicted would be 1.5 (i.e., 30 over 20). If it actually rained five days, he would be inaccurate, but conservative because the ratio of actual to predicted occurrences would now be 0.25.

In our study, standard deviation is like the weatherman–but instead of forecasting the chance of rain it might forecast that shares of Coca-Cola (KO) have a 5.6% chance of losing 20% in three months, or that gold futures have a 22% chance of losing 10% over the next month. The goal is to see how accurate these forecasts have been. Does standard deviation tend to over-or-under-predict the chance of loss?

To find out, 49 markets were selected for the study, encompassing equities, fixed income, emerging markets, commodities, hedge funds and managed futures asset classes.

Historical monthly return data was used to estimate an asset’s standard deviation, which then was used to project the probability of experiencing certain loss levels at given time horizons (see "Analysis procedure details," here).

This was compared to actual losses experienced on a forward rolling basis to determine how accurately standard deviation has predicted risk. Loss levels of 10%, 20%, 30%, 40% and 50% at time horizons of 1, 3, 6, 9, and 12 months were investigated.