.jpg)
Uses of Backtesting
In investing, backtesting refers to an analysis of the historic performance or one or more assets or securities. There is much confusion surrounding how useful backtesting can be to investors, with some claiming that it is is 'worthless' while others appear to believe that it can be used to predict the future with precision. In this post, I address how backtesting can provide value to investors along with its limitations.
There are many ways that backtesting can manifest itself. One is simply by examining the historic returns of an asset. Another is to examine the performance of one asset relative to another asset (e.g., how stocks performed relative to bonds, real estate, gold) or to a metric perceived to be relevant to the asset's performance (e.g., valuations). Yet another is to rearrange pieces of historic to conduct analyses of what the future might look like (e.g., Monte Carlo simulations). And there are other ways that backtesting can be performed.
Broadly speaking, it's very difficult, so much as to be almost impossible, to determine one's asset allocation (i.e., what proportion of one's portfolio to invest in multiple assets) without any reference at all to historic data. Virtually no one is completely ignoring the historic performance of broad asset classes like stocks and bonds, for instance. At a bare minimum, most investors assume that some assets, such as bonds, will be less volatile than some other assets, such as stocks. And this assumption is based on historic data (i.e., backtesting).
Many readers of this post are familiar with the idea of safe withdrawal rates, which I'll discuss at length in future posts. Such readers should recall that the entire body of safe withdrawal rate research is based in some way on backtested data. Apart from backtesting, I believe that it's nearly impossible to determine how much one can safely withdraw from one's portfolio in retirement (unless it is wholly invested in something like a TIPS ladder). Even the assumption of 0% real returns that some purport to use in their withdrawal plan is likely based in part on historic data.
That said, we should all know that the future never looks just like the past. As Mark Twain said, "History never repeats itself, but it does often rhyme." Keeping this in mind, backtesting can be useful as a rough guide of what the future is most likely to be, but the fact that there are literally millions (if not billions or more) of factors at work in the markets and a hefty element of randomness means that backtesting is, at best, only a somewhat blunt method and certainly not a means of predicting the future with precision.
Backtesting is an excellent means of examining the plausibility of an investment strategy. For instance, many believe in something referred to as the 'rebalancing bonus', the idea that rebalancing one's portfolio between something like stocks and bonds will lead to higher long-term returns than not rebalancing the portfolio. But data from the last 50 years has shown that annually rebalancing a portfolio with 60% in stocks and 40% in bonds led to slightly lower returns than not rebalancing the portfolio. This does not disprove the idea of the 'rebalancing bonus', per se, but it casts substantial doubt over its validity.
All too often, backtesting is done with data from only a relatively narrow span of time. I've often seen investors examine data from a single year to try to draw meaningful conclusions about some aspect of investing. As investment educator Paul Merriman has said, "A year in the markets is just noise." In the world of investments, even a decade is a fairly short period of time. Between all the variables at work and a strong element of at least short-term randomness, it's inappropriate to examine only a brief period of time for almost any investment purpose.
However, this does not mean that short-term events of the past cannot be instructive in some way to investors. For instance, the period from 2000-2009 saw U.S. stocks lag significantly behind inflation of the same period. Though this was 'only' a decade, it was a very tough one for many investors, especially the 2007-2009 period. Investors today should be cognizant of the reality that such a decline could certainly happen again, and many, including me, believe that it's practically assured that such a decline will happen again at some point. Another meaningful conclusion that can be drawn from such a period is that an asset class that has performed well over the long-term may perform very poorly over the short-term, including a decade or longer, and one must be prepared in some way to deal with such poor performance. Putting too much emphasis on a single metric, like long-term returns, of an asset or portfolio can lead investors to make decisions that aren't appropriate for them.
'Cherry picking' is another common problem with backtesting and generally refers to overreliance on a specific starting date to draw an inappropriate conclusion. For instance, some financial journalists have pointed out that 30 year U.S. Treasuries bought in 1982 outperformed U.S. stocks over the next 30 years (i.e., through 2011). While this was true, it absolutely reeked of cherry picking because this event (i.e., 30 year U.S. Treasuries outperforming U.S. stocks over a 30 year period) has only happened twice in over 150 years. As such, it should be treated as being a historical oddity and certainly not something reliable enough for an investor to consider as a likely outcome.
It's not always easy to avoid cherry picking and the 'start date problem', but I believe that Tyler's heat map chart at Portfolio Charts does this better than any other tool I've seen. It visually depicts the returns of an asset or a portfolio over different periods of time with different starting dates. In addition to going a long way toward overcoming cherry picking, it also helps investors to see how stable the returns of an asset or portfolio have been over time, both the short-term and the long-term.
Backtesting can be of great value to investors, and I personally believe that most investors do not have as well-grounded a view of the historic performance of their chosen investment strategy as they should. But we must always remember the limitations of applying historic data in any form to learn about the possibilities of the future, paying particular attention to the fact that the past does not enable us to predict the future.
