Skill or luck? How alpha, the t-stat and the p-value test an investing record

What does "skill or luck" mean here?

Beating the market in a good year proves little: plenty of random portfolios do. The real question is whether a record beat its benchmark by more than chance would usually produce, once you allow for how much it simply moved with the market. Luck Filter answers that with the same test researchers use on fund managers, run on your own daily returns.

How does Luck Filter measure alpha and beta?

It fits one line through every trading day of your record:

your return − cash rate = alpha + beta × (benchmark return − cash rate) + noise

• Beta is how much of your daily movement is the market's. A beta of 1.2 means that on a day the benchmark rose 1%, your account tended to rise about 1.2%.
• Alpha is what is left: the average daily return the market's moves do not explain, shown as a yearly figure. Positive alpha means you did better than your market exposure alone would have.
• The cash rate is what money earned sitting safe: the 3-month US Treasury bill for dollar accounts, and the matching overnight or 3-month rate for pounds, euros and Canadian dollars.

Taking beta out matters. An account that holds twice the market (with margin or options) will beat the market in a rising year without any skill. The regression charges it for that extra exposure, so only the part beyond it counts as alpha.

What is the t-stat?

Alpha divided by its own uncertainty. Daily returns are noisy, so any alpha comes with a margin of error. A t-stat of 2 means the alpha is twice as large as that margin. Around 2 (or −2) is the usual line for a result that is unlikely to be noise alone.

Luck Filter works out the margin of error with the Newey-West method, which allows for returns that run in streaks or swing more at some times than others. Trading records, and options in particular, often do both, and the simpler formula would make the margin look smaller than it is.

What does the p-value mean?

If there were no skill at all (a true alpha of zero), the p-value is how often a record would show an alpha at least this far from zero, in either direction, by chance alone. A p-value of 0.05 means 1 in 20 such records. The smaller it is, the harder the result is to put down to luck.

It is not the chance that you were lucky. A p-value of 0.05 does not mean a 5% chance of luck and a 95% chance of skill. That second question needs to know how common skill is to begin with, which is what the next number is for.

What does the luck cloud show?

It draws the p-value. Each faint line is a made-up investor with no skill over the same days as your record: the same exposure to the index as yours (your beta), daily ups and downs of the same size, and an alpha of zero. Your own line runs through them, and a shaded band marks where about 95 in 100 of them finish. The crowd is spread by the same margin of error the t-stat uses, so your line finishes outside the band exactly when the t-stat is past 2 either way, and the number of no-skill investors it finished above is the one-sided reading of the same p-value. Like the p-value, it shows how far luck alone usually reaches. It is not the chance that you were lucky.

What is "chance your edge is real"?

Skill is rare: studies of retail investors find most trail the market after costs. So a strong-looking record should be read against how common skill is, not as if no one had ever invested before. Luck Filter combines your record with an assumed spread of true alpha across investors (the typical investor trailing by 3% a year, with true skill varying by about 5% a year either way) and reports the chance that your true alpha is above zero. The method is the one Baks, Metrick and Wachter used for mutual funds in 2001. The assumed spread is stated because it is an assumption; it will be replaced by the spread measured across real connected records once there are enough of them.

Why do short records rarely prove anything?

The margin of error shrinks only with the square root of time, so a real edge takes years to stand out from daily noise. Luck Filter gives no verdict under about six months of daily values, and flags any result under a year as easy to get by chance. When a result is promising but not yet clear, it estimates how many more years at the same pace it would take to reach a t-stat of 2. Very often the honest answer is "not enough data yet".

Two free calculators let you try this on your own numbers: how long an edge takes to prove for an edge and a spread you choose, and skill or luck from yearly returns, a simpler version of the test run on a few years of returns you type in.

Which benchmark is used?

By default, the main index for your account's currency, with dividends included: the S&P 500 for dollars, the FTSE 100 for pounds, the TSX 60 for Canadian dollars and the Euro Stoxx 50 for euros. You can switch to any of them or the Nasdaq-100. Luck Filter also warns when the chosen index explains little of your account's daily moves and another one fits much better: alpha against a poorly fitting index mostly says the account holds different things, not that its holdings were well picked.

Can a leaning towards a kind of share look like skill?

Yes. Over long stretches some kinds of shares have earned more than the market: smaller companies, cheaper ones (value) and recent winners (momentum), among others. A record that simply leaned that way can show alpha against an index without any skill in picking. So for numbers in dollars, Luck Filter runs the test again with those leanings taken out, using the factors of Fama and French and Carhart's momentum from Kenneth French's data library: the same regression, with the market, size, value and momentum (and, in the five-factor model, profitability and investment) as the yardsticks. What is left is alpha after style, with its own t-stat. It covers the days the record shares with the library, which runs a month or two behind.

What can the test not tell you?

• It describes the past. A real edge can fade, and nothing here predicts what comes next.
• If many people run the test, about 1 in 20 records with no skill will still pass the 0.05 line. A pass is evidence, not proof.
• It is one test against one benchmark. A different benchmark can give a different answer, which is why you can switch.

See how the returns behind the test are measured in time-weighted return vs Modified Dietz, or the demo's test explained step by step on the sample skill check.