Part E · Module 19
Testing a method on the past
Every rule in this course has been tested against twenty years of global data, including companies that have disappeared. This module explains how to run such a test without fooling yourself, and gives the results as they are, failures included.
Tested on twenty years of worldwide data, including companies that disappeared, the method beat small caps as a whole by 7.5 points a year over 2006-2015 and by 1.6 points over 2016-2025. Its most robust result: the score ranks shares well in every region. A value method can nevertheless disappoint for several years in a row.
- Recognise the biases that make a method look better than it is.
- Understand why decision rules must be written down before the test.
- Read the results of the course method, its strengths and weaknesses.
- Know what to expect from a value-based method, including long difficult years.
1. Why test on the past
A rule can seem obvious and earn nothing. The only way to find out is to apply it to the past as if you were there: at each date, look only at what could be known that day, choose the shares, hold them for a year, and count what they earned, costs included. The test in this course covers 40 selection dates (1 May and 1 November from 2006 to 2025), tens of thousands of companies in some thirty countries, and compares the method with all small caps equally weighted (the "universe").
The past does not guarantee the future. But a rule that added nothing for twenty years has little chance of working well tomorrow.
2. The pitfalls of a test
Survivorship bias. A database containing only companies still listed today forgets the bankruptcies: the past looks rosier than it was. The test includes delisted companies, but the source does not have accounts for all of them: they are missing for 92% of the companies that disappeared between 2006 and 2010, against 7% after 2021. Returns in the first decade are therefore too flattering, for the universe as for the method.
Look-ahead bias. Annual accounts closed on 31 December are only published in March or April. The test only uses financial years closed at least 120 days before the selection date.
Bad data. One price error is enough to distort an average: a Hong Kong company whose price had been multiplied by 1,000 in the database added 21 points to the universe's return for one year on its own. About 14% of the series contained aberrant jumps: they were cleaned, and 561 companies too damaged were excluded.
Costs. On a thinly traded small cap, the gap between bid and ask costs far more than the commission. The test counts 0.5% to 3% per purchase and per sale depending on liquidity.
Over-optimisation. Try twenty variants and some will win by chance. Picking the best one afterwards means fitting the rule to the past, with nothing gained for the future.
Harvey, Liu and Zhu (2016) listed more than 300 published "factors" supposed to explain stock returns, and showed that a large share do not survive correction for multiple testing: search long enough and you will find something. Bailey and López de Prado formalised the risk of backtest over-optimisation: the more variants you try, the more the best result obtained is flattered by chance.
3. Writing the rules before looking
Before the full test, a protocol set out in writing what would be measured and what would decide:
- Two halves judged separately: 2006-2015 and 2016-2025. A rule must work in both.
- A variant only enters the model if it gains at least 1 point a year in each half, without worsening the worst drawdown by more than 2 points, and in at least two of the three major regions. It replaces an existing rule, never adds to it.
- An existing rule leaves the model if removing it costs neither return nor safety.
- Every change to the protocol is dated and justified, and written before seeing its effect.
Even with a protocol, a false positive gets through: a quality measure (growth in tangible equity) passed all the accuracy criteria. Checked more closely, it only did better on one date in two, and, taken alone, it even predicted the wrong way round. It was ruled out. A result just above the threshold always deserves a second check.
4. What the method delivers
"Best scores" portfolio (the 20 best-scored shares, equal weights, rebuilt every year), all countries, annual return in euros including costs:
| 2006-2015 | 2016-2025 | |
|---|---|---|
| Method (Best scores) | 14.9% a year, worst drawdown −49% | 9,6 %, −45 % |
| Universe (all small caps, equal weights) | 7,4 %, −59 % | 7,9 %, −37 % |
| Difference | +7.5 points a year, smaller drawdown | +1.6 points a year, drawdown 8 points larger |
Against the criteria set in advance, the method clearly succeeds over 2006-2015 but fails over 2016-2025: the gain there is small and the worst drawdown exceeds the universe's by more than 5 points.
The most robust result lies elsewhere: the score ranks shares well. Sorted into five groups by score, shares earn more the higher their score, in both halves and in every region.
| Average return over 12 months | Lowest-scored 20% | 2 | 3 | 4 | Highest-scored 20% |
|---|---|---|---|---|---|
| 2006-2015 | +7,9 % | +10,8 % | +13,2 % | +14,6 % | +17,3 % |
| 2016-2025 | +5,1 % | +9,2 % | +10,2 % | +11,6 % | +14,3 % |
The weakness comes from the last step: a 20-position portfolio, rebuilt once a year, is very sensitive to the luck of the few shares selected.
5. By region
| Region | 2006-2015: method / universe | 2016-2025: method / universe |
|---|---|---|
| Europe | 12,4 % / 6,3 % | 13,7 % / 6,8 % |
| Asia-Pacific | 20,2 % / 9,4 % | 14,1 % / 6,2 % |
| North America | 8,7 % / 7,0 % | 12.9% / 11.3% (worst drawdown −52% against −39%) |
Europe and Asia-Pacific give a clear and steady advantage. North America is the weak spot: from 2016 to 2020, the selections there lost money while the market as a whole rose. This was the "value winter": rates close to zero, growth stocks in great demand, and many cheap companies that were cheap for good reasons (physical retail hit by the internet, shale oil). Since late 2020, the method has regained ground there.
A value-based method can go through several years of lagging. Those who abandon it after three years of disappointment often leave just before it starts winning again. You need to know this before you start.
6. What was removed
Each rule in the model was removed on its own to see whether it was useful. Three changed nothing and were deleted:
- the penalty imposed on a share that is rising but less strongly than its market;
- the automatic exclusion of stage 4 shares from portfolios: the trend already counts in the score, and a falling share almost never reaches the top scores;
- the adjustment for major acquisitions and disposals in the growth calculation.
The five families of the score (valuation, financial strength, quality, trend, safety) and the warnings on the accounts are all useful in at least one of the two periods, and were kept.
7. What did not work
About twenty variants, written in advance, were compared with the model. None was adopted:
- Trend: variants of Weinstein's stages, return relative to volatility over one year, proximity to the annual high. The last one won overall, but in no region taken alone.
- Valuation: multiples compared with the sector rather than fixed thresholds, EV/Sales compared with its own ten-year average, growth profile based on EBITDA.
- Construction: risk parity (removed from the method, results almost identical), a blend of the two portfolios, lower liquidity thresholds, waiting for a pullback towards the 40-week average before buying.
- Market climate: six signals for leaving the market, all useful against major crashes but costly in return (module 18).
Result: a simpler model than before the test, with no rule added.
8. What to conclude
- The score works: choosing among the best-scored shares earned more than choosing at random, everywhere and over twenty years.
- The gain is not steady: very clear from 2006 to 2015, small from 2016 to 2025, with years of lagging, especially in the United States.
- The method does not protect against crashes: it reduces the fall without avoiding it. The protection signal and the share of small caps in your wealth are there for that.
- The figures are too good in absolute terms, especially before 2016, because of companies that disappeared without accounts. The gaps between the method and the universe are more reliable than the returns themselves.
- Better to diversify across regions: the advantage was stronger and steadier in Europe and Asia-Pacific.
- An honest test includes companies that have disappeared, uses only the information known at each date and counts realistic costs.
- Decision rules are written before looking at the results; a gain just above the threshold is checked twice.
- The method beat small caps as a whole by 7.5 points a year over 2006-2015 and by 1.6 points over 2016-2025, with a larger worst drawdown in the second period.
- The score ranks shares well in every region: this is the most robust contribution.
- A value-based method can disappoint for several years in a row; you have to accept this in advance.
- Simplify when a rule adds nothing: the model comes out of the test simpler than it went in.
Exercises
Exercise 1: the survivor
A database contains only the 800 companies listed today. Fifteen years ago, 1,200 companies were listed; 250 went bankrupt, 150 were taken over. What bias affects a test using this database?
Answer. Survivorship bias: the 250 bankruptcies, often total losses, vanish from the test, and so do the takeovers, often made at a premium. The average return looks higher and the worst drawdown smaller than in reality.
Exercise 2: twenty variants
You test twenty variants of a rule. Only one beats the reference by 1.2 points a year, in only one of the two periods. What do you do?
Answer. Nothing. Out of twenty tries, it is normal for one variant to win by chance. A rule that wins in only one period, and by little, does not meet the criterion: keep the simplest version.
Exercise 3: three years of lagging
Your portfolio does 4 points worse than the market three years running. What do you check before abandoning the method?
Answer. Whether the lag comes from the style (cheap shares neglected, as from 2016 to 2020) or from a mistake in application (rules not followed, shares poorly chosen, costs too high). If the score still ranks shares well and the rules were followed, the lag is probably that of the style, which the test showed is possible; if the rules were not followed, it is the application that needs fixing.