Invesmind

Part E · Module 19

Testing a method on the past

Every rule in this course has been tested against twenty years of global data, including companies that have disappeared. This module explains how to run such a test without fooling yourself, and gives the results as they are, failures included.

In short

Tested on twenty years of worldwide data, including companies that disappeared, the method beat small caps as a whole by 7.5 points a year over 2006-2015 and by 1.6 points over 2016-2025. Its most robust result: the score ranks shares well in every region. A value method can nevertheless disappoint for several years in a row.

Module objectives
  • Recognise the biases that make a method look better than it is.
  • Understand why decision rules must be written down before the test.
  • Read the results of the course method, its strengths and weaknesses.
  • Know what to expect from a value-based method, including long difficult years.

1. Why test on the past

A rule can seem obvious and earn nothing. The only way to find out is to apply it to the past as if you were there: at each date, look only at what could be known that day, choose the shares, hold them for a year, and count what they earned, costs included. The test in this course covers 40 selection dates (1 May and 1 November from 2006 to 2025), tens of thousands of companies in some thirty countries, and compares the method with all small caps equally weighted (the "universe").

The past does not guarantee the future. But a rule that added nothing for twenty years has little chance of working well tomorrow.

2. The pitfalls of a test

Survivorship bias. A database containing only companies still listed today forgets the bankruptcies: the past looks rosier than it was. The test includes delisted companies, but the source does not have accounts for all of them: they are missing for 92% of the companies that disappeared between 2006 and 2010, against 7% after 2021. Returns in the first decade are therefore too flattering, for the universe as for the method.

Look-ahead bias. Annual accounts closed on 31 December are only published in March or April. The test only uses financial years closed at least 120 days before the selection date.

Bad data. One price error is enough to distort an average: a Hong Kong company whose price had been multiplied by 1,000 in the database added 21 points to the universe's return for one year on its own. About 14% of the series contained aberrant jumps: they were cleaned, and 561 companies too damaged were excluded.

Costs. On a thinly traded small cap, the gap between bid and ask costs far more than the commission. The test counts 0.5% to 3% per purchase and per sale depending on liquidity.

Over-optimisation. Try twenty variants and some will win by chance. Picking the best one afterwards means fitting the rule to the past, with nothing gained for the future.

Academic basis

Harvey, Liu and Zhu (2016) listed more than 300 published "factors" supposed to explain stock returns, and showed that a large share do not survive correction for multiple testing: search long enough and you will find something. Bailey and López de Prado formalised the risk of backtest over-optimisation: the more variants you try, the more the best result obtained is flattered by chance.

3. Writing the rules before looking

Before the full test, a protocol set out in writing what would be measured and what would decide:

Pitfall

Even with a protocol, a false positive gets through: a quality measure (growth in tangible equity) passed all the accuracy criteria. Checked more closely, it only did better on one date in two, and, taken alone, it even predicted the wrong way round. It was ruled out. A result just above the threshold always deserves a second check.

4. What the method delivers

"Best scores" portfolio (the 20 best-scored shares, equal weights, rebuilt every year), all countries, annual return in euros including costs:

2006-20152016-2025
Method (Best scores)14.9% a year, worst drawdown −49%9,6 %, −45 %
Universe (all small caps, equal weights)7,4 %, −59 %7,9 %, −37 %
Difference+7.5 points a year, smaller drawdown+1.6 points a year, drawdown 8 points larger

Against the criteria set in advance, the method clearly succeeds over 2006-2015 but fails over 2016-2025: the gain there is small and the worst drawdown exceeds the universe's by more than 5 points.

The most robust result lies elsewhere: the score ranks shares well. Sorted into five groups by score, shares earn more the higher their score, in both halves and in every region.

Average return over 12 monthsLowest-scored 20%234Highest-scored 20%
2006-2015+7,9 %+10,8 %+13,2 %+14,6 %+17,3 %
2016-2025+5,1 %+9,2 %+10,2 %+11,6 %+14,3 %

The weakness comes from the last step: a 20-position portfolio, rebuilt once a year, is very sensitive to the luck of the few shares selected.

5. By region

Region2006-2015: method / universe2016-2025: method / universe
Europe12,4 % / 6,3 %13,7 % / 6,8 %
Asia-Pacific20,2 % / 9,4 %14,1 % / 6,2 %
North America8,7 % / 7,0 %12.9% / 11.3% (worst drawdown −52% against −39%)

Europe and Asia-Pacific give a clear and steady advantage. North America is the weak spot: from 2016 to 2020, the selections there lost money while the market as a whole rose. This was the "value winter": rates close to zero, growth stocks in great demand, and many cheap companies that were cheap for good reasons (physical retail hit by the internet, shale oil). Since late 2020, the method has regained ground there.

Pitfall

A value-based method can go through several years of lagging. Those who abandon it after three years of disappointment often leave just before it starts winning again. You need to know this before you start.

6. What was removed

Each rule in the model was removed on its own to see whether it was useful. Three changed nothing and were deleted:

The five families of the score (valuation, financial strength, quality, trend, safety) and the warnings on the accounts are all useful in at least one of the two periods, and were kept.

7. What did not work

About twenty variants, written in advance, were compared with the model. None was adopted:

Result: a simpler model than before the test, with no rule added.

8. What to conclude

Key points
  • An honest test includes companies that have disappeared, uses only the information known at each date and counts realistic costs.
  • Decision rules are written before looking at the results; a gain just above the threshold is checked twice.
  • The method beat small caps as a whole by 7.5 points a year over 2006-2015 and by 1.6 points over 2016-2025, with a larger worst drawdown in the second period.
  • The score ranks shares well in every region: this is the most robust contribution.
  • A value-based method can disappoint for several years in a row; you have to accept this in advance.
  • Simplify when a rule adds nothing: the model comes out of the test simpler than it went in.

Exercises

Exercise 1: the survivor

A database contains only the 800 companies listed today. Fifteen years ago, 1,200 companies were listed; 250 went bankrupt, 150 were taken over. What bias affects a test using this database?

Answer. Survivorship bias: the 250 bankruptcies, often total losses, vanish from the test, and so do the takeovers, often made at a premium. The average return looks higher and the worst drawdown smaller than in reality.

Exercise 2: twenty variants

You test twenty variants of a rule. Only one beats the reference by 1.2 points a year, in only one of the two periods. What do you do?

Answer. Nothing. Out of twenty tries, it is normal for one variant to win by chance. A rule that wins in only one period, and by little, does not meet the criterion: keep the simplest version.

Exercise 3: three years of lagging

Your portfolio does 4 points worse than the market three years running. What do you check before abandoning the method?

Answer. Whether the lag comes from the style (cheap shares neglected, as from 2016 to 2020) or from a mistake in application (rules not followed, shares poorly chosen, costs too high). If the score still ranks shares well and the rules were followed, the lag is probably that of the style, which the test showed is possible; if the rules were not followed, it is the application that needs fixing.