How'm I doing? A review of my predictions
What better way to break a two-year hiatus than checking myself?
Something that amuses me: I haven’t posted a write-up here in 27 months, but I’ve picked up more subscribers in those months than I had before I stopped posting.1
Coming back to an audience that’s overwhelmingly new, I like the idea of reintroducing myself by summarizing what I’ve already written. And one of the few upsides to aging the passage of time is learning which (and whose) predictions about the wider world have come true. So I’ve decided to extract every specific prediction I can find in my previous postletters, and summarize how accurate those predictions have been.
By “specific prediction” I mean a prediction with some indication of probability or confidence or certainty, whether I made a flat declaration that something would(n’t) happen, used the word “expect” without qualification (implying high confidence), or used an explicit probability judgment like “likely” or “11.1%”. Simply writing that something “might” or “could” happen doesn’t count. A prediction also needs at least a vague, implicit time window. (If I made an unqualified prediction in 2021 that Donald Trump “will run again”, would that have meant 2024, or 2028, or what?) Also, I’m only tallying predictions I made in the main text of my 19 past postletters, not in the many comments I’ve scattered around Substack.2
As well as marking predictions as simply true or false, I mark a prediction as “provisionally true” if it hasn’t been falsified yet, and has evidence in its favor, but could theoretically be falsified soon (because its time window hasn’t quite ended, or because the prediction’s a claim about cause and effect where the effect is clear but causation is nontrivial to determine).
The predictions
I identified 18 predictions in 6 topical groups (Donald Trump in 2024, oil prices, inflation in the US, Trump again, reasons to expect hard times from 2022 onward, and inflation in the UK) across 5 postletters.
November 10, 2020: in a short update to “Must consequentialists vote for Joe Biden?”, I wrote that Donald Trump (A) “won’t be too old to run again” in 2024, (B) had an 86% chance of surviving to make a 2024 presidential run, and (C) “won’t be in prison”.
November 11, 2021: near the end of “Inflation is mostly oil prices: US edition”, I (A) wrote that “[o]il prices boomed [that] year but a further year-long boom seems unlikely”, (B) wrote that “supply-chain issues [we]re liable to block” “[a]nother year of year-on-year price increases near 100%”, (C) “seriously doubt[ed] that the moving-average index [of increases in oil price] would be even 80%”, (D) reckoned that an index of “like 40% seem[ed] more plausible”.
November 11, 2021: applying my guesses about oil prices to inflation in consumer prices, I (A) “ma[d]e a low-confidence guess of 3% for October 2022 CPI inflation”, (B) “fe[lt] more confident (though still not certain!) in predicting that CPI inflation won’t be over 6% in a year”, and (C) didn’t count out a Team Transitory-friendly CPI “below, say, 4%”.
December 3, 2021: arguing that “Donald J ain't going away!!”, I predicted that (A) “if Trump isn’t incapacitated in 2024 he is very likely to run for the presidency again”, (B) “Trump could be the first billionaire ex-President to go to prison, but it would be a real aberration, (C) Trump had “an 11.1%” chance of dying before Election Day 2024, and (D) that the “[a]ssassination risk” of Trump being knocked out of the 2024 race was maybe 1% or 2%.3
January 5, 2022: listing reasons to expect further hard times in “Things are going to get worse before they get worse”, I wrote that (A) “we can expect a Republican Congress after th[at] year’s midterms” and (B) “the Pacific Ocean’s in the cool La Niña phase of the El Niño–Southern Oscillation, and when it inevitably drifts out of that cool phase we can expect an anomalously warm year or two”.
January 23, 2022: closing my “Inflation is mostly oil prices: UK edition” postletter, I predicted that (A) January 2022’s “CPI inflation stands a good chance of topping December’s 5.4%” and (B) “unless the Bank of England jerk[ed] its Bank Rate up unexpectedly”, CPI inflation would “likely remain above target through the spring”.
Evaluating the predictions
I take each of those 6 groups of predictions in the same order as above.
Trump is running again in 2024 and isn’t in prison (convicted, yes, but not in prison), so predictions A and C are both true. Prediction B is provisionally true, but would be falsified by Trump’s death in the next 4 months.
When I made the oil-price predictions on November 11, 2021, West Texas Intermediate prices were up 99% year on year, and Brent prices were up 98%. A year later, on November 11, 2022, those year-on-year increases were only 10% and 16% respectively, having been below 80% since April 2022. I therefore rate prediction A, that “a further year-long boom” seemed “unlikely”, true; Russia’s (re-)invasion of Ukraine temporarily spiked oil prices in March 2022 but in the autumn they’d returned to pre-invasion levels. Similarly, with year-on-year increases having been mostly below 80%, I rate predictions C and D true. Prediction B is plausible since prices didn’t double from Nov. 2021 to Nov. 2022, but it’s not obvious whether that’s down to “supply-chain issues”, so I call it provisionally true rather than just true.
The October 2022 US CPI inflation was 7.7%, substantially higher than my “low-confidence guess of 3%”, so prediction A is false. November 2022’s CPI inflation was 7.1%, so prediction B, my guess that it wouldn’t “be over 6%”, is also false. In the year before that, inflation never went below 4%, so prediction C too is false.
Prediction A is conditional with a true antecedent — Donald Trump hasn’t been incapacitated this year — and so I can evaluate prediction A, which is true (Trump is indeed running for the presidency). I wrote that Trump going to prison “would be a real aberration”, and while Trump has spent 20 minutes being booked in jail, he hasn’t gone to prison, so prediction B is true. I put a probability of just 11% on Trump dying before Election Day 2024 and he hasn’t died, so prediction C is provisionally true, and he remains alive in spite of yesterday’s assassination attempt, so prediction D is provisionally true too.
Republicans took back the House in 2022’s midterms, but Democrats actually expanded their effective Senate majority to an actual majority, so prediction A is false. ENSO drifted out of its cool phase in mid-2023, staying in its hot El Niño phase until this May, and that El Niño period was in fact the globe’s hottest on record, so prediction B is true.
FRED’s record of UK CPI inflation has it at 4.8% in January 2022, seemingly below “December’s 5.4%” — but FRED now has December 2021 at 4.8%, not 5.4%, so evidently the dataset’s been revised since I made the original prediction. The only way to compare December 2021 and January 2022 on a consistent basis with this dataset is to use the current values for both months, which give 4.845% and 4.849% respectively, so I call prediction A true: January 2022 did, just barely, have higher inflation than December 2021. And while the Bank of England increased the Bank Rate in the first half of 2022, it did so in consistent quarter-point increments, not by “jerk[ing]” the rate “up unexpectedly”, so the antecedent of prediction B holds, and the consequent of prediction B is true: CPI inflation remained “above target through the spring”.
How’d I do? I’d say pretty good! I count 10 outright true predictions, 4 provisionally true predictions, and 4 outright false predictions, one of the 4 false predictions being explicitly “a low-confidence guess”. Generously, that’s a 22% failure rate, or, less generously, a 29% failure rate if I set aside the predictions that’re merely provisionally true.
Even less generously, prediction groups 1 and 4 overlap a lot. Both groups are about Trump running for President in 2024. Predictions 1C and 4B are very similar (Trump in, or having gone to, prison), as are predictions 1B and 4C (Trump’s chance of surviving to Election Day), so maybe I should count only 8 out of 8 predictions as outright true, rather than 10 out of 10, to avoid double-counting.
Conclusion
Something that may be more important to note than my overall success rate is that there’s lots of clustering. That is, if one of my predictions in a group was good, the other predictions in that group tended to work out too. Conversely, all 3 of my predictions about US inflation were false; I was consistently too optimistic. Asking myself why, I suspect that’s mainly because I didn’t anticipate in November 2021 that 3 months later Vladimir Putin would attack Ukraine again! Sometimes a forecaster just has to take a low-probability shock on the chin.
My remaining wrong prediction came from applying too firmly the rule of thumb that the incumbent party loses Congressional seats in midterms. Lesson to learn: don’t make high-confidence inferences from a rule of thumb.
I hope this all gives some idea of what I try to do here. Even if I’ve failed in that, whoever’s reading now has some summary information to judge quickly and objectively whether I’m disconnected from reality. Not that I think I am, but why should anyone else trust that judgment, coming from me?
Significant credit for that, I think, must go to referrals from Tana Ganeva and her blogletter Substance. Certainly my thanks go to her.
Although, as an aside, I want to crow about calling the second round of France’s 2022 presidential election. 4 days after the first round and 10 days before the second, I guessed that “Macron'll get 59ish% and Le Pen 41ish%”; the final result was 58.547% to 41.453%. I also want to note that I did nothing clever or magical: I basically split the difference between the polls in 2022 and the 2017 election result.
More vaguely, I added that “[m]ore likely than not, Donald J Trump ain’t going away”, and that Trump “could regain unstoppable momentum when the primary campaigns begin in just a year and a half”, but those predictions may be too vague (or redundant) to properly evaluate.