Futurist prediction methods and accuracy | Patreon

I've been reading a lot of predictions from people who are looking to understand what problems humanity will face 10-50 years out (and sometimes longer) in order to work in areas that will be instrumental for the future and wondering how accurate these predictions of the future are. The timeframe of predictions that are so far out means that only a tiny fraction of people making those kinds of predictions today have a track record so, if we want to evaluate which predictions are plausible, we need to look at something other than track record.

The idea behind the approach of this post was to look at predictions from an independently chosen set of predictors (Wikipedia's list of well-known futurists1) whose predictions are old enough to evaluate in order to understand which prediction techniques worked and which ones didn't work, allowing us to then (mostly in a future post) evaluate the plausibility of predictions that use similar methodologies.

Unfortunately, every single predictor from the independently chosen set had a poor record and, on spot checking some predictions from other futurists, it appears that futurists often have a fairly poor track record of predictions so, in order to contrast techniques that worked with techniques that I didn't, I sourced predictors that have a decent track record from my memory, an non-independent source which introduces quite a few potential biases.

Something that gives me more confidence than I'd otherwise have is that I avoided reading independent evaluations of prediction methodologies until after I did the evaluations for this post and wrote 98% of the post and, on reading other people's evaluations, I found that I generally agreed with Tetlock's Superforecasting on what worked and what didn't work despite using a wildly different data set.

In particular, people who were into "big ideas" who use a few big hammers on every prediction combined with a cocktail party idea level of understanding of the particular subject to explain why a prediction about the subject would fall to the big hammer generally fared poorly, whether or not their favored big ideas were correct. Some examples of "big ideas" would be "environmental doomsday is coming and hyperconservation will pervade everything", "economic growth will create near-infinite wealth (soon)", "Moore's law is supremely important", "quantum mechanics is supremely important", etc. Another common trait of poor predictors is lack of anything resembling serious evaluation of past predictive errors, making improving their intuition or methods impossible (unless they do so in secret). Instead, poor predictors often pick a few predictions that were accurate or at least vaguely sounded similar to an accurate prediction and use those to sell their next generation of predictions to others.

By contrast, people who had (relatively) accurate predictions had a deep understanding of the problem and also tended to have a record of learning lessons from past predictive errors. Due to the differences in the data sets between this post and Tetlock's work, the details are quite different here. The predictors that I found to be relatively accurate had deep domain knowledge and, implicitly, had access to a huge amount of information that they filtered effectively in order to make good predictions. Tetlock was studying people who made predictions about a wide variety of areas that were, in general, outside of their areas of expertise, so what Tetlock found was that people really dug into the data and deeply understood the limitations of the data, which allowed them to make relatively accurate predictions. But, although the details of how people operated are different, at a high-level, the approach of really digging into specific knowledge was the same.

Because this post is so long, this post will contain a very short summary about each predictor followed by a moderately long summary on each predictor. Then we'll have a summary of what techniques and styles worked and what didn't work, with the full details of the prediction grading and comparisons to other evaluations of predictors in the appendix.

Ray Kurzweil

Ray Kurzweil has claimed to have an 86% accuracy rate on his predictions, a claim which is often repeated, such as by Peter Diamandis where he says:

Of the 147 predictions that Kurzweil has made since the 1990's, fully 115 of them have turned out to be correct, and another 12 have turned out to be "essentially correct" (off by a year or two), giving his predictions a stunning 86% accuracy rate.

The article is titled "A Google Exec Just Claimed The Singularity Will Happen by 2029" opens with "Ray Kurzweil, Google's Director of Engineering, is a well-known futurist with a high-hitting track record for accurate predictions." and it cites this list of predictions on wikipedia. 86% is an astoundingly good track record for non-obvious, major, predictions about the future. This claim seems to be the source of other people claiming that Kurzweil has a high accuracy rate, such as here and here. I checked the accuracy rate of the wikipedia list Diamandis cited myself (using archive.org to get the list from when his article was published) and found a somewhat lower accuracy of 7%.

Fundamentally, the thing that derailed so many of Kurzweil's predictions is that he relied on the idea of exponential and accelerating growth in basically every area he can imagine, and even in a number of areas that have had major growth, the growth didn't keep pace with his expectations. His basic thesis is that not only do we have exponential growth due to progress (improve technologically, etc.), improvement in technology feeds back into itself, causing an increase in the rate of exponential growth, so we have double exponential growth (as in e^x^x, not 2*e^x) in many important areas, such as computer performance. He repeatedly talks about this unstoppable exponential or super exponential growth, e.g., in his 1990 book, The Age of Intelligent Machines, he says "One reliable prediction we can make about the future is that the pace of change will continue to accelerate" and he discusses this again in his 1999 book, The Age of Spiritual Machines, his 2001 essay on accelerating technological growth, titled "The Law of Accelerating Returns", his 2005 book, The Singularity is Near, etc.

One thing that's notable is despite the vast majority of his falsifiable predictions from earlier work being false, Kurzweil continues to use the same methodology to generate new predictions each time, which is reminiscent of Andrew Gelman's discussion of forecasters who repeatedly forecast the same thing over and over again in the face of evidence that their old forecasts were wrong. For example, in his 2005 The Singularity is Near, Kurzweil notes the existence of "S-curves", where growth from any particular "thing" isn't necessarily exponential, but, as he did in 1990, concludes that exponential growth will continue because some new technology will inevitably be invented which will cause exponential growth to continue and that "The law of accelerating returns applies to all of technology, indeed to any evolutionary process. It can be charted with remarkable precision in information-based technologies because we have well-defined indexes (for example, calculations per second per dollar, or calculations per second per gram) to measure them".

In 2001, he uses this method to plot a graph and then predicts unbounded life expectancy by 2011 (the quote below isn't unambiguous on life expectancy being unbounded, but it's unambiguous if you read the entire essay or his clarification on his life expectancy predictions, where he says "I don’t mean life expectancy based on your birthdate, but rather your remaining life expectancy"):

Most of you (again I’m using the plural form of the word) are likely to be around to see the Singularity. The expanding human life span is another one of those exponential trends. In the eighteenth century, we added a few days every year to human longevity; during the nineteenth century we added a couple of weeks each year; and now we’re adding almost a half a year every year. With the revolutions in genomics, proteomics, rational drug design, therapeutic cloning of our own organs and tissues, and related developments in bio-information sciences, we will be adding more than a year every year within ten years.

Kurzweil pushes the date this is expected to happen back by more than one year per year (the last citation I saw on this was a 2016 prediction that we would have unbounded life expectancy by 2029), which is characteristic of many of Kurzweil's predictions.

Quite a few people have said that Kurzweil's methodology is absurd because exponential growth can't continue indefinitely in the real world, but Kurzweil explains why he believes this is untrue in his 1990 book, The Age of Intelligent Machines:

A remarkable aspect of this new technology is that it uses almost no natural resources. Silicon chips use infinitesimal amounts of sand and other readily available materials. They use insignificant amounts of electricity. As computers grow smaller and smaller, the material resources utilized are becoming an inconsequential portion of their value. Indeed, software uses virtually no resources at all.

That we're entering a world of natural resource abundance because resources and power are irrelevant to computers hasn't been correct so far, but luckily for Kurzweil, many of the exponential and double exponential processes he predicted would continue indefinitely stopped long before natural resource limits would come into play, so this wasn't a major reason Kurzweil's predictions have been wrong, although it would be if his predictions were less inaccurate.

At a meta level, one issue with Kurzweil's methodology is that he has a propensity to "round up" to make growth look faster than it is in order to fit the world to his model. For example, in "The Law of Accelerating Returns", we noted that Kurzweil predicted unbounded lifespan by 2011 based on accelerating lifespan when "now we’re adding almost a half a year every year" in 2001. However, life expectancy growth in the U.S. (which, based on his comments, seems to be most of what Kurzweil writes about) was only 0.2 years per year overall and 0.1 years per year in longer lived demographics and worldwide life expectancy was 0.3 years per year. While it's technically true that you can round 0.3 to 0.5 if you're rounding to the nearest 0.5, that's a very unreasonable thing to do when trying to guess when unbounded lifespan will happen because the high rate of worldwide increase life expectancy was mostly coming from "catch up growth" where there was a large reduction in things that caused "unnaturally" shortened lifespans.

If you want to predict what's going to happen at the high end, it makes more sense to look at high-end lifespans, which were increasing much more slowly. Another way in which Kurzweil rounded up to get his optimistic prediction was to select a framing that made it look like we were seeing extremely rapid growth in life expectancies. But if we simply plot life expectancy over time since, say, 1950, we can see that growth is mostly linear-ish trending to sub-linear (and this is true even if we cut the graph off when Kurzweil was writing in 2001), with some super-linear periods that trend down to sub-linear. Kurzweil says he's a fan of using indexes, etc., to look at growth curves, but in this case where he can easily do so, he instead chooses to pick some numbers out of the air because his "standard" methodology of looking at the growth curves results in a fairly boring prediction of lifespan growth slowing down, so there are three kinds of rounding up in play here (picking an unreasonably optimistic number, rounding up that number, and then selectively not plotting a bunch of points on the time series to paint the picture Kurzweil wants to present).

Kurzweil's "rounding up" is also how he came up with the predictions that, among other things, computer performance/size/cost and economic growth would follow double exponential trajectories. For computer cost / transistor size, Kurzweil plotted, on a log scale, a number of points on the silicon scaling curve, plus one very old point from the pre-silicon days, when transistor size was on a different scaling curve. He then fits what appears to be a cubic to this, and since a cubic "wants to" either have high growth or high anti-growth in the future, and the pre-silicon point puts pulls the cubic fit very far down in the past, the cubic fit must "want to" go up in the future and Kurzweil rounds up this cubic growth to exponential. This was also very weakly supported by the transistor scaling curve at the time Kurzweil was writing. As someone who was following ITRS roadmaps at the time, my recollection is that ITRS set a predicted Moore's law scaling curve and semiconductor companies raced to beat curve, briefly allowing what appeared to be super-exponential scaling since they would consistently beat the roadmap, which was indexed against Moore's law. However, anyone who actually looked at the details of what was going on or talked to semiconductor engineers instead of just looking at the scaling curve would've known that people generally expected both that super-exponential scaling was temporary and not sustainable and that the end of Dennard scaling as well as transistor-delay dominated (as opposed to interconnect delay-dominated) high-performance processors were imminent, meaning that exponential scaling of transistor sizes would not lead to the historical computer performance gains that had previously accompanied transistor scaling; this expectation was so widespread that it was discussed in undergraduate classes at the time. Anyone who spent even the briefest amount of time looking into semiconductor scaling would've known these things at the time Kurzweil was talking about how we were entering an era of double exponential scaling and would've thought that we would be lucky to even having general single exponential scaling of computer performance, but since Kurzweil looks at the general shape of the curve and not the mechanism, none of this knowledge informed his predictions, and since Kurzweil rounds up the available evidence to support his ideas about accelerating acceleration of growth, he was able to find a selected set of data points that supported the curve fit he was looking for.

We'll see this kind of rounding up done by other futurists discussed here, as well as longtermists discussed in the appendix, and we'll also see some of the same themes over and over again, particularly exponential growth and the idea that exponential growth will lead to even faster exponential growth due to improvements in technology causing an acceleration of the rate at which technology improves.

Jacque Fresco

In 1969, Jacque Fresco wrote Looking Forward. Fresco claims it's possible to predict the future by knowing what values people will have in the future and then using that to derive what the future will look like. Fresco doesn't describe how one can know the values people will have in the future and assumes people will have the values he has, which one might describe as 60s/70s hippy values. Another major mechanism he uses to predict the future is the idea that people of the future will be more scientific and apply the scientific method.

He writes about how "the scientific method" is only applied in a limited fashion, which led to thousands of years of slow progress. But, unlike in the 20th century, in the 21st century, people will be free from bias and apply "the scientific method" in all areas of their life, not just when doing science. People will be fully open to experimentation in all aspects of life and all people will have "a habitual open-mindedness coupled with a rigid insistence that all problems be formulated in a way that permits factual checking".

This will, among other things, lead to complete self-knowledge of one's own limitations for all people as well as an end to unhappiness due to suboptimal political and social structures.

The third major mechanism Fresco uses to derive his predictions is the idea that computers will be able solve basically any problem one can imagine and that manufacturing technology will also progress similarly.

Each of the major mechanisms that are in play in Fresco's predictions are indistinguishable from magic. If you can imagine a problem in the domain, the mechanism is able to solve it. There are other magical mechanisms in play as well, generally what was in the air at the time. For example, behaviorism and operant conditioning were very trendy at the time, so Fresco assumes that society at large will be able to operant condition itself out of any social problems that might exist.

Although most of Fresco's predictions are technically not yet judgable because they're about the far future, for the predictions he makes whose time has come, I didn't see one accurate prediction.

Buckminster Fuller

Fuller is best known for inventing the geodesic dome, although geodesic domes were actually made by Walther Bauersfeld decades before Fuller "invented" them. Fuller is also known for a variety of other creations, like the Dymaxion car, as well as his futurist predictions.

I couldn't find a great source of a very long list of predictions from Fuller, but I did find this interview, where he makes a number of predictions. Fuller basically free associates with words, making predictions by riffing off of the English meaning of the word (e.g., see the teleportation prediction) or sometimes an even vaguer link.

Predictions from the video:

For those who've heard that Fuller predicted the creation of Bitcoin, that last prediction about an accounting system for wealth is the one people are referring to. Typically, people who say this haven't actually listened to the interview where he states the whole prediction and are themselves using Fuller's free association method. Bitcoin comes from spending energy to mine Bitcoin and Fuller predicted that the future would have a system of wealth based on energy, therefore Fuller predicted the creation of Bitcoin. If you actually listen to the interview, Bitcoin doesn't even come close to satisfying the properties of the system Fuller describes, but that doesn't matter if you're doing Fuller-style free association.

In this post, Fuller has fewer predictions graded than almost anyone else, so it's unclear what his accuracy would be if we had a list of, say, 100 predictions, but the predictions I could find have a 0% accuracy rate.

Michio Kaku

Among people on Wikipedia's futurist list, Michio Kaku is probably relatively well known because, as part of his work on science popularization, he's had a nationally (U.S.) syndicated radio show since 2006 and he frequently appears on talk shows and is interviewed by news organizations.

In his 1997 book, Visions: How Science Will Revolutionize the 21st Century, Kaku explains why predictions from other futurists haven't been very accurate and why his predictions are different:

... most predictions of the future have floundered because they have reflected the eccentric, often narrow viewpoints of a single individual.

The same is not true of Visions. In the course of writing numerous books, articles, and science commentaries, I have had the rare privilege of interviewing over 150 scientists from various disciplines during a ten-year period.

On the basis of these interviews, I have tried to be careful to delineate the time frame over which certain predictions will or will not be realized. Scientists expect some predictions to come about by the year 2020; others will not materialize until much later—from 2050 to the year 2100.

Kaku also claims that his predictions are more accurate than many other futurists because he's a physicist and thinking about things in the ways that physicists do allows for accurate predictions of the future:

It is, I think, an important distinction between Visions, which concerns an emerging consensus among the scientists themselves, and the predictions in the popular press made almost exclusively by writers, journalists, sociologists, science fiction writers, and others who are consumers of technology, rather than by those who have helped to shape and create it. ... As a research physicist, I believe that physicists have been particularly successful at predicting the broad outlines of the future. Professionally, I work in one of the most fundamental areas of physics, the quest to complete Einstein’s dream of a “theory of everything.” As a result, I am constantly reminded of the ways in which quantum physics touches many of the key discoveries that shaped the twentieth century.

In the past, the track record of physicists has been formidable: we have been intimately involved with introducing a host of pivotal inventions (TV, radio, radar, X-rays, the transistor, the computer, the laser, the atomic bomb), decoding the DNA molecule, opening new dimensions in probing the body with PET, MRI, and CAT scans, and even designing the Internet and the World Wide Web.

He also specifically calls out Kurzweil's predictions as absurd, saying Kurzweil has "preposterous predictions about the decades ahead, from vacationing on Mars to banishing all diseases."

Although Kaku finds Kurzweil's predictions ridiculous, his predictions rely on some of the same mechanics Kurzweil relies on. For example, Kaku assumes that materials / commodity prices will tank in the then-near future because the advance of technology will make raw materials less important and Kaku also assumes the performance and cost scaling of computer chips would continue on the historical path it was on during the 70s and 80s. Like most of the other futurists from Wikipedia's list, Kaku also assumed that the pace of scientific progress would rapidly increase, although his reasons are different (he cites increased synergy between the important fields of quantum mechanics, computer science, and biology, which he says are so important that "it will be difficult to be a research scientist in the future without having some working knowledge of" all of those fields).

Kaku assumed that UV lithography would run out of steam and that we'd have to switch to X-ray or electron lithography, which would then run out of steam, requiring us to switch to a fundamentally different substrate for computers (optical, molecular, or DNA) to keep performance and scaling on track, but advances in other fundamental computing substrates have not materialized quickly enough for Kaku's predictions to come to pass. Kaku assigned very high weight to things that have what he considers "quantum" effects, which is why, for example, he cites the microprocessor as something that will be obsolete by 2020 (they're not "quantum") whereas fiber optics will not be obsolete (they rely on "quantum" mechanisms). Although Kaku pans other futurists for making predictions without having a real understanding of the topics they're discussing, it's not clear that Kaku has a better understanding of many of the topics being discussed even though, as a physicist, Kaku has more relevant background knowledge.

The combination of assumptions above that didn't pan out leads to a fairly low accuracy rate for Kaku's predictions in Visions.

I didn't finish Visions, but the prediction accuracy rate of the part of the book I read (from the beginning until somewhere in the middle, to avoid cherry picking) was 3% (arguably 6% if you give full credit to the prediction I gave half credit to). He made quite a few predictions I didn't score in which he said something "may" happen. Such a prediction is, of course, unfalsifiable because the statement is true whether or not the event happens.

John Naisbitt

Anyone who's a regular used book store bargain bin shopper will have seen this name on the cover of Megatrends, which must be up there with Lee Iacocca's autobiography as one of the most common bargain bin fillers.

Naisbitt claims that he's able to accurately predict the future using "content analysis" of newspapers, which he says was used to provide great insights during WWII and has been widely used by the intelligence community since then, but hadn't been commercially applied until he did it. Naisbitt explains that this works because there's a fixed amount of space in newspapers (apparently newspapers can't be created or destroyed nor can newspapers decide to print significantly more or less news or have editorial shifts in what they decide to print that are not reflected by identical changes in society at large):

Why are we so confident that content analysis is an effective way to monitor social change? Simply stated, because the news hole in a newspaper is a closed system. For economic reasons, the amount of space devoted to news in a newspaper does not change significantly over time. So, when something new is introduced, something else or a combination of things must be omitted. You cannot add unless you subtract. It is the principle of forced choice in a closed system.

Unfortunately, it's not really possible to judge Naisbitt's predictions because he almost exclusively deals in vague, horoscope-like, predictions which can't really be judged as correct or incorrect. If you just read Megatrends for the flavor of each chapter and don't try to pick out individual predictions, some chapters seem quite good, e.g., "Industrial Society -> Information Society", but some are decidedly mixed even if you very generously grade his vague predictions, e.g., "From Forced Technology to High Tech / High Touch". This can't really be compared to the other futurists in this post because it's much easier to make vague predictions sound roughly correct than to make precise predictions correct but, even so, if reading for general feel of what direction the future might go, Naisbitt's predictions are much more on the mark than any other futurists discussed.

That being said, as far as I read in his book, the one concrete prediction I could find was incorrect, so if you want to score Naisbitt comparably to the other futurists discussed here, you might say his accuracy rate is 0% but with very wide error bars.

Gerard K. O'Neill

O'Neill has two relatively well-known non-fiction futurist books, 2081 and The Technology Edge. 2081 was written in 1980 and predicts the future 100 years from then. The Technology Edge discusses what O'Neill thought the U.S. needed to do in 1983 to avoid being obsoleted by Japan.

O'Neill spends a lot more space on discussing why previous futurists were wrong than any other futurist under discussion. O'Neill notes that "most [futurists] overestimated how much the world would be transformed by social and political change and underestimated the forces of technological change" and cites Kipling, Verne, Wells, Haldane, and Ballamy, as people who did this. O'Neill also says that "scientists tend to overestimate the chances for major scientific breakthroughs and underestimate the effects of straightforward developments well within the boundaries of existing knowledge" and cites Haldane again on this one. O'Neill also cites spaceflight as a major miss of futurists past, saying that they tended to underestimate how quickly spaceflight was going to develop.

O'Neill also says that it's possible to predict the future without knowing the exact mechanism by which the change will occur. For example, he claims that the automobile could've been safely predicted even if the internal combustion engine hadn't been invented because steam would've also worked. But he also goes on to say that there are things it would've been unreasonable to predict, like the radio, TV, and electronic communications, saying that even though the foundations for those were discovered in 1865 and that the time interval between a foundational discovery and its application is "usually quite long", citing 30-50 years from quantum mechanics to integrated circuits and 100+ years from relativity to faster than light travel, and 50+ years from the invention of nuclear power without "a profound impact".

I don't think O'Neill ever really explains why his predictions are of the "automobile" kind in a convincing way. Instead, he relies on doing the opposite of what he sees as mistakes others made. The result is that he predicts huge advancements in space flight, saying we should expect we should expect large scale space travel and colonization by 2081, presaged by wireless transmission of energy by 2000 (referring to energy beamed down from satellites) and interstellar probes by 2025 (presumably something of a different class than the Voyager probes, which were sent out in 1977).

In 1981, he said "a fleet of reusable vehicles of 1990s vintage, numbering much less than today's world fleet of commercial jet transports, would be quite enough to provide transport into space and back again for several hundred million people per year", predicting that something much more advanced the the NASA Space Shuttle would be produced shortly afterwards. Continuing that progress "by the year 2010 or thereabouts there will be many space colonies in existence and many new ones being constructed each year".

Most of O'Neill's predictions are for 2081, but he does make the occasional prediction for a time before 1981. All of the falsifiable ones I could find were incorrect, giving him an accuracy rate of approximately 0% but with fairly wide error bars.

Patrick Dixon

Dixon is best known for writing Futurewise, but he has quite a few books with predictions about the future. In this post, we're just going to look at Futurewise, because it's the most prediction-oriented book Dixon has that's old enough that we ought to be able to make a call on a decent number of his predictions (Futurewise is from 1998; his other obvious candidate, The Future of Almost Everything is from 2015 and looks forward a century).

Unlike most other futurists featured in this post, Dixon doesn't explicitly lay out why you should trust his predictions in Futurewise in the book itself, although he sort of implicitly does so in the acknowledgements, where he mentions having interacted with many very important people.

I am indebted to the hundreds of senior executives who have shaped this book by their participation in presentations on the Six Faces of the Future. The content has been forged in the realities of their own experience.

And although he doesn't explicitly refer to himself, he also says that business success will come from listening to folks who have great vision:

Those who are often right will make a fortune. Trend hunting in the future will be a far cry from the seventies or eighties, when everything was more certain. In a globalized market there are too many variables for back-projection and forward-projection to work reliably .. That's why economists don't make good futurologists when it comes to new technologies, and why so many boards of large corporations are in such a mess when it comes to quantum leaps in thinking beyond 2000.

Second millennial thinking will never get us there ... A senior board member of a Fortune 1000 company told me recently: 'I'm glad I'm retiring so I don't have to face these decisions' ... 'What can we do?' another senior executive declares ...

Later, in The Future of Almost Everything, Dixon lays out the techniques that he says worked when he wrote Futurewise, which "has stood the test of time for more than 17 years". Dixon says:

All reliable, long-range forecasting is based on powerful megatrends that have been driving profound, consistent and therefore relatively predictable change over the last 30 years. Such trends are the basis of every well- constructed corporate strategy and government policy ... These wider trends have been obvious to most trend analysts like myself for a while, and have been well described over the last 20–30 years. They have evolved much more slowly than booms and busts, or social fads.

And lays out trends such as:

Dixon declines to mention trends he predicted that didn't come to pass (such as his prediction that increased tribalism will mean that most new wealth is created in small firms of 20 or fewer employees which will mostly be family owned, or his prediction that the death of "old economics" means that we'll be able to have high economic growth with low unemployment and no inflationary pressure indefinitely), or cases where the trend progression caused Dixon's prediction to be wildly incorrect, a common problem when making predictions off of exponential trends because a relatively small inaccuracy in the rate of change can result in a very large change in the final state.

Dixon's website is full of endorsements for him, with implicit and explicit claims that he's a great predictor of the future, as well as more general statements such as "Patrick Dixon has been ranked as one of the 20 most influential business thinkers alive today".

Back in Futurewise, Dixon relies heavily on the idea of a stark divide between "second millennial thinking" and "third millennial thinking" repeatedly comes up in Dixon's text. Like nearly everyone else under discussion, Dixon also extrapolates out from many existing trends to make predictions that didn't pan out, e.g., he looked at the falling cost and decreasing price of phone lines and predicted that people would end up with a huge number of phone lines in their home by 2005 and that screens getting thinner would mean that we'd have "paper-thin display sheets" in significant use by 2005. This kind of extrapolation sometimes works and Dixon's overall accuracy rate of 10% is quite good compared to the other "futurists" under discussion here.

However, when Dixon explains his reasoning in areas I have some understanding of, he seems to be operating at the buzzword level, so that when he makes a correct call, it's generally for the wrong reasons. For example, Dixon says that software will always be buggy, which seems true, at least to date. However, his reasoning for this is that new computers come out so frequently (he says "less than 20 months" — a reference to the 18 month timeline in Moore's law) and it takes so long to write good software ("at least 20 years") that programmers will always be too busy rewriting software to run on the new generation of machines (due to the age of the book, he uses the example of "brand new code ... written for Pentium chips").

It's simply not the case that most bugs or even, as a fraction of bugs, almost any bugs are due to programmers rewriting existing code to run on new CPUs. If you really squint, you can see things like Android devices having lots of security bugs due to the difficulty of updating Android and backporting changes to older hardware, but those kinds of bugs are both a small fraction of all bugs and not really what Dixon was talking about.

Similarly, on how computer backups will be done in the future, Dixon basically correctly says that home workers will be vulnerable to data loss and people who are serious about saving data will back up data online, "back up data on-line to computers in other cities as the ultimate security".

But Dixon's stated reason for this is that workstations already have large disk capacity (>= 2GB) and floppy disks haven't kept up (< 2MB), so it would take thousands of floppy disks to do backups, which is clearly absurd. However, even at the time, Zip drives (100MB per portable disk) were common and, although it didn't take off, the same company that made Zip drives also made 1GB "Jaz" drives. And, of course, tape backup was also used at the time and is still used today. This trend has continued to this day; large, portable, disks are available, and quite a few people I know transfer or back up large amounts of data on portable disks. The reason most people don't do disk/tape backups isn't that it would require thousands of disks to backup a local computer (if you look at the computers people typically use at home, most people could back up their data onto a single portable disk per failure domain and even keep multiple versions on one disk), but that online/cloud backups are more convenient.

Since Dixon's reasoning was incorrect (at least in the cases where I'm close enough to the topic to understand how applicable the reasoning was), it seems that when Dixon is correct, it can't be for the stated reason and Dixon is either correct by coincidence or because he's looking at the broader trend and came up with an incorrect rationalization for the prediction. But, per the above, it's very difficult to actually correctly predict the growth rate of a trend over time, so without some understanding of the mechanics in play, one could also say that a prediction that comes true based on some rough trend is also correct by coincidence.

Alvin Toffler / Heidi Toffler

Like most others on this list, Toffler claims some big prediction wins

The Tofflers claimed on their website to have foretold the breakup of the Soviet Union, the reunification of Germany and the rise of the Asia-Pacific region. He said in the People’s Daily interview that “Future Shock” envisioned cable television, video recording, virtual reality and smaller U.S. families.

In this post, we'll look at Future Shock, Toffler's most famous work, written in 1970.

According to a number of sources, Alvin Toffler's major works were co-authored by Heidi Toffler. In the books themselves, Heidi Toffler is acknowledged as someone who helped out a lot, but not as an author, despite the remarks elsewhere about co-authorship. In this section, I'm going to refer to Toffler in the singular, but you may want to mentally substitute the plural.

Toffler claims that we should understand the present not only by understanding the past, but also by understanding the future:

Previously, men studied the past to shed light on the present. I have turned the time-mirror around, convinced that a coherent image of the future can also shower us with valuable insights into today. We shall find it increasingly difficult to understand our personal and public problems without making use of the future as an intellectual tool. In the pages ahead, I deliberately exploit this tool to show what it can do.

Toffler generally makes vague, wish-y wash-y statements, so it's not really reasonable to score Toffler's concrete predictions because so few predictions are given. However, Toffler very strongly implies that past exponential trends are expected to continue or even accelerate and that the very rapid change caused by this is going to give rise to "future shock", hence the book's title:

I coined the term "future shock" to describe the shattering stress and disorientation that we induce in individuals by subjecting them to too much change in too short a time. Fascinated by this concept, I spent the next five years visiting scores of universities, research centers, laboratories, and government agencies, reading countless articles and scientific papers and interviewing literally hundreds of experts on different aspects of change, coping behavior, and the future. Nobel prizewinners, hippies, psychiatrists, physicians, businessmen, professional futurists, philosophers, and educators gave voice to their concern over change, their anxieties about adaptation, their fears about the future. I came away from this experience with two disturbing convictions. First, it became clear that future shock is no longer a distantly potential danger, but a real sickness from which increasingly large numbers already suffer. This psycho-biological condition can be described in medical and psychiatric terms. It is the disease of change .. Earnest intellectuals talk bravely about "educating for change" or "preparing people for the future." But we know virtually nothing about how to do it ... The purpose of this book, therefore, is to help us come to terms with the future— to help us cope more effectively with both personal and social change by deepening our understanding of how men respond to it

The big hammer that Toffler uses everywhere is extrapolation of exponential growth, with the implication that this is expected to continue. On the general concept of extrapolating out from curves, Toffler's position is very similar to Kurzweil's: if you can see a trend on a graph, you can use that to predict the future, and the ability of technology to accelerate the development of new technology will cause innovation to happen even more rapidly than you might naively expect:

Plotted on a graph, the line representing progress in the past generation would leap vertically off the page. Whether we examine distances traveled, altitudes reached, minerals mined, or explosive power harnessed, the same accelerative trend is obvious. The pattern, here and in a thousand other statistical series, is absolutely clear and unmistakable. Millennia or centuries go by, and then, in our own times, a sudden bursting of the limits, a fantastic spurt forward. The reason for this is that technology feeds on itself. Technology makes more technology possible, as we can see if we look for a moment at the process of innovation. Technological innovation consists of three stages, linked together into a self-reinforcing cycle. ... Today there is evidence that the time between each of the steps in this cycle has been shortened. Thus it is not merely true, as frequently noted, that 90 percent of all the scientists who ever lived are now alive, and that new scientific discoveries are being made every day. These new ideas are put to work much more quickly than ever before.

The first N major examples of this from the book are:

As we just noted above, when discussing Dixon, Kurzweil, etc., predicting the future by extrapolating out exponential growth is fraught. Toffler somehow manages to pull off the anti-predictive feat of naming a bunch of trends which were about to stop, some of which already had their writing on the wall when Toffler was writing.

Toffler then extrapolates from the above and predicts that the half-life of everything will become shorter, which will overturn how society operates in a variety of ways.

For example, companies and governments will replace bureaucracies with "adhocracies" sometime between 1995 and 2020 . The concern that people will feel like cogs as companies grow larger is obsolete because, in adhocracy, the entire concept of top-down command and control will disappear, obsoleted by the increased pace of everything causing top-down command and control structures to disappear. While it's true that some companies have less top-down direction than would've been expected in Toffler's time, many also have more, which has been enabled by technology allowing employers to keep stricter tabs on employees than ever before, making people more of a cog than ever before.

Another example is that Toffler predicted human colonization of the Ocean, "The New Atlantis", "long before the arrival of A.D. 2000".

Fabian Giesen points out that, independent of the accuracy of Toffler's predictions, Venkatesh Rao's Welcome to the Future Nauseous explains why "future shock" didn't happen in areas of very rapid technological development.

People from the Wikipedia list who weren't included

Steve Yegge

As I mentioned at the start, none of the futurists from Wikipedia's list had very accurate predictions, so we're going to look at a couple other people from other sources who aren't generally considered futurists to see how they rank.

We previously looked at Yegge's predictions here, which were written in 2004 and were generally about the next 5-10 years, with some further out. There were nine predictions (technically ten, but one isn't really a prediction). If grading them as written, which is how futurists have been scored, I would rank these at 4.5/9, or about 50%.

You might argue that this is unfair because Yegge was predicting the relatively near future, but if we look at relatively near future predictions from futurists, their accuracy rate is generally nowhere near 50%, so I don't think it's unfair to compare the number in some way.

If you want to score these like people often score futurists, where they get credit for essentially getting things directionally correct, then I'd say that Yegge's score should be between 7/9 and 8/9, depending on how much partial credit he gets for one of the questions.

If you want to take a more holistic "what would the world look like if Yegge's vision were correct vs. the world we're in today", I think Yegge also does quite well there, with the big miss being that Lisp-based languages have not taken over the world, the success of Clojure notwithstanding. This is quite different than the futurists here, who generally had a vision of many giant changes that didn't come to pass, e.g., if we look at Kurzweil's vision of the world, by 2010, we would've had self-driving cars, a "cure" for paraplegia, widespread use of AR, etc., by 2011, we would have unbounded life expectancy, and by 2019 we would have pervasive use of nanotechnology including computers having switched from transistors to nanotubes, effective "mitigations" for blindness and deafness, fairly widely deployed fully realistic VR that can simulate sex via realistic full-body stimulation, pervasive self-driving cars (predicted again), entirely new fields of art and music, etc., and all that these things imply, which is a very different world than the world we actually live in.

And we see something similar if we look at other futurists, who predicted things like living underground, living under the ocean, etc.; most predicted many revolutionary changes that would really change society, a few of which came to pass. Yegge, instead, predicted quite a few moderate changes (as well as some places where change would be slower than a lot of people expected) and changes were slower than he expected in the areas he predicted, but only by a bit.

Yegge described his methodology for the post above as:

If you read a lot, you'll start to spot trends and undercurrents. You might see people talking more often about some theme or technology that you think is about to take off, or you'll just sense vaguely that some sort of tipping point is occurring in the industry. Or in your company, for that matter.

I seem to have many of my best insights as I'm writing about stuff I already know. It occurred to me that writing about trends that seem obvious and inevitable might help me surface a few not-so-obvious ones. So I decided to make some random predictions based on trends I've noticed, and see what turns up. It's basically a mental exercise in mining for insights

In this essay I'll make ten predictions based on undercurrents I've felt while reading techie stuff this year. As I write this paragraph, I have no idea yet what my ten predictions will be, except for the first one. It's an easy, obvious prediction, just to kick-start the creative thought process. Then I'll just throw out nine more, as they occur to me, and I'll try to justify them even if they sound crazy.

He's not really trying to generate the best predictions, but still did pretty well by relying on his domain knowledge plus some intuition about what he's seen.

In the post about Yegge's predictions, we also noted that he's made quite a few successful predictions outside of his predictions post:

Steve also has a number of posts that aren't explicitly about predictions that, nevertheless, make pretty solid predictions about how things are today, written way back in 2004. There's It's Not Software, which was years ahead of its time about how people write “software”, how writing server apps is really different from writing shrinkwrap software in a way that obsoletes a lot of previously solid advice, like Joel's dictum against rewrites, as well as how service oriented architectures look; the Google at Delphi (again from 2004) correctly predicts the importance of ML and AI as well as Google's very heavy investment in ML; an old interview where he predicts "web application programming is gradually going to become the most important client-side programming out there. I think it will mostly obsolete all other client-side toolkits: GTK, Java Swing/SWT, Qt, and of course all the platform-specific ones like Cocoa and Win32/MFC/"; etc. A number of Steve's internal Google blog posts also make interesting predictions, but AFAIK those are confidential.

Quite a few of Yegge's predictions would've been considered fairly non-obvious at the time and he seemed to still have a fairly good success rate on his other predictions (although I didn't try to comprehensively find them and score them, I sampled some of his old posts and found the overall success rate to be similar to the ones in his predictions post).

With Yegge and the other predictors that were picked so that we can look at some accurate predictions there is, of course, a concern that there's survivorship bias in picking these predictors. I suspect that's not the case for Yegge because he continued to be accurate after I first noticed that he seemed to have accurate predictions, so it's not just that I picked someone who had a lucky streak after the fact. Also, especially with some of his Google internal G+ comments, made fairly high dimension comments that ended being right for the reasons he suggested, which provides a lot more information about how accurate his reasoning was than simply winning a bunch of coin flips in a row. This comment about depth of reasoning doesn't apply to Caplan, below, because I haven't evaluated Caplan's reasoning, but does apply to MS leadership circa 1990.

Bryan Caplan

Bryan Caplan reports that his track record is 23/23 = 100%. He is much more precise in specifying his predictions than anyone else we've looked at and tries to give a precise bet that will be trivial to adjudicate as well as betting odds.

Caplan started making predictions/bets around the time the concept that "betting is a tax on bullshit" became popular (the idea being that a lot of people are willing to say anything but will quiet down if asked to make a real bet and those that don't will pay a real cost if they make bad real bets) and Caplan seems to have a strategy as acting as a tax man on bullshit in that he generally takes the safe side of bets that people probably shouldn't have made. Andrew Gelman says:

Caplan’s bets are an interesting mix. The first one is a bet where he offered 1-to-100 odds so it’s no big surprise that he won, but most of them are at even odds. A couple of them he got lucky on (for example, he bet in 2008 that no large country would leave the European Union before January 1, 2020, so he just survived by one month on that one), but, hey, it’s ok to be lucky, and in any case even if he only had won 21 out of 23 bets, that would still be impressive.

It seems to me that Caplan’s trick here is to show good judgment on what pitches to swing at. People come at him with some strong, unrealistic opinions, and he’s been good at crystallizing these into bets. In poker terms, he waits till he has the nuts, or nearly so. 23 out of 23 . . . that’s a great record.

I think there's significant value in doing this, both in the general "betting is a tax on bullshit" sense as well as, more specifically, if you have high belief that someone is trying to take the other side of bad bets and has good judgment, knowing that the Caplan-esque bettor has taken the position gives you decent signal about the bet even if you have no particular expertise in the subject. For example, if you look at my bets, even though I sometimes take bets against obviously wrong positions, I much more frequently take bets I have a very good chance of losing, so just knowing that I took a bet provides much less information than knowing that Caplan took a bet.

But, of course, taking Caplan's side of a bet isn't foolproof. As Gelman noted, Caplan got lucky at least once, and Caplan also seems likely to lose the Caplan and Tabarrok v. Bauman bet on global temperature. For that particular bet, you could also make the case that he's expected to lose since he took the bet with 3:1 odds, but a lot of people would argue that 3:1 isn't nearly long enough odds to take that bet.

The methodology that Caplan has used to date will never result in a positive prediction of some big change until the change is very likely to happen, so this methodology can't really give you a vision of what the future will look like in the way that Yegge or Gates or another relatively accurate predictor who takes wilder bets could.

Bill Gates / Nathan Myhrvold / MS leadership circa 1990 to 1997

A handful of memos that were released to the world due to the case against Microsoft which laid out the vision Microsoft executives had about how the world would develop, with or without Microsoft's involvement. These memos don't lay out concrete predictions with timelines and therefore can't be scored in the same way futurist predictions were scored in this post. If rating these predictions on how accurate their vision of the future was, I'd rate them similarly to Steve Yegge (who scored 7/9 or 8/9), but the predictions were significantly more ambitious, so they seem much more impressive when controlling for the scope of the predictions.

Compared to the futurists we discussed, there are multiple ways in which the predictions are much more detailed (and therefore more impressive for a given level of accuracy on top of being more accurate). One is that MS execs have a much deeper understanding of the things under discussion and how they impact each other. With "our" futurists, they often discuss things at a high level and, when they discuss things in detail, they make statements that make it clear that they don't really understand the topic and often don't really know what the words they're writing mean. MS execs of the era pretty clearly had a deep understanding of the issues in play, which let them make detailed predictions that our futurists wouldn't make, e.g., while protocols like FTP and IRC will continue to be used, the near future of the internet is HTTP over TCP and the browser will become a "platform" in the same way that Windows is a "platform", one that's much more important and larger than any OS (unless Microsoft is successful in taking action to stop this from coming to pass, which it was not despite MS execs foreseeing the exact mechanisms that could cause MS to fail to own the internet). MS execs use this level of understanding to make predictions about the kinds of larger things that our futurists discuss, e.g., the nature of work and how that will change.

Actually having an understanding of the issues in play and not just operating with a typical futurist buzzword level understanding of the topics allowed MS leadership to make fairly good guesses about what the future would look like.

For a fun story about how much effort Gates spent on understanding what was going on, see this story by Joel Spolsky on his first Bill Gates review:

Bill turned to me.

I noticed that there were comments in the margins of my spec. He had read the first page!

He had read the first page of my spec and written little notes in the margin!

Considering that we only got him the spec about 24 hours earlier, he must have read it the night before.

He was asking questions. I was answering them. They were pretty easy, but I can’t for the life of me remember what they were, because I couldn’t stop noticing that he was flipping through the spec…

He was flipping through the spec! [Calm down, what are you a little girl?]

… [ed: ellipses are from the original doc] and THERE WERE NOTES IN ALL THE MARGINS. ON EVERY PAGE OF THE SPEC. HE HAD READ THE WHOLE GODDAMNED THING AND WRITTEN NOTES IN THE MARGINS.

He Read The Whole Thing! [OMG SQUEEE!]

The questions got harder and more detailed.

They seemed a little bit random. By now I was used to thinking of Bill as my buddy. He’s a nice guy! He read my spec! He probably just wants to ask me a few questions about the comments in the margins! I’ll open a bug in the bug tracker for each of his comments and makes sure it gets addressed, pronto!

Finally the killer question.

“I don’t know, you guys,” Bill said, “Is anyone really looking into all the details of how to do this? Like, all those date and time functions. Excel has so many date and time functions. Is Basic going to have the same functions? Will they all work the same way?”

“Yes,” I said, “except for January and February, 1900.”

Silence. ... “OK. Well, good work,” said Bill. He took his marked up copy of the spec ... and left

Gates (and some other MS execs) were very well informed about what was going on to a fairly high level of detail considering all of the big picture concerns they also had in mind.

A topic for another post is how MS leadership had a more effective vision for the future than leadership at old-line competitors (Novell, IBM, AT&T, Yahoo, Sun, etc.) and how this resulted in MS turning into a $2T company while their competitors became, at best, irrelevant and most didn't even succeed at becoming irrelevant and ceased to exist. Reading through old MS memos, it's clear that MS really kept tabs on what competitors were doing and they were often surprised at how ineffective leadership was at their competitors, e.g., on Novell, Bill Gates says "Our traditional competitors are just getting involved with the Internet. Novell is surprisingly absent given the importance of networking to their position"; Gates noted that Frankenberg, then-CEO of Novell, seemed to understand the importance of the internet, but Frankenberg only joined Novell in 1994 and left in 1996 and spent much of his time at Novell reversing the direction the company had taken under Noorda, which didn't leave Novell with a coherent position or plan when Frankenberg "resigned" two years into the pivot he was leading.

In many ways, a discussion of what tech execs at the time thought the future would look like and what paths would lead to success is more interesting than looking at futurists who basically don't understand the topics they're talking about, but I started this post to look at how well futurists understood the topics they discussed and I didn't know, in advance, that their understanding of the topics they discuss and resultant prediction accuracy would be so poor.

Common sources of futurist errors

Not learning from mistakes

The futurists we looked at in this post tend to rate themselves quite highly and, after the fact, generally claim credit for being great predictors of the future, so much so that they'll even tell you how you can predict the future accurately. And yet, after scoring them, the most accurate futurist (among the ones who made concrete enough predictions that they could be scored) came in at 10% accuracy with generous grading that gave them credit for making predictions that accidentally turned out to be correct when they mispredicted the mechanism by which the prediction would come to pass (a strict reading of many of their predictions would reduce the accuracy because they said that the prediction would happen because of their predicted mechanism, which is false, rendering the prediction false).

There are two tricks that these futurists have used to be able to make such lofty claims. First, many of them make vague predictions and then claim credit if anything vaguely resembling the prediction comes to pass. Second, almost all of them make a lot of predictions and then only tally up the ones that came to pass. One way to look at a 4% accuracy rate is that you really shouldn't rely on that person's predictions. Another way is that, if they made 500 predictions, they're a great predictor because they made 20 accurate predictions. Since almost no one will bother to go through a list of predictions to figure out the overall accuracy when someone does the latter, making a huge number of predictions and then cherry picking the ones that were accurate is a good strategy for becoming a renowned futurist.

But if we want to figure out how to make accurate predictions, we'll have to look at other people's strategies. There are people who do make fairly good, generally directionally accurate, predictions, as we noted when we looked at Steve Yegge's prediction record. However, they tend to be harsh critics of their predictions, as Steve Yegge was when he reviewed his own prediction record, saying:

I saw the HN thread about Dan Luu's review of this post, and felt people were a little too generous with the scoring.

It's unsurprising that a relatively good predictor of the future scored himself lower than I did because taking a critical eye to your own mistakes and calling yourself out for mistakes that are too small for most people to care about is a great way to improve. We can see in communications from Microsoft leadership as well, e.g., calling themselves out for failing to predict that a lack of backwards compatibility doomed major efforts like OS/2 and LanMan. Doing what most futurists do and focusing on the predictions that worked out without looking at what went wrong isn't such a great way to improve.

Cocktail party understanding

Another thing we see among people who make generally directionally correct predictions, as in the Steve Yegge post mentioned above, Nathan Myhrvold's 1993 "Road Kill on the Information Highway", Bill Gates's 1995 "The Internet Tidal Wave", etc., is that the person making the prediction actually understands the topic. In all of the above examples, it's clear that the author of the document has a fairly strong technical understanding of the topics being predicted and, in the general case, it seems that people who have relatively accurate predictions are really trying to understand the topic, which is in stark contrast to the futurists discussed in this post, almost all of whom display clear signs of having a having a buzzword level understanding2 of the topics they're discussing.

There's a sense in which it isn't too difficult to make correct predictions if you understand the topic and have access to the right data. Before joining a huge megacorp and then watching the future unfold, I thought documents like "Road Kill on the Information Highway" and "The Internet Tidal Wave" were eerily prescient, but once I joined Google in 2013, a lot of trends that weren't obvious from the outside seemed fairly obvious from the inside.

For example, it was obvious that mobile was very important for most classes of applications, so much so that most applications that were going to be successful would be "mobile first" applications where the web app was secondary, if it existed at all, and from the data available internally, this should've been obvious going back at least to 2010. Looking at what people were doing on the outside, quite a few startups in areas where mobile was critical were operating with a 2009 understanding of the future even as late as 2016 and 2017, where they focused on having a web app first and had no mobile app and a web app that was unusable on mobile. Another example of this is that, in 2012, quite a few people at Google independently wanted Google to make very large bets on deep learning. It seemed very obvious that deep learning was going to be a really big deal and that it was worth making a billion dollar investment in a portfolio of hardware that would accelerate Google's deep learning efforts.

This isn't to say that the problem is trivial — many people with access to the same data still generally make incorrect predictions. A famous example is Ballmer's prediction that "There’s no chance that the iPhone is going to get any significant market share. No chance."3 Ballmer and other MS leadership had access to information as good as MS leadership from a decade earlier, but many of their predictions were no better than the futurists we discussed here. And with the deep learning example above, a competitor with the same information at Google totally whiffed and kept whiffing for years, even with the benefit of years of extra information; they're still well behind Google now, a decade later, due to their failure to understand how to enable effective, practical, deep learning R&D.

Assuming high certainty

Another common cause of incorrect predictions was having high certainty. That's a general problem that's magnified when making predictions from looking at past exponential growth and extrapolating to the future both because mispredicting the timing of a large change in exponential growth can have a very large impact and also because relatively small sustained changes in exponential growth can also have a large impact. An example that exposed these weaknesses for a large fraction of our futurists was their interpretation of Moore's law, which many interpreted as a doubling of every good thing and/or halving of every bad thing related to computers every 18 months. That was never what Moore's law predicted in the first place, but it was a common pop-conception of Moore's law. One thing that's illustrative about that is that predictors who were writing in the late 90s and early 00s still made these fantastical Moore's law "based" predictions even though it was such common knowledge that both single-threaded computer performance and Moore's law would face significant challenges that this was taught in undergraduate classes at the time. Any futurist who spent a few minutes talking to an expert in the area or even an undergrad would've seen that there's a high degree of uncertainty about computer performance scaling, but most of the futurists we discuss either don't do that or ignore evidence that would add uncertainty to their narrative4.

As computing power increases, all constant-factor inefficiencies ("uses twice as much RAM", "takes three times as many RISC operations") tend to be ground under the heel of Moore's Law, leaving polynomial and exponentially increasing costs as the sole legitimate areas of concern. Flare, then, is willing to accept any O(C) inefficiency (single, one-time cost), and is willing to accept most O(N) inefficiencies (constant-factor costs), because neither of these costs impacts scalability; Flare programs and program spaces can grow without such costs increasing in relative significance. You can throw hardware at an O(N) problem as N increases; throwing hardware at an O(N**2) problem rapidly becomes prohibitively expensive.

For computer scaling in particular, it would've been possible to make a reasonable prediction about 2022 computers in, say, 2000, but it would've had to have been a prediction about the distribution of outcomes which had a lot of weight on severely reduced performance gains in the future with some weight on a portfolio of possibilities that could've resulted in continued large gains. Someone making such a prediction would've had to, implicitly or explicitly, been familiar with ITRS semiconductor scaling roadmaps of the era as well as recent causes of recent misses (my recollection from reading roadmaps back then was that, in the short term, companies had actually exceeded recent scaling predictions, but via mechanisms that were not expected to be scalable into the future) as well as things that could unexpectedly keep semiconductor scaling on track. Furthermore, such a predictor would also have to be able to evaluate architectural ideas that might have panned out to rule them out or assign them a low probability, such as dataflow processors, the basket of techniques people were working on in order to increase ILP in order an attempt to move from the regime Tjaden and Flynn discussed in their classic 1970 and 1973 papers on ILP to the something closer to the bound discussed by Riseman and Foster in 1972 and later by Nicolau and Fisher in 1984, etc.

Such a prediction would be painstaking work for someone who isn't in the field because of the sheer number of different things that could have impacted computer scaling. Instead of doing this, futurists relied heavily on the pop-understanding they had about semiconductors. Kaku was notable among futurists under discussion for taking seriously the idea that Moore's law wasn't smooth sailing in the future, but he incorrectly decided on when UV/EUV would run out of steam and also incorrectly had high certainty that some kind of more "quantum" technology would save computer performance scaling. Most other futurists who discussed computers used a line reasoning like Kurzweil's, who said that we can predict what will happen with "remarkable precision" due to the existence of "well-defined indexes":

The law of accelerating returns applies to all of technology, indeed to any evolutionary process. It can be charted with remarkable precision in information-based technologies because we have well-defined indexes (for example, calculations per second per dollar, or calculations per second per gram) to measure them

Another thing to note here is that, even if you correctly predict an exponential curve of something, understanding the implications of that precise fact also requires an understanding of the big picture which was shown by people like Yegge, Gates, and Myhrvold but not by the futurists discussed here. An example of roughly getting a scaling curve right but mispredicting the outcome was Dixon on the number of phone lines people would have in their homes. Dixon at least roughly correctly predicted the declining cost of phone lines but incorrectly predicted that this would result in people having many phone lines in their house despite also believing that digital technologies and cell phones would have much faster uptake than they did. With respect to phones, another missed prediction, one that came from not having an understanding of the mechanism was his prediction that the falling cost of phone calls would mean that tracking phone calls would be so expensive relative to the cost of calls that phone companies wouldn't track individual calls.

For someone who has a bit of understanding about the underlying technology, this is an odd prediction. One reason the prediction seems odd is that the absolute cost of tracking who called whom is very small and the rate at which humans make and receive phone calls is bounded at a relatively low rate, so even if the cost of metadata tracking were very high compared to the cost of the calls themselves, the absolute cost of tracking metadata would still be very low. Another way to look at it would be to look at the number of bits of information transferred during a phone call vs. the number of bits of information necessary to store call metadata and the cost of storing that long enough to bill someone on a per-call basis. Unless medium-term storage became relatively more expensive than network by a mind bogglingly large factor, it wouldn't be possible for this prediction to be true and Dixon also implicitly predicted exponentially falling storage costs via his predictions on the size of available computer storage with a steep enough curve that this criteria shouldn't be satisfied and, if it were to somehow be satisfied, the cost of storage would still be so low as to be negligible.

Panacea thinking

Another common issue is what Waleed Khan calls panacea thinking, where the person assumes that the solution is a panacea that is basically unboundedly great and can solve all problems. We can see this for quite a few futurists who were writing up until the 70s, where many assumed that computers would be able to solve any problem that required thought, computation, or allocation of resources and that resource scarcity would become irrelevant. But it turns out that quite a few problems don't magically get solved because powerful computers exist. For example, the 2008 housing crash created a shortfall of labor for housing construction that only barely got back to historical levels just before covid hit. Having fast computers neither prevented this nor fixed this problem after it happened because the cause of the problem wasn't a shortfall of computational resources. Some other topics to get this treatment are "nanotechnology", "quantum", "accelerating growth" / "decreased development time", etc.

A closely related issue that almost every futurist here fell prey to is only seeing the upside of technological advancements, resulting in a kind of techno utopian view of the future. For example, in 2005, Kurzweil wrote:

The current disadvantages of Web-based commerce (for example, limitations in the ability to directly interact with products and the frequent frustrations of interacting with inflexible menus and forms instead of human personnel) will gradually dissolve as the trends move robustly in favor of the electronic world. By the end of this decade, computers will disappear as distinct physical objects, with displays built in our eyeglasses, and electronics woven in our clothing, providing full-immersion visual virtual reality. Thus, "going to a Web site" will mean entering a virtual-reality environment—at least for the visual and auditory senses—where we can directly interact with products and people, both real and simulated.

Putting aside the bit about how non-VR interfaces about computers would disappear before 2010, it's striking how Kurzweil assumes that technological advancement will mean that corporations make experiences better for consumers instead of providing the same level of experience at a lower cost or a worse experience at an even lower cost.5

Although that example is from Kurzweil, we can see the same techno utopianism in the other authors on Wikipedia's list with the exception of Zerzan, whose predictions I didn't tally up because prediction wasn't really his shtick. For example, a number of other futurists combined panacea thinking with techno utopianism to predict that computers would cause things to operate with basically perfect efficiency without human intervention, allowing people at large to live a life of leisure. Instead, the benefits to the median person in the U.S. are subtle enough that people debate whether or not life has improved at all for the median person. And on the topic of increased efficiency, a number of people predicted an extreme version of just-in-time delivery that humanity hasn't even come close to achieving and described its upsides, but no futurist under discussion mentioned the downsides of a world-wide distributed just-in-time manufacturing system and supply chain, which includes increased fragility and decreased robustness, notably impacted quite a few industries from 2020 through at least 2022 due to covid despite the worldwide system not being anywhere near as just-in-time or fragile as a number of futurists predicted.

Though not discussed here because they weren't on Wikipedia's list of notable futurists, there are pessimistic futurists such as Jaron Lanier and Paul Ehrlich. From a quick informal look at relatively well-known pessimistic futurists, it seems that pessimistic futurists haven't been more accurate than optimistic futurists. Many made predictions that were too vague to score and the ones who didn't tended to predict catastrophic collapse or overly dystopian futures which haven't materialized. Fundamentally, dystopian thinkers made the same mistakes as utopian thinkers. For example, Paul Ehrlich fell prey to the same issues utopian thinkers fell prey to and he still maintains that his discredited book, The Population Bomb, was fundamentally correct, just like utopian futurists who maintain that their discredited work is fundamentally correct.

Ehrlich's 1968 book opened with

The battle to feed all of humanity is over. In the 1970s the world will undergo famines — hundreds of millions of people are going to starve to death in spite of any crash programs embarked upon now. At this late date nothing can prevent a substantial increase in the world death rate, although many lives could be saved through dramatic programs to "stretch" the carrying capacity of the earth by increasing food production. But these programs will only provide a stay of execution unless they are accompanied by determined and successful efforts at population control. Population control is the conscious regulation of the numbers of human beings to meet the needs, not just of individual families, but of society as a whole.

Nothing could be more misleading to our children than our present affluent society. They will inherit a totally different world, a world in which the standards, politics, and economics of the 1960s are dead.

When this didn't come to pass, he did the same thing as many futurists we looked at and moved the dates on his prediction, changing the text in the opening of his book from "1970s" to "1970s and 1980s". Ehrlich then wrote a new book with even more dire predictions in 1990.

And then later, Ehrlich simply denied ever having made predictions, even though anyone who reads his book can plainly see that he makes plenty of statements about the future with no caveats about the statements being hypothetical:

Anne and I have always followed UN population projections as modified by the Population Reference Bureau — so we never made "predictions," even though idiots think we have.

Unfortunately for pessimists, simply swapping the sign bit on panacea thinking doesn't make predictions more accurate.

Evidence free assumptions

Another major source of errors among these futurists was making an instrumental assumption without any supporting evidence for it. A major example of this is Fresco's theory that you can predict the future by starting from people's values and working back from there, but he doesn't seriously engage with the idea of how people's values can be predicted. Since those are pulled from his intuition without being grounded in evidence, starting from people's values creates a level of indirection, but doesn't fundamentally change the problem of predicting what will happen in the future.

Fin

A goal of this project is to look at current predictors to see who's using methods that have historically had a decent accuracy rate, but we're going to save that for a future post. I normally don't like splitting posts up into multiple parts, but since this post is 30k words (the number of words in a small book, and more words than most pop-sci books have once you remove the pop stories) and evaluating futurists is relatively self-contained, we're going to stop with that (well, with a bit of an evaluation of some longtermist analyses that overlap with this post in the appendix)6.

In terms of concrete takeaways, you could consider this post a kind of negative result that supports the very boring idea that you're not going to get very far if you make predictions on topics you don't understand, whereas you might be able to make decent predictions if you have (or gain) a deep expertise of a topic and apply well-honed intuition to predict what might happen. We've looked at, in some detail, a number of common reasoning errors that cause predictions to miss at a high rate and also taken a bit of a look into some things that have worked for creating relatively accurate predictions.

A major caveat about what's worked is that while using high-level techniques that work poorly is a good way to generate poor predictions, using high-level techniques that work well doesn't mean much because the devil is in the details and, as trite as this is to say, you really need to think about things. This is something that people who are serious about looking at data often preach, e.g., you'll see this theme come up on Andrew Gelman's blog as well as in Richard McElreath's Statistical Rethinking. McElreath, in a lecture targeted at social science grad students who don't have a quantitative background, likens statistical methods to a golem. A golem will mindlessly do what you tell it to do, just like statistical techniques. There's no substitute for using your brain to think through whether or not it's reasonable to apply a particular statistical technique in a certain way. People often seem to want to use methods as a talisman to ward off incorrectness, but that doesn't work.

We see this in the longtermist analyses we examine in the appendix which claim to be more accurate than "classical" futurists analyses because they, among other techniques, state probabilities, which the literature on forecasting (e.g., Tetlock's Superforecasting) says that one should do. But the analyses fundamentally use the same techniques as the futurists analyses we looked at here and then add a few things on top that are also things that people who make accurate predictions do. This is backwards. Things like probabilities need to be a core part of modelling, not something added afterwards. This kind of backwards reasoning is a common error when doing data analysis and I would caution readers who think they're safe against errors because their analyses can, at a high level, be described roughly similarly to good analyses7. An obvious example of this would be the Bill Gates review we looked at. Gates asked a lot of questions and scribbled quite a few notes in the margins, but asking a lot of questions and scribbling notes in the margins of docs doesn't automatically cause you to have a good understanding of the situation. This example is so absurd that I don't think anyone even remotely reasonable would question it, but most analyses I see (of the present as well as of the future) make this fundamental error in one way or another and, as Fabian Giesen might say, are cosplaying what a rigorous analysis looks like.

Thanks to nostalgebraist, Arb Research (Misha Yagudin, Gavin Leech), Laurie Tratt, Fabian Giesen, David Turner, Yossi Kreinin, Catherine Olsson, Tim Pote, David Crawshaw, Jesse Luehrs, @TyphonBaalAmmon, Jamie Brandon, Tao L., Hillel Wayne, Qualadore Qualadore, Sophia, Justin Blank, Milosz Danczak, Waleed Khan, Mindy Preston, @ESRogs, Tim Rice, and @s__video for comments/corrections/discussion (and probably some others I forgot because this post is so long and I've gotten so many comments).

Update / correction: an earlier version of this post contained this error, pointed out by ESRogs. Although I don't believe the error impacts the conclusion, I consider it a fairly major error. If we were doing a tech-company style postmortem, that it doesn't significantly impact the conclusion would be included in the "How We Got Lucky" section of the postmortem. In particular, this was a "lucky" error because the error was made when picking out a few examples from a large portfolio of errors to give examples of one predictors errors, so a single incorrect error doesn't change the conclusion since another error could be substituted in and, even if no other error were substituted, the reasoning quality of the reasoning being evaluated still looks quite low. But, incorrect concluding that something is an error could lead to a different conclusion in the case of a predictor who made few or no errors, which is why this was a lucky mistake for me to make.

Appendix: brief notes on Superforecasting

See also, this Tetlock interview with Tyler Cowen if you don't want to read the whole book, although the book is a very quick read because it's written the standard pop-sci style, with a lot of anecdotes/stories.

On the people we looked at vs. the people Tetlock looked at, the predictors we looked at are operating in a very different style from the folks studied in the studies that led to the Superforecasting book. Both futurists and tech leaders were trying to predict a vision for the future whereas superforecasters were asked to answer very specific questions.

Another major difference among the accurate predictors is that the accurate predatictors we looked at (other than Caplan) had very deep expertise in their fields. This may be one reason for the difference in timelines here, where it appears that some of our predictors can predict things more than 3-5 years out, contra Tetlock's assertion. Another difference is in the kind of thing being predicted — a lot of the predictions we're looking at here are fundamentally whether or not a trend will continue or if a nascent trend will become a long-running trend, which seems easier than a lot of the questions Tetlock had his forecasters try to answer. For example, in the opening of Superforecasting, Tetlock gives predicting the Arab Spring as an example of something that would've been practically impossible — while the conditions for it had been there for years, the proximal cause of the Arab Spring was a series of coincidences that would've been impossible to predict. This is quite different from and arguably much more difficult than someone in 1980 guessing that computers will continue to get smaller and faster, leading to handheld computers more powerful than supercomputers from the 80s.

Appendix: other evaluations

Of these, the evaluations above, the only intersection with the futurists evaluated here is Kurzweil. Holden Karnofsky says:

A 2013 project assessed Ray Kurzweil's 1999 predictions about 2009, and a 2020 followup assessed his 1999 predictions about 2019. Kurzweil is known for being interesting at the time rather than being right with hindsight, and a large number of predictions were found and scored, so I consider this study to have similar advantages to the above study. ... Kurzweil is notorious for his very bold and contrarian predictions, and I'm overall inclined to call his track record something between "mediocre" and "fine" - too aggressive overall, but with some notable hits

Karnofsky's evaluation of Kurzweil being "fine" to "mediocre" relies on these two analyses done on LessWrong and then uses a very generous interpretation of the results to conclude that Kurzweil's predictions are fine. Those two posts rate predictions as true, weakly true, cannot decide, weakly false, or false. Karnofsky then compares the number of true + weakly true to false + weakly false, which is one level of rounding up to get an optimistic result; another way to look at it is that any level other than "true" is false when read as written. This issue is magnified if you actually look at the data and methodology used in the LW analyses.

In the second post, the author, Stuart Armstrong indirectly noted that there were actually no predictions that were, by strong consensus, very true when he noted that the "most true" prediction had a mean score of 1.3 (1 = true, 2 = weakly true ... , 5 = false) and the second highest rated prediction had a mean score of 1.4. Although Armstrong doesn't note this in the post, if you look at the data, you'll see that the third "most true" prediction had a mean score of 1.45 and the fourth had a mean score of 1.6, i.e., if you round to the nearest prediction score, only 3 out of 105 predictions score "true" and 32 are >= 4.5 and score "false". Karnofsky reads Armstrong's as scoring 12% of predictions true, but the post effectively makes no comment on what fraction of predictions were scored true and the 12% came from summing up the total number of each rating given.

I'm not going to say that taking the mean of each question is the only way one could aggregate the numbers (taking the median or modal values could also be argued for, as well as some more sophisticated scoring function, an extremizing function, etc.), but summing up all of the votes across all questions results in a nonsensical number that shouldn't be used for almost anything. If every rater rated every prediction or there was a systematic interleaving of who rated what questions, then the number could be used for something (though not as a score for what fraction of predictions are accurate), but since each rater could skip any questions (although people were instructed to start rating at the first question and rate all questions until they stop, people did not do that and skipped arbitrary questions), aggregating the number of each score given is not meaningful and actually gives very little insight into what fraction of questions are true. There's an air of rigor about all of this; there are lots of numbers, standard deviations are discussed, etc., but the way most people, including Karnofsky, interpret the numbers in the post is incorrect. I find it a bit odd that, with all of the commentary of these LW posts, few people spent the one minute (and I mean one minute literally — it took me a minute to read the post, see the comment Armstrong made which is a red flag, and then look at the raw data) it would take to look at the data and understand what the post is actually saying, but as we've noted previously, almost no one actually reads what they're citing.

Coming back to Karnofsky's rating of Kurzweil as fine to mediocre, this relies on two levels of rounding. One, doing the wrong kind of aggregation on the raw data to round an accuracy of perhaps 3% up to 12% and then rounding up again by doing the comparison mentioned above instead of looking at the number of true statements. If we use a strict reading and look at the 3%, the numbers aren't so different from what we see in this post. If we look at Armstrong's other post, there are too few raters to really produce any kind of meaningful aggregation. Armstrong rated every prediction, one person rated 68% of predictions, and no one else even rated half of the 172 predictions. The 8 predictors rated 506 predictions, so the number of ratings is equivalent to having 3 raters rate all predictions, but the results are much noisier due to the arbitrary way people decided to pick predictions. This issue is much worse for the 2009 predictions than the 2019 predictions due to the smaller number of raters combined with the sparseness of most raters, making this data set fairly low fidelity; if you want to make a simple inference from the 2019 data, you're probably best off using Armstrong's ratings and discarding the rest (there are non-simple analyses one could do, but if you're going to do that, you might as well just rate the predictions yourself).

Another fundamental issue with the analysis is that it relies on aggregating votes of from a population that's heavily drawn from Less Wrong readers and the associated community. As we discussed here, it's common to see the most upvoted comments in forums like HN, lobsters, LW, etc., be statements that can clearly be seen to be wrong with no specialized knowledge and a few seconds of thought (and an example is given from LW in the link), so why should an aggregation of votes from the LW community be considered meaningful? I often see people refer to the high-level "wisdom of crowds" idea, but if we look at the specific statements endorsed by online crowds, we can see that these crowds are often not so wise. In the Arb Research evaluation (discussed below), they get around this problem by checking reviewing answers themselves and also offering a bounty for incorrectly graded predictions, which is one way to deal with having untrustworthy raters, but Armstrong's work has no mitigation for this issue.

On the Karnofsky / Arb Research evaluation, Karnofsky appears to use a less strict scoring than I do and once again optimistically "rounds up". The Arb Research report scores each question as "unambiguously wrong", "ambiguous or near miss", or "unambiguously right" but Karnofsky's scoring removes the ambiguous and near miss results, whereas my scoring only removes the ambiguous results, the idea being that a near miss is still a miss. Accounting for those reduces the scores substantially but still leaves Heinlen, Clarke, and Asimov with significantly higher scores than the futurists discussed in the body of this post. For the rest, many of the predictions that were scored as "unambiguously right" are ones I would've declined to rate for similar reasons to predictions which I declined to rate (e.g., a prediction that something "may well" happen was rated as "unambiguously right" and I would consider that unfalsifiable and therefore not include it). There are also quite a few "unambiguously right" predictions that I would rate as incorrect using a strict reading similar to the readings that you can see below in the detailed appendix.

Another place where Karnofsky rounds up is that Arb research notes that 'The predictions are usually very vague. Almost none take the form “By Year X technology Y will pass on metric Z”'. This makes the prediction accuracy from futurists Arb Research looked at not comparable to precise predictions of the kind Caplan or Karnofsky himself makes, but Karnofsky directly uses those numbers to justify why his own predictions are accurate without noting that the numbers are not comparable. Since the non-comparable numbers were already rounded up, there are two levels of rounding here (more on this later).

As noted above, some of the predictions are ones that I wouldn't rate because I don't see where the prediction is, such as this one (this is the "exact text" of the prediction being scored, according to the Arb Research spreadsheet), which was scored "unambiguously right"

application of computer technology to professional sports be counterproduc- tive? Would the public become less interested in sports or in betting on the outcome if matters became more predictable? Or would there always be enough unpredictability to keep interest high? And would people derive particular excitement from beat ing the computer when low-ranking players on a particular team suddenly started

This seems like a series of questions about something that might happen, but wouldn't be false if none of these happened, so would not count as a prediction in my book.

Similarly, I would not have rated the following prediction, which Arb also scored "unambiguously right"

its potential is often realized in ways that seem miraculous, not because of idealism but because of the practical benefits to society. Thus, the computer's ability to foster human creativity may well be utilized to its fullest, not because it would be a wonderful thing but because it will serve important social functions Moreover, we are already moving in the

Another kind of prediction that was sometimes scored "unambiguously correct" that I declined to score were predictions of the form "this trend that's in progress will become somewhat {bigger / more important}, such as the following:

The consequences of human irresponsibility in terms of waste and pollution will become more apparent and unbearable with time and again, attempts to deal with this will become more strenuous. It is to be hoped that by 2019, advances in technology will place tools in our hands that will help accelerate the process whereby the deterioration of the environment will be reversed.

On Karnofsky's larger point, that we should trust longtermist predictions because futurists basically did fine and longtermsists are taking prediction more seriously and trying harder and should therefore generate better prediction, that's really a topic for another post, but I'll briefly discuss here because of the high intersection with this post. There are two main pillars of this argument. First, that futurists basically did fine which, as we've seen, relies on a considerable amount of rounding up. And second, that the methodologies that longtermists are using today are considerably more effective than what futurists did in the past.

Karnofsky says that the futurists he looked at "collect casual predictions - no probabilities given, little-to-no reasoning given, no apparent attempt to collect evidence and weigh arguments", whereas Karnofsky's summaries use (among other things):

We've seen, when evaluating futurists with an eye towards evaluating longtermists, Karnofsky heavily rounds up in the same way Kurzweil and other futurists do, to paint the picture they want to create. There's also the matter of his summary of a report on Kurzweil's predictions being incorrect because he didn't notice the author of that report used a methodology that produced nonsense numbers that were favorable to the conclusion that Karnofsky favors. It's true that Karnofsky and the reports he cites do the superficial things that the forecasting literature notes is associated with more accurate predictions, like stating probabilities. But for this to work, the probabilities need to come from understanding the data. If you take a pile of data, incorrectly interpret it and then round up the interpretation further to support a particular conclusion, throwing a probability on it at the end is not likely to make it accurate. Although he doesn't use these words, a key thing Tetlock notes in his work is that people who round things up or down to conform to a particular agenda produce low accuracy predictions. Since Karnofsky's errors and rounding heavily lean in one direction, that seems to be happening here.

We can see this in other analyses as well. Although digging into material other than futurist predictions is outside of the scope of this post, nostalgebraist has done this and he said (in a private communication that he gave me permission to mention) that Karnofsky's summary of https://openphilanthropy.org/research/could-advanced-ai-drive-explosive-economic-growth/ is substantially more optimistic about AI timelines than the underlying report in that there's at least one major concern raised in the report that's not brought up as a "con" in Karnofsky's summary and nostalgebraist later wrote this post, where he (implicitly) notes that the methodology used in a report he examined in detail is fundamentally not so different than what the futurists we discussed used. There are quite a few things that may make the report appear credible (it's hundreds of pages of research, there's a complex model, etc.), but when it comes down to it, the model boils down to a few simple variables. In particular, a huge fraction of the variance of whether or not TAI is likely or not likely comes down to the amount of improvement will occur in terms of hardware cost, particularly FLOPS/$. The output of the model can range from 34% to 88% depending how much improvement we get in FLOPS/$ after 2025. Putting in arbitrarily large FLOPS/$ amounts into the model, i.e., the scenario where infinite computational power is free (since other dimensions, like storage and network aren't in the model, let's assume that FLOPS/$ is a proxy for those as well), only pushes the probability of TAI up to 88%, which I would rate as too pessimistic, although it's hard to have a good intuition about what would actually happen if infinite computational power were on tap for free. Conversely, with no performance improvement in computers, the probability of TAI is 34%, which I would rate as overly optimistic without a strong case for it. But I'm just some random person who doesn't work in AI risk and hasn't thought about too much, so your guess on this is as good as mine (and likely better if you're the equivalent of Yegge or Gates and work in the area).

The part about all of this that makes this fundamentally the same thing that the futurists here did is that the estimate of the FLOPS/$ which is instrumental for this prediction is pulled from thin air by someone who is not a deep expert in semiconductors, computer architecture, or a related field that might inform this estimate.

As Karnofsky notes, a number of things were done in an attempt to make this estimate reliable ("the authors (and I) have generally done calibration training, and have tried to use the language of probability") but, when you come up with a model where a single variable controls most of the variances and the estimate for that variable is picked out of thin air, all of the modeling work actually reduces my confidence in the estimate. If you say that, based on your intuition, you think there's some significant probability of TAI by 2100; 10% or 50% or 80% or whatever number you want, I'd say that sounds plausible (why not? things are improving quickly and may continue to do so) but wouldn't place any particular faith in the estimate. If you build a model where the output hinges on a relatively small number of variables and then say that there's an 80% chance, based a critical variable out of thin air, should that estimate be more or less confidence inspiring than the estimate based solely on intuition? I don't think the answer should be that output is higher confidence. The direct guess of 80% is at least honest about its uncertainty. In the model-based case, since the model doesn't propagate uncertainties and the choice of a high but uncertain number can cause the model to output a fairly certain number, like 88%, there's a disconnect between the actual uncertainty produced by the model and the probability estimate.

At one point, in summarizing the report, Karnofsky says

I consider the "evolution" analysis to be very conservative, because machine learning is capable of much faster progress than the sort of trial-and-error associated with natural selection. Even if one believes in something along the lines of "Human brains reason in unique ways, unmatched and unmatchable by a modern-day AI," it seems that whatever is unique about human brains should be re-discoverable if one is able to essentially re-run the whole history of natural selection. And even this very conservative analysis estimates a ~50% chance of transformative AI by 2100

But it seems very strong to call this a "very conservative" estimate when the estimate implicitly relies on future FLOPS/$ improvement staying above some arbitrary, unsupported, threshold. In the appendix of the report itself, it's estimated that there will be a 6 order of magnitude (OOM) improvement and that a 4 OOM improvement would be considered conservative, but why should we expect that 6 OOM is the amount of headroom left for hardware improvement and 4 OOM is some kind of conservative goal that we'll very likely reach? Given how instrumental these estimates are to the output of the model, there's a sense in which the uncertainty of the final estimate has to be at least as large as the uncertainty of these estimates multiplied by their impact on the model but that can't be the case here given the lack of evidence or justification for these inputs to the model.

More generally, the whole methodology is backwards — if you have deep knowledge of a topic, then it can be valuable to put a number down to convey the certainty of your knowledge to other people, and if you don't have deep knowledge but are trying to understand an area, then it can be valuable to state your uncertainties so that you know when you're just guessing. But here, we have a fairly confidently stated estimate (nostalgebraist notes that Karnofsky says "Bio Anchors estimates a >10% chance of transformative AI by 2036, a ~50% chance by 2055, and an ~80% chance by 2100.") that's based off of a model that's nonsense that relies on a variable that's picked out of thin air. Naming a high probability after the fact and then naming a lower number and saying that's conservative when it's based on this kind of modeling is just window dressing. Looking at Karnofsky's comments elsewhere, he lists a number of extremely weak pieces of evidence in support of his position, e.g., in the previous link, he has a laundry list of evidence of mixed strength, including Metaculus, which nostaglebraist has noted is basically worthless for this purpose here and here. It would be very odd for someone who's truth seeking on this particular issue to cite so many bad pieces of evidence; creating a laundry list of such mixed evidence is consistent with someone who has a strong prior belief and is looking for anything that will justify it, no matter how weak. That would also be consistent with the shoddy direct reasoning noted above.

Back to other evaluators, on Justin Rye's evaluations, I would grade the predictions "as written" and therefore more strictly than he did and would end up with lower scores.

For the predictors we looked at in this document who mostly or nearly exclusively give similar predictions, I declined to give them anything like a precise numerical score. To be clear, I think there's value in trying to score vague predictions and near misses, but that's a different thing than this document did, so the scores aren't directly comparable.

A number of people have said that predictions by people who make bold predictions, the way Kurzweil does, are actually pretty good. After all, if someone makes a lot of bold predictions and they're all off by 10 years, that person will have useful insights even if they lose all their bets and get taken to the cleaners in prediction markets. However, that doesn't mean that someone who makes bold predictions should always "get credit for" making bold predictions. For example, in Kurzweil's case, 7% accuracy might not be bad if he uniformly predicted really bold stuff like unbounded life span by 2011. However, that only applies if the hits and misses are both bold predictions, which was not the case in the sampled set of predictions for Kurzweil here. For Kurzweil's predictions evaluated in this document, Kurzweil's correct predictions tended to be very boring, e.g., there will be no giant economic collapse that stops economic growth, cochlear implants will be in widespread use in 2019 (predicted in 1999), etc.

The former is a Caplan-esque bet against people who were making wild predictions that there would be severe or total economic collapse. There's value in bets like that, but it's also not surprising when such a bet is successful. For the latter, the data I could quickly find on cochlear implant rates showed that implant rates slowly linearly increased from the time Kurzweil made the bet until 2019. I would call that a correct prediction, but the prediction is basically just betting that nothing drastically drops cochlear implant rates, making that another Caplan-esque safe bet and not a bet that relies on Kurzweil's ideas about the law of accelerating growth that his wild bets rely on.

If someone makes 40 boring bets of which 7 are right and another person makes 40 boring bets and 22 wild bets and 7 of their boring bets and 0 of their wild bets are right (these are arbitrary numbers as I didn't attempt to classify Kurzweil's bets as wild or not other than the 7 that were scored as correct), do you give the latter person credit for having "a pretty decent accuracy given how wild their bets were"? I would say no.

On the linked HN thread from a particular futurist, a futurist scored themselves 5 out of 10, but most HN commenters scored the same person at 0 out of 10 or, generously, at 1 out of 10, with the general comment that the person and other futurists tend to score themselves much too generously:

sixQuarks: I hate it when “futurists” cherry pick an outlier situation and say their prediction was accurate - like the bartender example.

karaterobot: I wanted to say the same thing. He moved the goal posts from things which "would draw hoots of derision from an audience from the year 2022" to things which there has been some marginal, unevenly distributed, incremental change to in the last 10 years, then said he got it about 50% right. More generally, this is the issue I have with futurists: they get things wrong, and then just keep making more predictions. I suppose that's okay for them to do, unless they try to get people to believe them, and make decisions based on their guesses.

chillacy: Reminded me of the ray [kurzweil] predictions: extremely generous grading.)

Appendix: other reading

Appendix: detailed information on predictions

Ray Kurzweil

4/59 for rated predictions. If you feel like the ones I didn't include that one could arguably include should count, then 7/62.

This list comes from wikipedia's bulleted list of Kurzweil's predictions at the time Peter Diamadis, Kurzweil's co-founder for SingularityU, cited it to bolster the claim that Kurzweil has an 86% prediction accuracy rate. Off the top of my head, this misses quite a few predictions that Kurzweil made, such as life expectancy being "over one hundred" by 2019 and 120 by 2029 (prediction made in 1999) and unbounded (life expectancy increasing at one year per year) by 2011 (prediction made in 2001), that a computer would beat the top human in chess by 2000 (prediction made in 1990).

It's likely that Kurzweil's accuracy rate would change somewhat if we surveyed all of his predictions, but it seems extremely implausible for the rate to hit 86% and, more broadly, looking at Kurzweil's vision of what the world would be like, it also seems impossible that we live in a world that's generally close to Kurzweil's imagined future.

The list above only uses the bulleted predictions from Wikipedia under the section that has per-timeframe sections. If you pull in other ones from the same page that could be evaluated, which includes predictions like " "nanotechnology-based" flying cars would be available [by 2026]", this doesn't hugely change the accuracy rate (and actually can't due to the relatively small number of other predictions).

Jacque Fresco

The foreword to Fresco's book gives a pretty good idea of what to expect from Fresco's predictions:

Looking forward is an imaginative and fascinating book in which the authors take you on a journey into the culture and technology of the twenty-first century. After an introductory section that discusses the "Things that Shape Your Future." you will explore the whys and wherefores of the unfamiliar, alarming, but exciting world of a hundred years from now. You will see this society through the eyes of Scott and Hella, a couple of the next century. Their living quarters are equipped with a cybernator. a seemingly magical computer device, but one that is based on scientific principles now known. It regulates sleeping hours, communications throughout the world, an incredible underwater living complex, and even the daily caloric intake of the "young" couple. (They are in their forties but can expect to live 200 years.) The world that Scott and Hella live in is a world that has achieved full weather control, has developed a finger-sized computer that is implanted in the brain of every baby at birth (and the babies are scientifically incubated the women of the twenty-first century need not go through the pains of childbirth), and that has perfected genetic manipulation that allows the human race to be improved by means of science. Economically, the world is Utopian by our standards. Jobs, wages, and money have long since been phased out. Nothing has a price tag, and personal possessions are not needed. Nationalism has been surpassed, and total disarmament has been achieved; educational technology has made schools and teachers obsolete. The children learn by doing, and are independent in this friendly world by the time they are five.

The chief source of this greater society is the Correlation Center, "Corcen," a gigantic complex of computers that serves but never enslaves mankind. Corcen regulates production, communication, transportation and all other burdensome and monotonous tasks of the past. This frees men and women to achieve creative challenging experiences rather than empty lives of meaningless leisure. Obviously this book is speculative, but it is soundly based upon scientific developments that are now known

As mentioned above, Fresco makes the claim that it's possible to predict the future and to do so, one should start with the values people will have in the future. Many predictions are about "the 21st century" so can arguably be defended as still potentially accurate, although the way the book talks about the stark divide between "the 20th century" and "the 21st century", we should have already seen the changes mentioned in the book since we're no longer in "the 20th century" and the book makes no reference to a long period of transition in between. Fresco does make some specific statements about things that will happen by particular dates, which are covered later. For "the 21st century", his predictions from the first section of his book are:

As mentioned above, the next part of Fresco's prediction is about how science will work. He writes about how "the scientific method" is only applied in a limited fashion, which led to thousands of years of slow progress. But, unlike in the 20th century, in the 21st century, people will be free from bias and apply "the scientific method" in all areas of their life, not just when doing science. People will be fully open to experimentation in all aspects of life and all people will have "a habitual open-mindedness coupled with a rigid insistence that all problems be formulated in a way that permits factual checking".

This will, among other things, lead to complete self-knowledge of one's own limitations for all people as well as an end to unhappiness due to suboptimal political and social structures:

The success of the method of science in solving almost every problem put to it will give individuals in the twenty-first century a deep confidence in its effectiveness. They will not be afraid to experiment with new ways of feeling, thinking, and acting, for they will have observed the self-corrective aspect of science. Science gives us the latest word, not the last word. They will know that if they try something new in personal or social life, the happiness it yields can be determined after sufficient experience has accumulated. They will adapt to changes in a relaxed way as they zigzag toward the achievement of their values. They will know that there are better ways of doing things than have been used in the past, and they will be determined to experiment until they have found them. They will know that most of the unhappiness of human beings in the mid-twentieth century was not due to the lack of shiny new gadgets; it was due, in part, to not using the scientific method to check out new political and social structures that could have yielded greater happiness for them

After discussing, at a high level, the implications on people and society, Fresco gets into specifics, saying that doing everything with computers, what Fresco calls a "cybernated" society, could be achieved by 1979, giving everyone a post-tax income of $100k/yr in 1969 dollars (about $800k/yr in 2022 dollars):

How would you like to have a guaranteed life income of $100,000 per year—with no taxes? And how would you like to earn this income by working a three-hour day, one day per week, for a five-year period of your life, providing you have a six-months vacation each year? Sound fantastic? Not at all with modern technology. This is not twenty-first-century pie-in-the-sky. It could probably be achieved in ten years in the United States if we applied everything we now know about automation and computers to produce a cybernated society. It probably won't be done this rapidly, for it would take some modern thinking applied in an intelligent crash program. Such a crash program was launched to develop the atomic bomb in a little over four years.

Other predictions about "cybernation":

Michio Kaku

That gives a prediction rate of 3%. I stopped reading at this point, so may have missed a number of correct predictions. But, even if the rest of the book was full of correct predictions, the correct prediction rate is likely to be low.

There were also a variety of predictions that I didn't include because they were statements that were true in the present. For example

If the dirt road of the Internet is made up of copper wires, then the paved information highway will probably be made of laser ber optics. Lasers are the perfect quantum device, an instrument which creates beams of coherent light (light beams which vibrate in exact synchronization with each other). This exotic form of light, which does not occur naturally in the universe, is made possible by manipulating the electrons making quantum jumps between orbits within an atom

This doesn't seem like much of a prediction since, when the book was written, the "information highway" already used a lot of fiber. Throughout the book, there's a lot of mysticism around quantum-ness which is, for example, on display above and cited as a reason that microprocesses will become obsolete by 2020 (they're not "quantum") and fiber optics won't (it's quantum):

John Naisbitt

Here are a few quotes that get at the methodology of Naisbitt's hit book, Megatrends:

For the past fifteen years, I have been working with major American corporations to try to understand what is really happening in the United States by monitoring local events and behavior, because collectively what is going on locally is what is going on in America.

Despite the conceits of New York and Washington, almost nothing starts there.

In the course of my work, 1 have been overwhelmingly impressed with the extent to which America is a bottom-up society, that is, where new trends and ideas begin in cities and local communities—for example, Tampa, Hartford, San Diego, Seattle, and Denver, not New York City or Washington, D.C. My colleagues and I have studied this great country by reading its local newspapers. We have discovered that trends are generated from the bottom up, fads from the top down. The findings in this book are based on an analysis of more than 2 million local articles about local events in the cities and towns of this country during a twelve-year period.

Out of such highly localized data bases, I have watched the general outlines of a new society slowly emerge.

We learn about this society through a method called content analysis, which has its roots in World War II. During that war, intelligence experts sought to find a method for obtaining the kinds of information on enemy nations that public opinion polls would have normally provided.

Under the leadership of Paul Lazarsfeld and Harold Lasswell, later to become well-known communication theorists, it was decided that we would do an analysis of the content of the German newspapers, which we could get—although some days after publication. The strain on Germany's people, industry, and economy be- gan to show up in its newspapers, even though information about the country's supplies, production, transportation, and food situation remained secret. Over time, it was possible to piece together what was going on in Germany and to figure out whether conditions were improving or deteriorating by carefully tracking local stories about factory openings, clos- ings, and production targets, about train arrivals, departures, and delays, and so on. ... Although this method of monitoring public behavior and events continues to be the choice of the intelligence community—the United States annually spends millions of dollars in newspaper content analysis in various parts of the world it has rarely been applied commercially. In fact. The Naisbitt Group is the first, and presently the only, organization to utilize this approach in analyzing our society.

Why are we so confident that content analysis is an effective way to monitor social change? Simply stated, because the news hole in a newspaper is a closed system. For economic reasons, the amount of space devoted to news in a newspaper does not change significantly over time. So, when something new is introduced, something else or a combination of things must be omitted. You cannot add unless you subtract. It is the principle of forced choice in a closed system.

In this forced-choice situation, societies add new preoccupations and forget old ones. In keeping track of the ones that are added and the ones that are given up, we are in a sense measuring the changing share of the market that competing societal concerns command.

Evidently, societies are like human beings. A person can keep only so many problems and concerns in his or her head or heart at any one time. If new problems or concerns are introduced, some existing ones are given up. All of this is reflected in the collective news hole that becomes a mechanical representation of society sorting out its priorities.

Naisbitt rarely makes falsifiable predictions. For example, on the "information society", Naisbitt says

In our new information society, the time orientation is to the future. This is one of the reasons we are so interested in it. We must now learn from the present how to anticipate the future. When we can do that, we will understand that a trend is not destiny; we will be able to learn from the future the way we have been learning from the past.

This change in time orientation accounts for the growing popular and professional interest in the future during the 1970s. For example, the number of universities offering some type of futures-oriented degree has increased from 2 in 1969 to over 45 in 1978. Membership in the World Future Society grew from 200 in 1967 to well over 30,000 in 1982, and the number of popular and professional periodicals devoted to un- derstanding or studying the future has dramatically increased from 12 in 1965 to more than 122 in 1978.

This could be summed up as "in the future, people will think more about the future". Pretty much any case one might make that Naisbitt's claims ended up being true or false could be argued against.

In the chapter on the "information society", one of the most specific predictions is

New information technologies will at first be applied to old industrial tasks, then, gradually, give birth to new activities, processes, and products.

I'd say that this is false in the general case, but it's vague enough that you could argue it's true.

A, rare, falsifiable comment is this prediction about the price of computers

The home computer explosion is upon us. soon to be followed by a software implosion to fuel it. It is projected that by the year 2000, the cost of a home computer system (computer, printer, monitor, modem, and so forth) should only be about that of the present telephone-radio-recorder-television system.

From a quick search, it seems that reference devices cost something like $300 in 1982? That would be $535 in 2000, which wasn't really a reasonable price for a computer as well as the peripherals mentioned and implied by "and so forth".

Gerard K. O'Neill

We discussed O'Neill's predictions on space colonization in the body of this post. This section contains a bit on his other predictions.

On computers, O'Neill says that in 2081 "any major central computer will have rapid access to at least a hundred million million words of memory (the number '1' followed by 14 zeros). A computer of that memory will be no larger than a suitcase. It will be fast enough to carry out a complete operation in more more time than it takes light to travel from this page to your eye, and perhaps a tenth of that time", which is saying that a machine will have 100TWords of RAM or, to round things up simply, let's say 1PB of RAM and a clock speed of something between 300 MHz and 6 GHz, depending on how far away from your face you hold a book.

On other topics, O'Neill predicts we'll have fully automated manufacturing, people will use 6 times as much energy per capita in 2081 as in 1980, pollution other than carbon dioxide will be a solved problem, coal plants will still be used, most (50% to 95%) of energy will be renewable (with the caveat that "ground-based solar" is a "myth" that can never work, and that wind, tide, and hydro are all forms of solar that even combined with geothermal thrown in, can't reasonably provide enough energy), that solar power from satellites is the answer to then-current and future energy needs.

In The Technology Edge, O'Neill makes predictions for the 10 years following the book's publication in 1983. O'Neill says "the book is primarily based on interviews with chief executives". It was written at a time when many Americans were concerned about the impending Japanese dominance of the world. O'Neill says

As an American, I cannot help being angry — not at the Japanese for succeeding, but at the forces of timidity, shortsightedness, greed, laziness and misdirection here in America that have mired us down so badly in recent years, sapped our strength and kept us from equal achievements.

As we will see, opportunities exist now for the opening of whole new industries that can become even greater than those we have lost to the Japanese. Are we to delay and lose those too?

In an interview about the book, O'Neill said

microengineering, robotics, genetic engineering, magnetic flight, family aircraft, and space science. If the U.S. does not compete successfully in these areas, he warns, it will lose the technological and economic leadership it has enjoyed.

This seems like a big miss with both serious false positives as well as false negatives. O'Neill failed to cite industries that ended up being important to the then-continued U.S. dominance of the world economy, e.g, software, and also predicted that space and flight were much more important than they turned out to be.

On the specific mechanism, O'Neill also generally misses, e.g., in the book, O'Neill cites the lack of U.S. PhD production and people heading directly into industry as a reason the U.S. was falling behind and would continue to fall behind Japan, but in a number of important industries, like software, a lot of the major economic/business contributions have been made by people going to industry without a PhD. The U.S. didn't need to massively increase PhD production in the decades following 1983 to stay economically competitive.

There's quite a bit of text dedicated to a commonly discussed phenomenon at the time, how Japanese companies are going to wipe the floor with American and European companies because they're able to make and execute long-term plans, unlike American companies. I'll admit that it's a bit of a mystery to me how short-term thinking has worked so well for American companies and I would've, at least to date.

Patrick Dixon

Dixon opens with:

The next millennium will witness the greatest challenges to human survival ever in human history, and many of them will face us in the early years of its first century ...

The future has six faces, each of which will have a dramatic effect on all of us in the third millennium ... [Fast, Urban, Radical, Universal, Tribal, Ethical, which spells out FUTURE]

Out of these six faces cascade over 500 key expectations, specific predictions as logical workings-out of these important global trends. These range from inevitable to high probability to lower probability — but still significant enough to require strategic planning and personal preparation.

That's the end of the introduction. Some of these predictions are arguably too early to call since, in places, Dixon write as if Futurewise is about the entire "third millenia", but Dixon also notes that drastic changes are expected in the first years and decades of the 21st century and these generally have not come to pass, both the specific cases where Dixon calls out particular timelines or in the cases where Dixon doesn't name a particular timeline. In general, I'm trying to only include predictions where it seems that Dixon is referring to the 2022 timeframe or before, but his general vagueness makes it difficult to make the right call 100% of the time.

The next chapter is titled "Fast" and is about the first of the six "faces" of the future.

This marks the end of the "Fast" chapter. From having skimmed the rest of the book, the hit rate isn't really higher later nor is the style of reasoning any different, so I'm going to avoid doing a prediction-by-prediction grading. Instead, I'll just mention a few highlights (some quite accurate, but mostly not; not included in the prediction accuracy rate since I didn't ensure consistent or random sampling):

Overall accuracy, 8/79 = 10%

Toffler

Intro to Future Shock:

Another reservation has to do with the verb "will." No serious futurist deals in "predictions." These are left for television oracles and newspaper astrologers. ... Yet to enter every appropriate qualification in a book of this kind would be to bury the reader under an avalanche of maybes. Rather than do this, I have taken the liberty of speaking firmly, without hesitation, trusting that the intelligent reader will understand the stylistic problem. The word "will" should always be read as though it were preceded by "probably" or "in my opinion." Similarly, all dates applied to future events need to be taken with a grain of judgment.

[Chapter 1 is about how future shock is going to be a big deal in the future and how we're presently undergoing a revolution]

Despite the disclaimer in the intro, there are very few concrete predictions. The first that I can see is in the middle of chapter two and isn't even really a prediction, but is a statement that very weakly implies world population growth will continue at the same pace or accelerate. Chapter 1 has a lot of vague statements about how severe future shock will be, and then Chapter 2 discusses how the world is changing at an unprecedented rate and cite a population doubling time eleven years to note how much this must change the world since it would require the equivalent of a new Tokyo, Hamburg, Rome, and Rangoon in eleven years, illustrating how shockingly rapid the world is changing. There's a nod to the creation of future subterranean cities, but stated weakly enough that it can't really be called a prediction.

There's a similar implicit prediction that economic growth will continue with a doubling time of fifteen years, meaning that by the time someone is thirty, the amount of stuff (and it's phrased as amount of stuff and not wealth) will have quadrupled and then by the time someone is seventy it will have increased by a factor of thirty two. This is a stronger implicit prediction than the previous one since the phrasing implies this growth rate should continue for at least seventy years and is perhaps the first actual prediction in the book.

Another such prediction appears later in the chapter, on the speed of travel, which took millions of years to reach 100 mph in the 1880s, only fifty-eight years to reach 400 mph in 1938, and then twenty to double again, and then not much more time before rockets could propel people at 4000 mph and people circled the earth at 18000 mph. Strictly speaking, no prediction is made as to the speed of travel in the future, but since the two chapters are about how this increased rate of change will, in the future, cause future shock, citing examples where exponential growth is expected to level off as reasons the future is going to cause future shock would be silly and implicit in the citation is that the speed of travel will continue to grow.

Toffler then goes on to cite a series of examples where, at previous times in history, the time between having an idea and applying the idea was large, shrinking as we get closer to the present, where it's very low because "we have, with the passage of time, invented all sorts of social devices to hasten the process".

Through Chapter 4, Toffler continued to avoid making concrete, specific predictions, but also implied that buildings would be more temporary and, in the United States specifically, there would be an increase in tearing down old buildings (e.g., ten year old apartment buildings) to build new ones because new buildings would be so much better than old ones that it wouldn't make sense to live in old buildings, and that schools will move to using temporary buildings that are quickly dismantled after they're no longer necessary, perhaps often using geodesic domes.

Also, a general increase in modularity, which parts of buildings being swapped out to allow more rapid changes during the short, 25-year life, of a modern building.

Another implied prediction is that everything will be rented instead of owned, with specific examples cited of cars and homes, with an extremely rapid growth in the rate of car rentership over ownership continuing through the 70s in the then-near future.

Through Chapter 5, Toffler continued to avoid making specific predictions, but very strongly implies that the amount of travel people will do for mundane tasks such as committing will hugely increase, making location essentially irrelevant. As with previous implied predictions, this is based on a very rapid increase in what Toffler views as a trend and is implicitly a prediction of the then very near future, citing people who commute 50k miles in a year and 120 miles in a day and citing stats showing that miles traveled have been increasing. When it comes to an actual prediction, Toffler makes the vague comment

among those I have characterized as "the people of the future," commuting, traveling, and regularly relocating one's family have become second nature.

Which, if read very strictly, not technically not a prediction about the future, although it can be implied that people in the future will commute and travel much more.

In a similar implicit prediction, Toffler implies that, in the future, corporations will order highly skilled workers to move to whatever location most benefits the corporation and they'll have no choice but to obey if they want to have a career.

In Chapter 6, in a rare concrete prediction, Toffler writes

When asked "What do you do?" the super-industrial man will label himself not in terms of his present (transient) job, but in terms of his trajectory type, the overall pattern of his work life.

Some obsolete example job types that Toffler presents are "machine operator", "sales clerk", and "computer programmer". Implicit in this section is that career changes will be so rapid and so frequent that the concept of being "a computer programmer" will be meaningless in the future. It's also implied that the half-life of knowledge will be so short in the future that people will no longer accumulate useful knowledge over the course of their career in the future and people, especially in management, shouldn't expect to move up with age and may be expected to move down with age as their knowledge becomes obsolete and they end up in "simpler" jobs.

It's also implied that more people will work for temp agencies, replacing what would previously have been full-time roles. The book is highly U.S. centric and, in the book, this is considered positive for workers (it will give people more flexibility) without mentioning any of the downsides (lack of benefits, etc.). The chapter has some actual explicit predictions about how people will connect to family and friends, but the predictions are vague enough that it's difficult to say if the prediction has been satisfied or not.

In chapter 7, Toffler says that bureaucracies will be replaced by "adhocracies". Where bureaucracies had top down power and put people into well-defined roles, in adhocracies, roles will change so frequently that people won't get stuck into defined roles.. Toffler notes that a concern some people have about the future is that, since organizations will get larger and more powerful, people will feel like cogs, but this concern is unwarranted because adhocracy will replace bureaucracy. This will also mean an end to top-down direction because the rapid pace of innovation in the future won't leave time for any top down decision making, giving workers power. Furthermore, computers will automate all mundane and routine work, leaving no more need for bureaucracy because bureaucracy will only be needed to control large groups of people doing routine work and has no place in non-routine work. It's implied that "in the next twenty-five to fifty years [we will] participate in the end of bureaucracy". As Toffler was writing in 1970, his timeframe for that prediction is 1995 to 2020.

Chapter 8 takes the theme of everything being quicker and turns it to culture. Toffler predicts that celebrities, politicians, sports stars, famous fictional characters, best selling books, pieces of art, knowledge, etc., will all have much shorter careers and/or durations of relevance in the future. Also, new, widely used, words will be coined more rapidly than in the past.

Chapter 9 takes the theme of everything accelerating and notes that social structures and governments are poised to break down under the pressure of rapid change, as evidenced by unrest in Berlin, New York, Turin, Tokyo, Washington, and Chicago. It's possible this is what Toffler is using to take credit for predicting the fall of the Soviet Union?

Under the subheading "The New Atlantis", Toffler predicts an intense race to own the bottom of the ocean and the associated marine life there, with entire new industries springing up to process the ocean's output. "Aquaculture" will be as important as "agriculture", new textiles, drugs, etc., will come from the ocean. This will be a new frontier, akin to the American frontier, people will colonize the ocean. Toffler says "If all this sounds too far off it is sobering to note that Dr. Walter L. Robb, a scientist at General Electric has already kept a hamster alive under water by enclosing it in a box that is, in effect, an artificial gill--a synthetic membrane that extracts air from the surrounding water while keeping the water out." Toffler gives the timeline for ocean colonization as "long before the arrival of A.D. 2000".

Toffler also predicts control over the weather starting in the 70s, that "It is clearly only a matter of years" before women are able to birth children "without the discomfort of pregnancy".

I stopped reading at this point because the chapters all seem very similar to each other, applying the same reasoning to different areas and the rate of accuracy of predictions didn't seem likely to increase in later chapters.

  1. I used web.archive.org to pull an older list because the current list of futurists is far too long for people to evaluate. I clicked on an arbitrary time in the past on archive.org and that list seemed to be short enough to evaluate (though, given the length of this post, perhaps that's not really true) and then looked at those futurists. [return]
  2. While there are cases where people can make great predictions or otherwise show off expertise while making "cocktail party idea" level statements because it's possible to have a finely honed intuition without being able to verbalize the intuition, developing that kind of intuition requires taking negative feedback seriously in order to train your intuition, which is the opposite of what we observed with the futurists discussed in this post. [return]

  3. Ballmer is laughing with incredulity when he says this; $500 is too expensive for phone and will be the most expensive phone by far; a phone without a keyboard won't appeal to business users and won't be useful for writing emails; you can get "great" Windows Phone devices like the Motorola QPhone for $100, which will do everything (messaging, email, etc.), etc.

    You can see these kinds of futurist-caliber predictions all over the place in big companies. For example, on internal G+ at Google, Steve Yegge made a number of quite accurate predictions about what would happen with various major components of Google, such as Google cloud. If you read comments from people who are fairly senior, many disagreed with Yegge for reasons that I would say were fairly transparently bad at the time and were later proven to be incorrect by events. There's a sense in which you can say this means that what's going to happen isn't so obvious even with the right information, but this really depends on what you mean by obvious.

    A kind of anti-easter egg in Tetlock's Superforecasting is that Tetlock makes the "smart contrarian" case that the Ballmer quote is unjustly attacked since worldwide iPhone marketshare isn't all that high and he also claims that Ballmer is making a fairly measured statement that's been taken out of context, which seems plausible if you read the book and look at the out of context quote Tetlock uses but is obviously untrue if you watch the interview the quote comes from. Tetlock has mentioned that he's not a superforecaster and has basically said that he doesn't have the patience necessary to be one, so I don't hold this against him, but I do find it a bit funny that this bogus Freakonomics-style contrarian "refutation" is in this book that discusses, at great length, how important it is to understand the topic you're discussing.

    [return]

  4. Although this is really a topic for another post, I'll note that longtermists not only often operate with the same level of certainty, but also on the exact same topics, e.g., in 2001, noted longetermist Eliezer Yudkowsky said the following in a document describing Flare, his new programming language:

    A new programming language has to be really good to survive. A new language needs to represent a quantum leap just to be in the game. Well, we're going to be up-front about this: Flare is really good. There are concepts in Flare that have never been seen before. We expect to be able to solve problems in Flare that cannot realistically be solved in any other language. ... Back in the good old days, it may have made sense to write "efficient" programming languages. This, however, is a new age. The age of microwave ovens and instant coffee. The age of six-month-old companies, twenty-two-year-old CEOs and Moore's Law. The age of fiber optics. The age of speed. ... "Efficiency" is the property that determines how much hardware you need, and "scalability" is the property that determines whether you can throw more hardware resources at the problem. In extreme cases, lack of scalability may defeat some problems entirely; for example, any program built around 32-bit pointers may not be able to scale at all past 4GB of memory space. Such a lack of scalability forces programmer efforts to be spent on efficiency - on doing more and more with the mere 4GB of memory available. Had the hardware and software been scalable, however, more RAM could have been bought; this is not necessarily cheap but it is usually cheaper than buying another programmer. ... Scalability also determines how well a program or a language ages with time. Imposing a hard limit of 640K on memory or 4GB on disk drives may not seem absurd when the decision is made, but the inexorable progress of Moore's Law and its corollaries inevitably bumps up against such limits. ... Flare is a language built around the philosophy that it is acceptable to sacrifice efficiency in favor of scalability. What is important is not squeezing every last scrap of performance out of current hardware, but rather preserving the ability to throw hardware at the problem. As long as scalability is preserved, it is also acceptable for Flare to do complex, MIPsucking things in order to make things easier for the programmer. In the dawn days of computing, most computing tasks ran up against the limit of available hardware, and so it was necessary to spend a lot of time on optimizing efficiency just to make computing a bearable experience. Today, most simple programs will run pretty quickly (instantly, from the user's perspective), whether written in a fast language or a slow language. If a program is slow, the limiting factor is likely to be memory bandwidth, disk access, or Internet operations, rather than RAM usage or CPU load. ... Scalability often comes at a cost in efficiency. Writing a program that can be parallelized traditionally comes at a cost in memory barrier instructions and acquisition of synchronization locks. For small N, O(N) or O(N**2) solutions are sometimes faster than the scalable O(C) or O(N) solutions. A two-way linked list allows for constant-time insertion or deletion, but at a cost in RAM, and at the cost of making the list more awkward (O(N) instead of O(C) or O(log N)) for other operations such as indexed lookup. Tracking Flare's two-way references through a two-way linked list maintained on the target burns RAM to maintain the scalability of adding or deleting a reference. Where only ten references exist, an ordinary vector type would be less complicated and just as fast, or faster. Using a two-way linked list adds complication and takes some additional computing power in the smallest case, and buys back the theoretical capability to scale to thousands or millions of references pointing at a single target... though perhaps for such an extreme case, further complication might be necessary.

    As with the other Moore's law predictions of the era, this is not only wrong in retrospect, it was so obviously wrong that undergraduates were taught why this was wrong.

    [return]

  5. My personal experience is that, as large corporations have gotten more powerful, the customer experience has often gotten significantly worse as I'm further removed from a human who feels empowered to do anything to help me when I run into a real issue. And the only reason my experience can be described as merely significantly worse and not much worse is that I have enough Twitter followers that when I run into a bug that makes a major corporation's product stop working for me entirely (which happened twice in the past year), I can post about it on Twitter and it's likely someone will escalate the issue enough that it will get fixed.

    In 2005, when I interacted with corporations, it was likely that I was either directly interacting with someone who could handle whatever issue I had or that I only needed a single level of escalation to get there. And, in the event that the issue wasn't solvable (which never happened to me, but could happen), the market was fragmented enough that I could just go use another company's product or service. More recently, in the two cases where I had to go resort to getting support via Twitter, one of the products essentially has no peers, so my ability to use any product or service of that kind would have ended if I wasn't able to find a friend of a friend to help me or if I couldn't craft some kind of viral video / blog post / tweet / etc. In the other case, there are two companies in the space, but one is much larger and offers effective service over a wider area, so I would've lost the ability to use an entire class of product or service in many areas with no recourse other than "going viral". There isn't a simple way to quantify whether or not this effect is "larger than" the improvements which have occurred and if, on balance, consumer experiences have improved or regressed, but there are enough complaints about how widespread this kind of thing is that degraded experiences should at least have some weight in the discussion, and Kurzweil assigns them zero weight.

    [return]

  6. If it turns out that longtermists and other current predictors of the future very heavily rely on the same techniques as futurists past, I may not write up the analysis since it will be quite long and I don't think it's very interesting to write up a very long list of obvious blunders. Per the comment above about how this post would've been more interesting if it focused on business leaders, it's a lot more interesting to write up an analysis if there are some people using reasonable methodologies that can be compared and contrasted.

    Conversely, if people predicting the future don't rely on the techniques discussed here at all, then an analysis informed by futurist methods would be a fairly straightforward negative result that could be a short Twitter thread or a very short post. As Catherine Olsson points out, longtermists draw from a variety of intellectual traditions (and I'm not close enough to longtermist culture to personally have an opinion of the relative weights of these traditions):

    Modern 'longtermism' draws on a handful of intellectual traditions, including historical 'futurist' thinking, as well as other influences ranging from academic philosophy of population ethics to Berkeley rationalist culture.

    To the extent that 'longtermists' today are using similar prediction methods to historical 'futurists' in particular, [this post] bodes poorly for longtermists' ability to anticipate technological developments in the coming decades

    If there's a serious "part 2" to this post, we'll look at this idea and others but, for the reasons mentioned above, there may not be much of a "part 2" to this post.

    [return]
  7. This post by nostalgebraist gives another example of this, where metaculus uses Brier scores for scoring, just like Tetlock did for his Superforecasting work. This gives it an air of credibility until you look at what's actually being computed, which is not something that's meaningful to take a Brier score over, meaning the result of using this rigorous, Superforecasting-approved, technique is nonsense; exactly the kind of thing McElreath warns about. [return]