VITAL STATISTICS

Dick Pountain/18 June 2006/16:09/Idealog 143

Who'd have thought that statistics could ever become sexy? I don't of course mean Benny-Hillish sniggering about '36-24-36', though no doubt cup-size does follow a Gaussian distribution. No, I'm talking about the surprise success of books like Steven D. Levitt and Stephen J. Dubner's 'Freakonomics', or my subject this month, Philip Ball's 'Critical Mass' (Arrow Books, 2004). I rarely encounter books that I wish I'd written but this is one, though I'm aware that I lack the patience for the monumental research that underlies it. Ball doesn't set out to make statistics sexy in quite the sizzling populist way the Freakonomists do, but with a rather deeper purpose of showing us what statistical methods can tell us about the way both natural and social worlds work - what he calls a 'physics of society'. If you immediately recoil, thinking him an insensitive clod who denies free will and reduces humans to machines, then think again; on the contrary he's hyper-sensitive to such objections and begins the book with a discussion of Hobbes' 'Leviathan' in the light of exactly such concerns. Nor is this a book about the interpretation of bureaucratic government statistics - tax revenues, employment rates, NHS expenditures and the like. It's more abstract, and far, far more interesting than that.

Ball's ultimate subject is statistical mechanics, that branch of physics that deals with the average behaviour of huge numbers of particles, which developed from the classical studies of gasses by Clausius, Rumford, Clerk Maxwell and Boltzmann in the late 19th century. Regarded at first  with suspicion because of its approximating nature, statistical mechanics soon uncovered regularities every bit as certain as the algebraicised predictions of Newtonian mechanics, and nowadays of course it underpins solid-state physics and the behaviour of semiconductors, from which the computer industry grew. Very early on people began to realize that some of the truths uncovered by statistical mechanics might be applicable to human societies, which are also are composed of large numbers of independently-moving particles, albeit rather more moody and variable ones. A 'science of society' has been dreamt of from Hobbes, through Comte and his Positivism, all the way up to Karl Marx, but Ball shows how they were all doomed to failure because they understood science to mean Newtonian science - equations you can solve to give you definite answers. A real 'physics of society' doesn't look for analytical solutions in terms of equations, but for principles that describe the behaviours of large populations independently of the kind of particles involved. It turns out there are quite a few such principles, and this book is all about them.

I've already alluded to one of them above - the Gaussian distribution (the familiar bell-shaped 'normal curve') which describes any property of a population that deviates randomly around some mean value, like the weight of people, the height of hollyhocks, the diameter of raindrops. For a long time statisticians reached for a Gaussian distribution whenever they saw a random-looking data set, but not any longer. Since the work of Mandelbrot (yes, he of a million fractal posters) and others it's been realised that sometimes the Gaussian distribution can lead us very far astray - it assumes a particularly even sort of randomness that may seriously underestimate the likelihood of extreme events. That's what brought down the Nobel-prize winning Black-Scholes equation (used to price futures contracts and similar derivatives) in 1997 and almost brought down the world financial system with it. It may be why several supertankers disappear without trace on the ocean every year, not abducted by aliens but swamped by giant waves that a Gaussian distribution wouldn't predict. Many phenomena actually follow a Power Law distribution, which looks rather Gaussian at first glance but has fatter 'tails' that conceal such monsters, and this insight can be applied to everything from gasses at their critical point to traffic jams on the M25.

The physics of society says that Karl Marx was right about one thing - namely that free markets are inherently unstable - even if his prescriptions for what to do about it were wrong. Stock prices fluctuate with a distribution that lies somewhere between Gaussian and Power Law, with big events like market crashes infrequent, but unpredictable and inevitable. The rosy view propagated by US neo-liberal economists that 'The Market' is a quasi-magical mechanism that incorporates all wisdom and leads to equilibrium is pure hooey - the big dips aren't externally caused by government meddling, but arise naturally because markets are always very far from equilibrium. 

Most of Ball's subjects are computer-related, because these systems can only be studied via computer simulation - you just can't solve the equations even for something as simple as magnetisation of a metal (which can be solved in 1 and 2 but not in 3 dimensions). You'll encounter systems for simulating footpaths across parks, jams on the motorway, the life cycles of companies (including hard-workers versus slackers) and the stability of alliances - the latter almost exactly predicts the outcome of the 1990s war over Unix standards as a test. Ball describes all of these in clear and highly readable prose, with a gift for simple explanations of complex ideas and an intelligent and humane appreciation of the problems such approaches can raise, which makes my authorship-envy even more acute. If you only read one book this year, it ought to be this one.