The Signal and the Noise

Title: The Signal and the Noise – The Art and Science of Prediction

Author: Nate Silver

Publisher: Allen Lane, 2012 (First)

ISBN: 978-1-846-14752-4

Pages: 454

Uncertainty is an inseparable feature of natural and social lives of man. We come across unpredictability at every corner, and encounter experts predicting the outcomes of various events based on painstaking research – at least that is what they say. Normally, this incertitude is so much a part of our way of life that we hardly pose to realize that there may be other ways, less uncertain, about them. This book is an excellent beginning to inspect those events in a rational way and to reach impressive conclusions. Even though I have used terms like uncertainty and unpredictability in a synonymous way, there are subtle differences between them which the author is at great pains to explain in the course of the narrative. And Nate Silver is just the right man for doing that, being a statistician and political forecaster at The New York Times. In 2012, he correctly predicted the outcome of all the states in the US presidential election. He has also been named one of Time’s 100 Most Influential People in the world. Being a forecaster himself, he explains the pitfalls many of them fall into, when analyzing complex fields such as electoral outcomes, stock markets, spread of contagious diseases, sports betting, weather, climate change and even some of the nuances in Chess tournaments. Every prediction is wrought with uncertainty, but the quantum of this factor is not always mentioned in some of the startling announcements. When skill is also a factor to account for, experts find it easy to outsmart the novices who are ignorant about the probabilities which determine the outcome to a great extent. Hence the importance of the book – it helps to assess the predictability of an event, the margin of error inherent in a prediction and how best to effectively use such advice in reaching conclusions that have impacts on the financial, political or climate fronts.

Silver starts his masterly discourse with a brief but inimitable introduction into the necessity of separating the information in the signal from the background noise. If only all authors used such lucid analysis to explain their concepts! The author asserts that mankind began facing the challenge of richness of data that originated with the invention of printing press, at which time the information revolution really began. The number of books skyrocketed in the years succeeding that momentous event and cost of books and printed information plummeted, making them affordable to a large class of common people. Along with this surge of information came noise, the signal which doesn’t carry any information at all. Man is evolutionarily well equipped to discern patterns in a forest of random shapes and the problem reared its ugly head when this supersensitive faculty was turned against the flood of data that suddenly became available. This ended up in a large number of predictions not matching up with the outcome. Silver describes about the art and science of prediction, the tools with which people go about predicting the results and the pitfalls that await them on the road

Predictions that mainly come our way in our normal course of life are about political events like the result of an election. The author submits the flurry of TV predictions to an exhaustive analysis to come out with the stunning observation that all of them don’t stand a chance better than flicking a coin. But the efforts to predict the future career of baseball players are not that random. Here, software as well inquisitive researchers have made proven track record in identifying talent from early stages. The author himself is immensely attracted to this field, who has made software for predicting this, and the readers gets the impression that Silver is not totally unbiased when he argues that the computer’s efforts in baseball is entirely worthwhile. Another common task is predicting the weather. Here, the meteorologist is solidly assisted with two things – persistence, which maintains that the weather tomorrow would be very similar to what it is today and climatology, which states the statistical probability of a day’s weather collected from data collated over many previous years. In order to classify a weather prediction as accurate, the person must exceed the utility provided by the two. However, the commercial analysis of weather is not unbiased. A wet bias is argued to exist, in which the predictor assigns a chance to rain when in fact the data claims the chance to be very small. This is because people tend to ignore non-occurrence of rain when it was predicted than the other case of rain occurring when it was predicted not to, which may ruin their picnic.

Climate change in the form of man-made global warming as the result of increased carbon dioxide emissions from industrial processes is a phenomenon seems to be occurring on a planetary scale. The UN-spawned IPCC (Inter-governmental Panel on Climate Change) monitors the temperatures regularly and comes down with predictions about long-term averages. The predictions of IPCC do not take into account the full measures of the complexity of the situation, as the author asserts. There is a full chapter on global warming in the book but uncharacteristically it does not delve deep into the details and don’t say conclusively whether the UN-body’s prediction would be right or wrong. Silver is contented with presenting a balanced picture, the arguments for and against the theory. There was indeed a rising trend from 1970 to 2000, but the first decade of the present century was relatively cool. But the author quickly picks up his Bayesian calculator and claims that the probability of the theory to be still true is a solid 85% even after accounting for the cool decade. A new argument is also presented to be behind the decline. This has to do with sulphur dioxide. The molecules of this gas spreads as aerosol in the upper layers of the atmosphere and reflect sunlight back to the space, thereby lessening the greenhouse effect. But the substance is highly polluting, being the source of acid rain. Sulphur emissions were cut down drastically as a sequel to the enactment of Clean Air Act in the mid-70s. The reduction might have contributed to the disappearance of the cooling effect of sulphur dioxide in the period leading up to 2000. Then how did the mercury go down in the next 10 years? According to Silver the impetus to industrial production in China, which doesn’t enforce any environmental regulations would have pumped more Sulphur into the atmosphere, ensuring a cooler decade. He ends with a premise that IPCC’s predictions of temperatures, revised in 1995, may well be true.

Silver’s examples and fields of application for his original thought and insightful ideas are very apt and fitting for the issue at hand. Unfortunately this fine discretion is unfortunately not applied in a few examples on prediction related to sports. The vile contraption going by the name of baseball dominates American thinking, even though nowhere else would you find sensible people pitching for this strange game. The author devotes a full chapter to the nitty-gritty of baseball prediction, which is really a pain-in-the-neck for the non-American readers who are not at all familiar with how the game is played. A similar argument holds for Poker, which is also one of the author’s favourite pastimes that have come to haunt the reader. This must surely be counted as a disadvantage to the book. At the same time, however, the author more than makes up for the shortcoming through several other chapters excellently structured with relevant concepts. We need not look further than the section in which he introduces Bayesian theorem which evaluates the probability of an event occurring due to a phenomenon which has a definite prior probability of occurring. Silver explains the concepts with an extremely hilarious instance of calculating the chances that your partner is cheating on you, if you happen to find a piece of underwear in the wardrobe which does not belong to you. If the prior probability of a cheating partner is 4% (collected from social data), Silver asserts humorously that, even after finding the suspicious object mentioned above, the probability that the person is cheating only rises to 29%. The reasoning is crystal clear, but the probability of a person being consoled by such figures is highly unlikely.

This book is highly recommended and is a must read. I would have given it a 4-star rating, if the author was not so particular about the lengthy chapters on baseball and poker.

Rating: 3 Star