Reading Nate Silver‘s Signal and the Noise: Why Most Predictions Fail but Some Don’t this last week, I’ve been going over a summary of how far we’ve come in our ability to forecast our future. Elections, earthquakes, hurricanes, economic recessions, sports scores etc. can all be forecast today, with varying degrees of accuracy. The book talks about how and why these forecasting models are still not as accurate as we’d like them to be, and details the shortcomings that need to be fixed. It talks for example, about how we’re able to almost accurately predict where hurricanes would land to (only) a hundred mile radius with (only) a seventy-two hour window. As a non-statistician however, this is still a huge improvement over what we could manage a few decades ago. All of this predictability adds so much certainty to our lives. Compared to our ancestors, for whom every moment carried a multitude of different threats from being hit by lightning to becoming dinner for a hungry tiger, our lives are far more certain with far fewer threats.
Unlike the book, the theme of this post isn’t why predictions fail. I instead wanted to think about the consequences of this added predictability on our happiness and on life in general. When weather forecasts go wrong we blame the weathermen. We blame economists for being unable to foresee and forewarn us of economic recessions. Inability to predict threats is almost criminal for experts today. There is an expectation of certainty and anything that results in that expectation not being met, results in a post-mortem and a corresponding punishment. Guess who is the expert of our own individual lives? Guess who do we blame when we’re unable to predict our own miserable future?
All of this certainty hasn’t come without consequences. As a general pattern, we all create short and long term futures for ourselves that we believe are most achievable based on our abilities and circumstances and would maximize our overall short and long term happiness scores. We assign a high-certainty score to these futures and over time if we find that score reducing, we blame ourselves for a bad forecast. With so much certainty in external variables, we have very little outside of our control that can go wrong – forcing us to agree that it had to be our own flawed model that resulted in the sorry predicament. In India, I could blame the power failure (a fairly common unpredictable threat) on the eve of the exam for my low score; in US I no longer had that luxury. My score is a function: more of my abilities and less of the power-supply or the public transport systems, than it was in India. Just like certainty, the pain & frustration of failure is also exacerbated several times as a result of this purer model. Beyond a certain point, we completely lose track of our original intent and all that remains is our tireless effort to keep that certainty score in chasing that now miserable future, as high as possible.
The other consequence of certainty is the inertia it creates. In developed societies, where stimuli for the mind, body and intellect are easily accessible, it is easy to find people become numb to desire and experience (mostly from an overdose of the same). Shuffling between Ambien and caffeine, they cling on to the last thing separating them from insanity – The certainty in their current state. At the cusp of our physical, intellectual and emotional being, our craving for certainty creates an inertia that prevents, or at least resists any change in our zombied state of existence. This state is what Gita refers to as Tamas.
Even if you’re not as damaged as the folks mentioned above, the certainty and clear visibility into future that your current state brings, does create a comfort zone around you. This comfort zone and the lack of certainty in the alternatives, become the biggest barrier to change and to a happier existence. We often hide our desire to retain this comfort zone under the guise of ‘risk-aversion’. Once we’ve made the choice of certainty over uncertainty, our Confirmation bias kicks in to strengthen our belief in this choice, but our misery continues. Over time our attachment to the current state and the fear of losing it creates more stress, frustration and pain.
The solutions to these problems of personal forecasting lie in the same principles of forecasting that Nate highlights in the book. He uses them for forecasting scores and election results. I’ve altered the same to suit personal-life forecasting as discussed in this post:
- Be probabilistic in your forecasting models: Instead of creating a single future-story for yourself create multiple futures each with it’s own probabilistic score.
- The forecasting model should change with time: As the circumstances, personal-interests and abilities change, be open to changing the probabilistic scores you assigned initially and to adding more futures to the model.
- Look for consensus: It is important to seek external validation of your forecasting model and to note how others feel about the probability scores you’ve assigned to the different futures in your model, given how others perceive your abilities and circumstances. Of course, you need to be smart in your choice of these ‘others’.
In essence, this analysis above seems to suggest embracing the chaos in life (albeit in a better & smarter way). We should welcome it instead of being afraid of it. We should be able to find happiness in change, as was suggested in my previous post on choice. In fact, choice and certainty seem quite close to each other. Where choice creates an illusion of power over desires and accentuates our ego and inherent biases, certainty induces attachment towards this illusion and makes us resistant to change that might actually improve our current state. Like a coy, cute, conniving, coquettish temptress she’ll flirt with you and tempt you into getting what she needs from you, and when ultimately, thanks to her, you’ll end up in a state of utter misery, she’ll even seduce you into absolving her and instead blaming yourself for your predicament.
उन के देखे से जो आ जाती है रौनक़ मुंह पर
वो समझते हैं कि बीमार का हाल अचछा है (ref.)
-the mere sight of her is enough to light up my face
=and she takes it to imply that my sickness is healed and health is good
देखिये पाते हैं `उशशाक़ बुतों से कया फ़ैज़
इक बरहमन ने कहा है कि ये साल अचछा है (ref.)
-lets see what grace/favor/benefit lovers find from idols
=an idol worshiping oracle has foretold this year is good
from the reference– “…to a lover, the very idea of a year’s being ‘good’ can have only one possible meaning that’s of any interest at all. The lover’s wild myopia, his form of tunnel vision, is funny in itself.”
हम को म`लूम है जननत की हक़ीक़त लेकिन
दिल के खुश रखने को ग़ालिब ये ख़याल अचछा है (ref.)
-and even though i know the reality/truth of heaven/paradise
=yet, to keep the heart happy, Ghalib, this thought is good