How the solution to the Monty Hall Problem is also the key to predictive analytics

If contestants on the game show Let’s Make a Deal knew Bayes’ Theorem, they’d be much more likely to win the big ticket prize than head home with the undesirable items the show referred to as “Zonks.”

The show used what came to be known as the Monty Hall Problem, a probability puzzle named after the original host. It works like this: You choose between three doors. Behind one is a car and the other two are Zonks. You pick a door – say, door number one – and the host, who knows where the prize is, opens another door – say, door number three – which has a goat. He then asks if you want to switch doors. Most contestants assume that since they have two equivalent options, they have a 50/50 shot of winning, and it doesn’t matter whether or not they switch doors. Makes sense, right?

A data scientist, on the other hand, would use Bayes’ Theorem to calculate the probability of winning the car, instead of relying on their intuition. And they’d have a much higher chance of winning. The correct answer, it turns out, is to switch the door.

Thomas Bayes, born in 1703, was an English mathematician, statistician and Presbyterian minister. The work that made him famous – Bayes Theorem – wasn’t published until two years after his death. This has become the preferred method for calculating the probability of an event that is dependent on other events preceding it. A multitude of modern techniques used by data-scientists, e.g., naive Bayes, Bayesian estimation, Bayesian Networks, etc., all derive from the original Bayes Theorem.

One theorem, many impressive applications

Whether you realize or not, Bayes Theorem is used in many of the consumer technologies we use daily. It allows us to make predictions on everything from the weather, to fantasy football and presidential elections. It powers Amazon and LinkedIn’s recommendations and Google’s search products. It even keeps spam out of your inbox.

The Monty Hall Problem is also at the core of business. Just as a Let’s Make a Deal contestant can better their chance of winning by using probability theory, so can businesses. As enterprises move away from gut-based decision making to relying on data science for applications in risk management, marketing, sales, hiring, retaining customers and pricing correctly, it’s no surprise that Bayes Theorem has started to make a broader impact on the economy. Where people often guess the probability of events incorrectly, Bayes Theorem does not.

The Theorem considers “Priors” (also called “signals”) – events that we have pre-existing knowledge on – and “Posteriors” – events that we want to predict. As more data on Priors has become available, one can use cloud computing to calculate the probability of even very complex events using Bayes Theorem. Modern predictive analytics is powered by two major trends – big data and cloud computing – and Bayes Theorem is the link that connects the two.

Why Bayes’ is even more important today

When businesses use Bayes’ Theorem to solve a complex challenge, the accuracy of the prediction is limited by the quantity and accuracy of Signals. While machine learning allows models, which often have a learning curve, to improve over time, the key to better predictions is higher quality data on Signals. This is where cloud-based providers of data and insight will have the largest impact since they can harvest Signals at a scale that’s hard to replicate at the individual enterprise level.

They do what Nate Silver calls the “Bayesian convergence” – dispel myths and opposing opinions as evidence of the most likely outcome is uncovered. This is Bayes’ greatest gift to enterprise technology.

Bayes Theorem isn’t shiny or new. In fact, it is taught in most high schools across the world and is actually a very logical way of thinking about business decisions. It’s been around for a long time – 251 years to be exact. But the value that Thomas Bayes has brought to the enterprise has become considerably more apparent alongside the progression of Big Data. Every business is capable of using these methodologies for decision making. Don’t let yours operate like a Let’s Make a Deal contestant.
Shashi Upadhyay is co-founder and CEO at Lattice, a company that offers predictive applications that help companies market and sell more intelligently. Follow him at @shashiSFGigaom’s Structure:Data event, held March 19-20 in New York City will examine other tools companies are using to analyze data.
(This post has been edited to reflect that probability theory can improve the odds of solving the Monty Hall Problem.)