Weather forecasting, Big Data and Business Analytics

[date-stamp]We all talk about the weather. It is a conversation starter, and we experience weather every day and would like to know how the weather will be in the coming days. From that need comes the weather forecast, which more or less accurately predicts how the weather will be tomorrow and the next days. The weather on this planet is an extremely complex system, and creating a reliable forecast is demanding. Some say ths is a skill none have yet perfected, since the forecasts are not always correct. But they are mostly correct, and the creation of a weather forecast is a study in how to use big data in a rapidly changing environment in order to predict what will happen next. Sounds familiar? This is also what big data analytics is trying to do for businesses, predict how tomorrow will look like in terms of sales and market outlook. What can we learn from how the weather forecasts are made?

Gathering data

When thinking about gathering data for weather forecasts, I picture this is done using measuring stations scattered around the globe. These measuring stations contain a thermometer to measure temperature, an anemometer to measure wind gauge and speed, and a barometer to measure air pressure. But these stations can only measure the weather on the surface of our planet, so weather satellites do a similar job for observing and measuring what happens higher up in the atmosphere. Weather satellites have very good cameras which are used to monitor cloud formations, hurricanes and other large-scale weather systems.

There are two types of sensors on weather satellites: a visible light sensor called the "imager" and an infrared sensor called the "sounder". The "imager" is a camera which gathers information on cloud movements and patterns.  The camera can only be used in daylight, since it works by capturing reflected light to create images. Since water reflects little light, oceans appear dark on satellite images from weather satellites. Land masses reflect more light, and often appear as a shade of grey. The "sounder" sensor is an infrared sensor that measures temperature. The temperature of an ocean can be measured, for instance. The infrared sensor can be used both at day and night, since it does not depend on daylight to function.

Another source of data is Doppler radars. A Doppler radar measures reflected sound waves from objects, like ice crystals or dust particles. A radar antenna broadcasts sound waves, and whenever a sound wave comes into contact with an object the wave is reflected back to the radar antenna. The frequency of the reflected sound wave will indicate whether the object it hit moves towards the radar or away from it, which is used to track storms and where they are headed.

So there's a lot of data gathered about the weather, and at different times reflecting the changing weather.

The analogy to business data is striking, there's also a large and growing number of data sources and increasingly complex patterns to understand and do forecasting for in the business world. Most businesses gather more and more data about their customers. Take foodstores as an example, for every customer which is using their loyalty card the foodstore knows exactly what they are buying. If someone recently bought large amounts of beer and diapers, they can issue rebate coupons for these items to this customer in order to lure him back. This kind of data gathering is present in both the US and Europe. A friend of mine just got a bunch of coupons from a foodstore, and his reaction was 'they know me better than I do'. No need for a shopping list anymore, just bring the coupons!

Data 'sensors' in businesses are any interaction point with customers, where it is possible to learn about customer behaviour. For instance when they use their loyalty card, or when they buy items from an online store.

Raw computing power

In order to process all the gathered data massive amounts of computing power is required. Weather forecasting consists of computing millions of equations. In the beginning of advanced weather forecasting (in the 1920s), the English mathematician Lewis Fry Richardson estimated it would take 64.000 people doing computations by hand to get a good weather forecast in time for it to be useful. Clearly that was not practically possible, but with the computing power available now it is fully possible. Supercomputers which can do more than 1000 trillion calculations per second are used to calculate weather forecasts. Hundred of thousands of weather observations are used as input, and an atmospheric model is built which shows the current weather and the most  likely weather scenarios in the upcoming days and weeks.

The general increase in computing power has been an advantage for all sectors of society, and has enabled any business to perform data analytics. Maybe not on the same scale as weather forecasting, but most businesses also have less complex customers and buying patterns than the global weather system. So the ability to use raw computing power has saturated every business, and has thus armed them with a new weapon: forecasting of customer behaviour. Weather forecasting shows it is possible to calculate a pretty decen prediction of what will happen, imaging having a forecast of customer behaviour for the next two weeks?

The Result

Weather forecating produces pretty good predictions on how the weather will be, thanks to all the measuring points both on the surface of the planet and those provided by weather satellites. 100% accuracy has not been achieved, but with such a comoplex system with so many variablees as the weather that is hardly reaslistic either.There are always different scenarios with different probability values, and the weather scenario with the highest probability of happening does not always happen. But overall, weather forecasts are pretty good. And they are a prime example of how big data is being used to produce something we ll care about.

Forecasting based on business data is not as well-known as weather forecasting, but it is fairly advanced. Using multiple data points to predict future behaviour sounds like black magic, but vritually all big internastional companies use it to minimize customer churn and sell more to their existing customers. What can be learnt from weather forecasting is the streamlined process of gathering data from all possible data points and feeding them into an algorithm which generates multiple scenarios with varying probabilities. That gives companies the possibilities to determine where it is best to adapt in order to maximize sales and both long-term and short-term revenue.

Take Google as one example. There many things are data-driven, like the design on their web page. Users don't click on the links to Gmail, Drive and so on? Then change it. That was done some time ago, and now we have the 'nine squares' button up to the right with shotcuts to Google's other services. And their users now successuflly use this icon to access oher services from Google, more so that with the old 'menu-bar'-style buttons.  So being data-driven and building on available data is a competitive advantage for those businesses who's using it actively.


Sources: Annenberg Learner, UK Met Office