In the manufacturing World, demand forecasting is one of the most important foundations for accurate, timely, and effective production. It is no surprise then that in a recent survey more than 50% of manufacturing executives emphasized the need to improve their demand forecasting capability in order to achieve their targets. So, the importance of demand forecasting is not going away. What is changing is how people perform demand forecasting today.
“It’s all about that data”!
While In the past, demand forecasting has always been veiled in mystery or magic and relied heavily on the gut feelings and experience of demand planners, today it is mostly based on big data and machine learning time series modeling techniques.
So, what actually is Demand Forecasting? It is simply estimating the number of products or services that customers will purchase in the near term or long-term future. However, at the higher level, demand forecasting is a field of predictive analytics that tries to understand and predict customer demand to optimize supply decisions by corporate supply chain and business management. The goal is to use demand forecasts as a baseline to compare to actual orders and to align with the market directions and how to align inventories, resources/capacity, and suppliers in general. Demand Forecasting is driving all these activities and becoming a key pillar for the success of the company. In order to maintain a competitive position in the market, manufacturers try to quickly adapt to a fast-changing world as well as continuously deliver innovative products and services to customers.
So, why demand forecasting using Big Data and data-driven time-series approaches in today’s World is so challenging? There are several main reasons which I try to summarize here:
- Data is believed to be the new oil but everything about data can be quite challenging starting from data availability and inefficient data ingestion processes to data quality and complexity. In addition, all the data are coming at different speeds and from different sources, which makes it extremely difficult to aggregate them at the right level and frequency.
- Market dynamics are constantly changing, new competitors are emerging all the time, and anything that Amazon does disrupts a whole industry! This situation has even further worsened with the emergence of the COVID-19 virus. All these changes require predictive models to be constantly refreshed.
- Legacy systems typically span many different teams which require a level of coordination that is difficult. In addition, very often, there is a common belief that companies should stick with ad-hoc approaches that are working relatively well and apply the principle “we always did It this way”.
- There are often hundreds or thousands of demand forecasting units for which forecast needs to be provided and scaling up these efforts could require steady increases in computational power and infrastructure.
- Traditional time series approaches are too simplistic and may not be able to address various challenges like intermittent demand or lack of historical data especially for new or less popular products. Different time series algorithms make different assumptions about data and it is time-consuming to evaluate the right methodology.
Let’s see what data scientists could do to address some of these challenges. The figure below illustrates the overall data science framework for demand forecasting.
- It is critical to understand what type of time series data you have. You may have well-established products where you have stationary demand or well established seasonal demand (e.g., lawnmowers during the spring and summer or snow blowers during the winter) or well-established trend based on company growth, market trends or product popularity (wi-fi enabled products, battery-operated products). When you have more than 100,000 products, you have fluctuations in your data all the time.
- You need to generate some additional variables/features that could better capture the demand signal. You may use various transformation techniques for this task
- You have to check various correlation and causality facts. It is important to understand not only what features are correlated but also how some features impact others.
- You would need to understand the connection between your products’ promotions and their demand. Promoting some products may not cause only increasing their demand but also increasing demand for some of the supplemental products through halo effect. On the other hand, some of the products may be negatively affected by promoting products with similar functionality.
- When you deal with products with intermittent demand and lack of historical data, it is a good practice to identify their similar products and replace the missing data with historical data from similar products. This could help your demand forecasts for such products.
- Finally, you can build multiple time series models through various open-source packages and/or a handful of auto ML platforms and combine them through ensemble methods.
In the end, it is important to understand that all modes are biased and it is up to data scientists to constantly check their data, find relevant data that could bring additional signals, AND frequently validate their models.
