How do you design a forecasting at scale ML system?

In this article

1. Before you start
2. Why interviewers ask this
3. Forecasting at scale from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page builds on the machine learning operations (MLOps) interview questions and on the basics in machine learning algorithms and how to choose one.

Why interviewers ask this

Forecasting questions can appear in machine learning system design rounds for retail, supply chain and capacity planning, because the naive design breaks at scale in a way that is easy to probe. If you say "one model per product", the follow-up is what happens with a very large number of products, a new product with no history, or a total that must equal the sum of its stores.

A weak answer names one algorithm and a train-test split. A strong answer says why many related series are modelled together, why the output is a range rather than one number, and how the evaluation avoids leaking the future. A Staff-level answer adds the failure that is easy to miss: forecasts made separately for each level of a hierarchy disagree, so the system must reconcile them and measure accuracy at the aggregate levels, not only at the bottom.

Forecasting at scale from first principles

Global forecasting model

A global forecasting model is a model that is fitted once on many related time series together and forecasts each of them from features such as the series' recent values, the calendar and the price, instead of fitting a separate model to every series.

Coherent forecast

A coherent forecast is a set of forecasts for a hierarchy of series in which every aggregate equals the sum of its parts, so the forecast for a company total equals the sum of the forecasts for each of its stores, and no forecast contradicts another.

Forecast reconciliation

Forecast reconciliation is a post-processing step that takes forecasts made separately for every level of a hierarchy, which usually disagree, and adjusts them, using the structure of the hierarchy and an estimate of the forecast errors, so that they become coherent.

In plain words

Picture a grocery chain with thousands of products. One way to plan is to hire a planner for each product. Each planner only studies one sales history. That is a lot of planners, and a new product has nobody who knows it.

The other way is one planner team that has studied all the products. They learn shared patterns, such as weekends and holidays. They forecast each product from those patterns plus its own history. The analogy breaks in one place. Real products differ, so the shared model needs features that say which product it is looking at.

In one precise sentence, a forecasting at scale ML system fits global models on many related series, forecasts a distribution for each, makes the forecasts coherent across the hierarchy, and scores them with rolling-origin evaluation.

The business reason is stated plainly by Salinas et al. (2019): "forecasting demand is crucial for having the right inventory available at the right time at the right place."

Step 1: Train a few global models on many related series

A forecasting at scale ML system fits a few global models, not one model per series. In DeepAR, Salinas et al. (2019) train one autoregressive recurrent network on a large number of related series. The M5 winner trained LightGBM models at store, store-category and store-department level and combined them (Makridakis et al., 2022).

Input: the history of every series plus calendar, price and promotion features, which a feature store can serve the same way in training and in forecasting. Output: a forecast for each series. A new product with a short history borrows patterns from similar ones.

A long horizon needs a multi-step strategy. A recursive strategy fits a model for the next step and feeds each prediction back in as a lag, so errors compound. A direct strategy fits one model per horizon step, or adds the horizon as a feature, which costs more models. One M5 entry, the fourth-placed, used non-recursive LightGBM models, one per week of the horizon (Makridakis et al., 2022). Either way, a lag feature for a later step must use only values known at forecast time: yesterday's sales are not known for a forecast ten days ahead unless the model predicts them, and using the real value leaks the future.

The scale difference is large. One local model per M5 bottom series would be 30,490 fits per retrain, against 40 global models in an illustrative design, about 762 times fewer.¹ You now have a forecast for every series, and the next step makes it useful for decisions.

Step 2: Forecast a range, not a single number

A forecasting system should output quantiles, not only one number, because stock and capacity decisions depend on how wrong a forecast could be. Salinas et al. (2019) call estimating the probability distribution of a series' future "a key enabler for optimizing business processes."

A model trained for one quantile uses pinball loss. Pinball loss is a score that charges an under-forecast and an over-forecast differently, weighting the miss by the quantile level on one side and by one minus that level on the other. For the 0.9 quantile with a forecast of 100 (illustrative), an actual of 120 costs 18 and an actual of 80 costs 2, so the model learns to forecast high on purpose.¹

One trap follows. Quantiles of separate series do not add up to the quantile of their total, so a distribution needs more than the point-forecast fix in the next step. Probabilistic reconciliation methods exist for this, but the worked example reconciles point forecasts only.

Step 3: Reconcile forecasts so the parts add up

Reconciliation makes every aggregate forecast equal the sum of its parts, the condition Wickramasuriya et al. (2019) call coherence. Independent forecasts for each level usually break it, and a global model does not fix that. An M5 robustness check found that global LightGBM models did very well at the bottom levels but lacked robustness once summed to aggregates.

The simplest fix is bottom-up. The bottom-up approach is a reconciliation method that forecasts only the bottom level and sums upward (Hyndman and Athanasopoulos). It ignores the relationships between series. Minimum trace (MinT) reconciliation instead minimises the sum of variances of the reconciled errors while staying unbiased (Wickramasuriya et al., 2019). Reconciliation makes the parts add up to the whole.

Step 4: Backtest with a rolling origin at every level

A forecasting system must be scored on forecasts of data it has not seen, and a rolling-origin backtest does that. In the Hyndman and Athanasopoulos chapter on time series cross-validation, residuals are "not a reliable indication of how large true forecast errors are likely to be", because "fitted values are often not true forecasts".

Errors in raw units cannot compare series with different units or sizes, so many-series evaluation uses scaled errors; the M5 competition scored with a variant of the mean absolute scaled error (MASE), the root mean squared scaled error (RMSSE) (Makridakis et al., 2022). The M5 winner used the last four 28-day windows for validation (Makridakis et al., 2022).

A random k-fold split breaks time order and lets the model see the future. Finally, score the aggregate levels as well as the bottom, because Step 3 showed that strong bottom-level scores can hide weak totals. A rolling backtest scores the forecasts at every level.

1 · Train2 · Forecast3 · Reconcile4 · BacktestReplay

Step-by-step description

1. One global model learns from many related series.
2. The model outputs quantiles for every series.
3. Forecasts are adjusted so the parts add up.
4. A rolling origin scores every level.

• A global model trains on many related series.
• Quantile forecasts give a range, not a point.
• Reconciliation makes the parts add up to the whole.
• A rolling backtest scores the forecasts at every level.

Worked example

Take a toy hierarchy with the same size as the example in the fpp3 hierarchy chapter: one total, two groups and five bottom series, so n = 1+2+5 = 8 series with m = 5 bottom series. Let group A hold two bottom series and group B three (illustrative). The independent forecasts are illustrative: total 110, group A 35, group B 75, and bottom series 10, 20, 30, 15 and 25.¹

Bottom-up sums the five bottom forecasts to a total of 100, group A of 30 and group B of 70, so the independent total of 110 was 10 too high against its own parts.¹ Least-squares reconciliation, the simplest member of the family and equal to MinT when every error has the same variance and none correlate, spreads the disagreement across every series and moves the total to 108.79.¹ Ordinary least squares (OLS) is the name of that projection.

The five bottom forecasts rise by between 1.55 and 2.07, and the reconciled values add up: the five bottom values sum to 108.79, the reconciled total.¹

Text description of the diagram

• Bottom-up sums the five bottom forecasts to a total of 100.
• Least squares reconciliation moves the total to 108.79.

Real hierarchies are far larger. The M5 data has 42,840 series in 12 aggregation levels, of which 30,490 are the lowest level (Makridakis et al., 2022). A MinT error covariance over all 42,840 series has 1,835,265,600 entries, and Hyndman and Athanasopoulos note that estimating it is unreliable when the number of bottom series is large relative to the series length, so they use a shrinkage estimator.¹

For the backtest, take a series of 1,000 days (illustrative) and four 28-day windows: the fits end at day 888, 916, 944 and 972, and each is scored on the 28 days after it.¹

Trade-offs and edge cases

Global models cut the number of models to maintain, but they do not make forecasts coherent. A local model is fitted separately to each series. A global model shares one fit across many series. Only reconciliation makes the parts sum to the total, by construction, and it brings its own cost: the covariance estimate.

Text description of the diagram

• A local model is fitted separately to each series.
• A global model shares one fit across many series.
• Only reconciliation makes the parts sum to the total, by construction.

Edge cases to name in a round:

• Bottom-up is not always safest. It is noisy at the bottom and ignores relationships between series, so it can do poorly at aggregates.
• Deep learning does not always win. In M5, LightGBM was reported to have "superior forecasting performance over all other alternatives" (Makridakis et al., 2022).
• The hierarchy changes. A new product or a closed store changes the summing structure, so the reconciliation inputs must be rebuilt with it.
• Sparse demand. Series that are mostly zeros need a distribution that allows zeros; a plain point forecast hides this. The M5 winner's models maximised "the negative log-likelihood of the Tweedie distribution", which suits data "with a probability mass of zero" (Makridakis et al., 2022).

The sources above do not describe a serving architecture, so treat the rest as design advice, not reported practice. Forecasts for planning are usually produced in scheduled batches, and batch versus streaming versus real-time inference explains how to choose. Retrain on a schedule, watch error by level, and compare the error against drift (trained in January, bad in July). The data platform under ML is what keeps the history and features consistent.

How to answer it in an interview

The spoken structure: say why global models, say you forecast a range, name reconciliation as the step that fixes totals, then describe the backtest and the batch job.

Common mistakes

• "I will train one model per product." (wrong) That is one fit per series at every retrain, and a new product has no history. (right) "I will train a few global models on related series and add series features."
• "The residual error is low, so the model is good." (wrong) Residuals come from a model fitted on all the data, so they flatter it. (right) "I score with rolling-origin cross-validation on held-out windows."
• "I will use random k-fold cross-validation." (wrong) It breaks time order and leaks the future. (right) "Every fold trains on the past and tests on the period after it."
• "The global model's forecasts already add up." (wrong) Independent forecasts usually do not, and strong bottom-level scores can hide weak totals. (right) "I reconcile the hierarchy and score every level."
• "A point forecast is enough." (wrong) Stock and capacity decisions depend on how wrong a forecast could be. (right) "I output quantiles and train with a quantile loss."

Follow-up questions

Why use one global model instead of a model per series?

A global model shares one fit across many related series, so there are far fewer models to maintain and a short series borrows patterns from similar ones. With 30,490 M5 bottom series the local design means 30,490 fits per retrain.¹

When would you choose bottom-up over MinT?

Choose bottom-up when the bottom forecasts are good and aggregates matter less, because it is simple and needs no covariance estimate. Choose MinT when aggregate accuracy matters and you can estimate the error covariance, with shrinkage if the number of bottom series is large relative to the series length.

How do you evaluate forecasts across series with different scales?

Use scaled errors such as MASE rather than raw-unit errors, and compute them in a rolling-origin backtest. Report the score at every level of the hierarchy, not only the bottom.

Recursive or direct multi-step forecasting?

A recursive model feeds its own predictions back as lags, so it needs one model but compounds errors. A direct approach fits a model per horizon step, or uses the horizon as a feature, so it needs more models but avoids that feedback. In both, a lag for a later step may only use values known at forecast time.

Does deep learning beat gradient boosting for forecasting at scale?

Not always. In M5 the best entries used LightGBM (Makridakis et al., 2022), while DeepAR shows a neural global model that outputs distributions. Pick by backtest on your own data.

How do you ship a new forecasting model safely?

Run the new model beside the current one, compare backtest scores at each level, and promote it only if the aggregate levels hold up; the design for continuous integration, delivery and training of ML apps covers the gates.

Sources

1. Forecasting: Principles and Practice (3rd ed.), chapter 11: Hierarchical and grouped time series — OTexts (Hyndman and Athanasopoulos), 1 Jan 2021
2. Forecasting: Principles and Practice (3rd ed.), hierarchical time series — OTexts (Hyndman and Athanasopoulos), 1 Jan 2021
3. Optimal forecast reconciliation for hierarchical and grouped time series through trace minimization — Journal of the American Statistical Association 114 (Wickramasuriya, Athanasopoulos, Hyndman), 1 Jan 2019
4. DeepAR: Probabilistic forecasting with autoregressive recurrent networks (arXiv 1704.04110) — arXiv (Salinas, Flunkert, Gasthaus, Januschowski), 1 Feb 2019
5. The M5 Accuracy competition: Results, findings and conclusions — International Journal of Forecasting (Makridakis, Spiliotis, Assimakopoulos), 1 Jan 2022
6. The performance of the global bottom-up approach in the M5 accuracy competition: A robustness check — International Journal of Forecasting, 1 Jan 2021
7. Forecasting: Principles and Practice, time series cross-validation — OTexts (Hyndman and Athanasopoulos), 1 Jan 2021