Demand Forecasting: Build vs Buy Costs

Final article

A hybrid SARIMA model that added three inputs (maximum temperature, minimum temperature and solar exposure) reached a 3.41% mean absolute percentage error when forecasting weekly peak power demand across three Australian states over a yearly time scale. A crude SARIMA built only on power demand variables came in 46.3% worse on MAPE, with the weather inputs as the single difference (arXiv 1911.00817). Same model class, three new columns, materially different economics for the utility running it.

That split between modeling work and data work is where forecasting budgets go sideways. Asking which library or which forecasting vendor to use is the cheap question, and most teams settle it in a week. The expensive ones are whether you can turn the promotions, stockouts and price changes sitting in your own systems into clean features, and whether anyone on the team can show a forecast beating a naive baseline on periods the model never saw. What follows prices those two questions.

Key Takeaways:

• Adding three weather variables to an unchanged model class produced a 46.3% improvement in MAPE over a crude SARIMA that used power demand variables alone.
• A deep Poisson mixture network improved overall CRPS by 11.8% on Australian domestic tourism data and by 8.1% on the Favorita grocery sales dataset, and showed no significant difference on randomly assigned Bay Area highway traffic hierarchies.
• A first production forecast costs person-months of data engineering before model work starts; plan for a three-person team across two quarters, most of it on extraction and feature construction.
• Multi-metric tuning on R2, RMSE and MAE beat a single-metric reference function across the Walmart, M3, M4 and M5 benchmarks regardless of optimizer.
• Zero-heavy series need a matching likelihood: Gaussian process models with Tweedie and negative binomial likelihoods produced better probabilistic forecasts than competitors across thousands of intermittent count series.

What Published Forecasting Benchmarks Actually Measure

The results worth reading before anyone signs a build order share a pattern: the lever that moved the error was usually features or evaluation design, not architecture.

The Portugal study compared multi-layer perceptrons, LSTMs, N-BEATS, temporal convolutional networks and temporal fusion transformers on one day-ahead task. N-BEATS took the top slot and the plain multi-layer perceptron came second, which the authors read as evidence for feed-forward networks over more sophisticated architectures; hour of day and temperature were named as key accuracy drivers (arXiv 2302.12168). A week of architecture benchmarking settles that argument. The quarter goes into features.

Hierarchical reconciliation is the one place where structure earns its keep, and only when the hierarchy is informative. Constraining forecasts to stay coherent across the tree improved overall Continuous Ranked Probability Score by 11.8% on Australian domestic tourism data and by 8.1% on the Favorita grocery sales dataset, while the same method showed no significant performance difference on Bay Area highway traffic, where the hierarchy was randomly assigned and the correlations were less informative (arXiv 2110.13179). If your product-by-region tree reflects genuine shared demand, fund reconciliation. If it reflects how the ERP was configured, skip it.

Evaluation design is the other lever. A hierarchical evaluation function combining R2, RMSE and MAE produced better model selection than a single-metric reference function across the Walmart, M3, M4 and M5 benchmarks, and the result held whether the search ran on grid search, particle swarm optimization or Optuna (arXiv 2508.13057). Single-metric tuning is cheap to run and expensive to trust.

Build vs Buy: Where the Budget Really Goes

Forecasting spend lands in four buckets, and only the third one looks like data science. The first is extraction: pulling order, shipment, price and promotion history out of whatever ERP and warehouse systems you own, then reconciling their definitions. The second is feature construction: calendars, holidays, paydays, competitor pricing and weather where it genuinely moves demand. The third is the model and its serving layer. The fourth is the part that never appears in the original business case: backtesting, retraining and monitoring for the life of the system.

Run a scenario. A mid-size retailer assigns two data engineers and one machine learning engineer for two quarters before the first forecast reaches a planner. In most organizations more than half of that time goes into buckets one and two. A managed service compresses buckets three and four but rarely touches the first two, because the vendor still needs your promotion calendar and still needs someone internal to explain why the forecast for a discontinued SKU looks wrong.

Buying carries trade-offs worth naming to a board. Per-series pricing scales with assortment, so a retailer that keeps widening its range grows the bill without a matching gain in value. Exogenous inputs are usually limited to whatever the vendor ingests, and that is the lever that matters most. The vendor’s model is also a black box against your reconciliation rules and your override workflow.

Building has the mirror-image problem. Your team owns drift, retraining cadence and on-call, and the second year consumes a large share of the first year’s effort with no new capability to show for it. That maintenance load is the line item missing from most build business cases.

The cleanest decision rule is about features rather than models. If demand is driven by variables you already store and can join reliably at daily granularity, build, because nobody outside your company has that join. If the drivers are mostly seasonality and price, and you cannot staff the monitoring bucket, buy and spend the saved engineering time on the plumbing that makes the vendor’s output usable.

Delivery Timelines by Team Size and Data Maturity

Delivery time depends on two variables: whether a usable history of clean daily data exists, and whether anyone on staff has shipped a forecast into an operational process before. The ranges below are planning guidance for a single business unit, not measured industry benchmarks.

• Two to three engineers, clean history, stable assortment: one quarter to a baseline statistical model plus gradient boosting, with rolling-origin backtests and a manual override path for planners. Add a second quarter for probabilistic output if inventory decisions need a service level rather than a point estimate.
• Same team, promotional retail: add a quarter for promotion and stockout features. This is where most overruns happen, because a promotion’s lift cannot be learned from two instances.
• Intermittent SKU-level demand: add a quarter for zero handling and likelihood choice. Series with long runs of zeros need a model built for that shape, which is what the Tweedie and negative binomial Gaussian process work addresses.
• One machine learning engineer, no dedicated data engineer: buy, or accept a much longer path with scope deliberately cut to a single category. Undersizing extraction is the most common cause of a stalled program.

Set the retraining cadence against how fast the assortment and pricing actually change, and review error by segment every month. A forecast retrained on last year’s catalog is measuring the past, not the business.

Monitoring, Drift and the Evaluation Harness

The evaluation harness is the deliverable most teams under-scope, and it decides whether the system survives year two. Rolling-origin validation, which repeatedly refits on a growing window and tests on the next unseen period, is the only backtest that resembles production. A supply-chain explainability framework published for the M5 Walmart dataset pairs SHAP and a time-aware variant called SHAPTime with rolling-origin validation, specifically to keep explanations stable across forecast horizons (summary of the lean XAI framework). Unstable explanations are a governance problem: planners stop trusting a system whose top drivers reshuffle at every refresh.

Monitoring needs three layers, and each costs engineering time rather than license fees. Track error by segment and horizon so a flattering aggregate number cannot hide a large miss on one category. Track input drift, because a supplier change or a broken promotion feed degrades accuracy long before it breaks anything visibly. Track decision outcomes, since a forecast only earns money through the replenishment, staffing or pricing action downstream of it.

Two operational traps deserve budget. When a product is out of stock, recorded sales understate demand, so a model trained on those rows learns to under-forecast the items most likely to sell out. And single-metric dashboards hide trade-offs: the model with the best RMSE can be the wrong pick when the business is judged on the 90th percentile, which is why the multi-metric evaluation work matters in production as much as in model selection.

Where a Simpler Model Beats the Newer One

A quantile regression method that ignored public holidays entirely placed second in the open data track and fourth in the definite data track at GEFCom2017 (arXiv 1809.03561). The lesson is not that holidays do not matter. A well-specified simple model with sound seasonality handling clears most of the field, and the marginal quarter of engineering effort is better spent on data than on a transformer.

Averaging helps too, and it is nearly free. Combining 650 point forecasts from one model family across different calibration window lengths, with LASSO selection and principal component averaging, produced lower MAE than simple mean, window-weighted averaging, LASSO-only and PCA-only approaches across four markets over roughly two and a half years out of sample (arXiv 2207.04794). If you already generate several candidate forecasts, pooling them beats another modeling cycle.

The value question sits one level above accuracy. The M6 competition had participants forecast 100 publicly traded assets with live evaluation, and the organizers reported that accurate relative-performance forecasting was hard, that consistently beating the market was harder, and that the link between submitted forecasts and investment decisions was limited (arXiv 2310.13357). Demand planning fails the same way: a model that improves MAPE but never changes a purchase order produces no return. Wire the forecast to a decision before you tune it.

Procurement Questions That Separate Vendors

Forecast vendors answer capability questions well and evidence questions poorly. Ask these in writing and score the answers.

• Which metric, on which dataset, against which baseline? An accuracy claim with no named baseline on your data shape has told you nothing usable.
• How long is the backtest window, and is it rolling origin? A single holdout period overstates accuracy in seasonal businesses.
• Which exogenous inputs can I attach? Ask for the full list, including promotion calendars, price changes and weather. Feature access determines whether the published accuracy gaps are even reachable.
• How are intermittent SKUs handled? Request the likelihood or the zero-inflation approach, not a marketing adjective.
• What is the pricing axis? Insist on a quote priced per active series and per forecast run so you can model assortment growth before signing.
• What happens on exit? Ask whether historical forecasts, feature definitions and evaluation results are exportable. Model portability is cheap to negotiate now and impossible later.

Whatever the answers, keep one internal capability regardless of vendor: the backtest harness and the segment-level error review. That is the asset that lets you compare a vendor’s forecast to a baseline, and to next year’s replacement. Build it before you build a model.

Sources and References

Sources cited while researching and writing this article:

• The Importance of Environmental Factors in Forecasting Australian Power Demand
• A comparative assessment of deep learning models for day-ahead load forecasting: Investigating key accuracy drivers
• Probabilistic Hierarchical Forecasting with Deep Poisson Mixtures
• Quantile Regression for Qualifying Match of GEFCom2017 Probabilistic Load Forecasting
• LASSO Principal Component Averaging — a fully automated approach for point forecast pooling
• Forecasting intermittent time series with Gaussian Processes and Tweedie likelihood
• Hierarchical Evaluation Function: A Multi-Metric Approach for Optimizing Demand Forecasting Models
• Stable and time-aware explanations for supply-chain forecasts: A lean XAI framework
• The M6 forecasting competition: Bridging the gap between forecasting and investment decisions