Time-Series Forecasting

Electricity demand may repeat each week; traffic changes within a day; rainfall can be irregular. A reusable forecasting model must recognize these patterns from a supplied history even when the scale, sampling frequency and prediction horizon change.

The evaluation covers 97 configurations across 23 datasets and seven domains. Forecasts range from six to 900 future steps. For an electricity operator, the middle forecast describes typical demand while an upper forecast describes unusually high demand. Both the central estimate and the uncertainty around it matter.

From observations to a distribution

Follow a history through the reference model. The waveform above is a separate synthetic example.

1. History + mask≤ 2,048 past points
   One variate at a time
2. Normalize → patches32 points per token
   Past-based normalization
3. 14 transformer layersConnect patterns across time
   Keep track of temporal order
4. 64-point output patch9 quantiles per point
   Pinball training objective
5. Forecast evaluatorRequested horizon
   Restore original units

What changes between training and evaluation?

Training teaches the model to predict future patches from past ones. Evaluation supplies a new history and asks for nine quantiles without changing the model. This baseline reads one variable at a time, so it cannot directly connect several related signals.

The allowed training corpus contains about 63 billion observations, with climate and building data contributing much of its volume. Simply choosing series uniformly can let these sources overwhelm smaller but useful domains. Balancing sources is therefore a scientific choice, not just a data-loading detail.

Models begin from random weights. Evaluated series stay outside training, and a forecast may use only observations before its own cutoff. Overlapping windows need particular care: one window’s future must not become another’s source of answers. The interactive waveform is synthetic and illustrates this boundary.

The model predicts nine quantiles, including the median. A 90th-percentile forecast is an upper estimate that, when calibrated, should be exceeded about once in ten observations. Pinball loss penalizes predicting this upper quantile too low more heavily than predicting it equally far too high.

Median accuracy is measured with scaled absolute error; quantile accuracy uses weighted quantile loss. Both are compared with a seasonal forecast that repeats a past pattern, then combined across datasets. If relative errors are 0.80 and 0.45, reward is 1 − √(0.80 × 0.45) = 0.40. Better uncertainty alone cannot hide poor central predictions.

The roughly 51-million-parameter reference groups consecutive observations into patches, like chunks of a sentence. A transformer relates those patches to find recurring patterns and changes over time. It then predicts future patches at all nine quantiles.

Normalization helps it compare histories with different units and scales; forecasts are converted back afterward. The baseline handles one variable at a time and reads at most 2,048 past observations. Its balanced mixture includes diverse real sources and synthetic series, encouraging broader exposure than the largest datasets alone would provide.

The task-trained reference reaches a reward of about 0.275. Larger released forecasting models reach roughly 0.42–0.45, providing context for remaining headroom. They use different training data and procedures, so this is not a controlled comparison of architectures, and their pretrained weights cannot initialize a candidate here.

The smaller models improve on seasonal repetition at short horizons but still struggle with median accuracy at 480–900 steps. Long forecasts demand more than extending a short pattern: errors and uncertainty have more time to grow.

Expanding the corpus under the same model and uniform-sampling setup raises reward from about 0.206 to 0.217. The balanced recipe reaches 0.275, but also trains longer with more hardware. Its larger improvement cannot be credited to sampling alone.

The final run also shows diminishing returns: an earlier model scores about 0.274. These comparisons helped select the recipe and included test results, so they are development evidence rather than independent confirmation. Repeated runs and a fresh held-out comparison would strengthen the conclusion.

Longer history may help a model identify slow cycles; jointly modeling several variables may help it connect related signals. Neither is an established fix here. Compare one change at a time with the same data and computation, then inspect short and long horizons separately. For uncertainty, check how often observations actually fall inside the forecast ranges across many series.