Abstract
It is a common practice to split a time series into an in-sample and pseudo-out-of-sample segments and estimate the out-of-sample loss for a given statistical model by evaluating forecasting performance over the pseudo-out-of-sample segment. We propose an alternative estimator of the out-of-sample loss, which, contrary to the conventional wisdom, utilizes criteria measured both in- and out-of-sample via a carefully constructed system of affine weights.
We prove that, provided that the time series is stationary, the proposed estimator is the best linear unbiased estimator of the out-of-sample loss and outperforms the conventional estimator in terms of sampling variability. Application of the optimal estimator to Diebold-Mariano type tests of predictive ability leads to a substantial power gain without increasing finite sample size distortions.
An extensive evaluation on real-world time series from the M4 forecasting competition confirms superiority of the proposed estimator and also demonstrates substantial robustness to violations of the underlying assumption of stationarity.