Metric

MeanAbsoluteScaledError

Mean absolute scaled error (MASE).

MASE output is non-negative floating point, in fractional units relative to a specified denominator. Lower is better, and the lowest possible value is 0.0.

For a univariate, non-hierarchical sample of true values \(y_1, \dots, y_n\), pred values \(\widehat{y}_1, \dots, \widehat{y}_n\) (in \(\mathbb{R}\)), and in-sample training values \(y_1^{\text{train}}, \dots, y_m^{\text{train}}\), evaluate or call returns the Mean Absolute Scaled Error (MASE), defined as:

\[\]

text{MASE} = frac{: frac{1}{n} sum_{i=1}^n |y_i - \widehat{y}_i|
}{: frac{1}{m - s} sum_{j=s+1}^m |y^{\text{train}}_j - y^{\text{train}}_{j-s}|

}

where \(s\) is the seasonal periodicity (sp), and the denominator is the in-sample mean absolute error of the seasonal naive forecast.

To avoid division by zero, the denominator above is replaced by eps if it is smaller than eps; the value of eps defaults to np.finfo(np.float64).eps if not specified.

Like other scaled performance metrics, this scale-free error metric can be used to compare forecast methods on a single series and also to compare forecast accuracy between series.

This metric is well suited to intermittent-demand series because it will not give infinite or undefined values unless the training data is a flat timeseries. In this case the function returns a large value instead of inf.

Works with multioutput (multivariate) timeseries data with homogeneous seasonal periodicity.

Quickstart

python

from sktime.performance_metrics.forecasting import MeanAbsoluteScaledError

estimator = MeanAbsoluteScaledError(multioutput='uniform_average', multilevel='uniform_average', sp=1, by_index=False, eps=None)

Parameters(5)

spint, default = 1: Seasonal periodicity of the data
epsfloat, default=None: Numerical epsilon used in denominator to avoid division by zero. Absolute values smaller than eps are replaced by eps. If None, defaults to np.finfo(np.float64).eps
multioutput‘uniform_average’ (default), 1D array-like, or ‘raw_values’: Whether and how to aggregate metric for multivariate (multioutput) data.
If 'uniform_average' (default), errors of all outputs are averaged with uniform weight.
If 1D array-like, errors are averaged across variables, with values used as averaging weights (same order).
If 'raw_values', does not average across variables (outputs), per-variable errors are returned.
multilevel{‘raw_values’, ‘uniform_average’, ‘uniform_average_time’}: How to aggregate the metric for hierarchical data (with levels).
If 'uniform_average' (default), errors are mean-averaged across levels.
If 'uniform_average_time', metric is applied to all data, ignoring level index.
If 'raw_values', does not average errors across levels, hierarchy is retained.
by_indexbool, default=False: Controls averaging over time points in direct call to metric object.
If False (default), direct call to the metric object averages over time points, equivalent to a call of the evaluate method.
If True, direct call to the metric object evaluates the metric at each time point, equivalent to a call of the evaluate_by_index method.

Examples

>>> import numpy as np
>>> from sktime.performance_metrics.forecasting import MeanAbsoluteScaledError
>>> y_train = np. array ([5, 0.5, 4, 6, 3, 5, 2 ])
>>> y_true = np. array ([3, - 0.5, 2, 7, 2 ])
>>> y_pred = np. array ([2.5, 0.0, 2, 8, 1.25 ])
>>> mase = MeanAbsoluteScaledError ()
>>> mase (y_true, y_pred, y_train = y_train) np.float64(0.18333333333333335)
>>> y_train = np. array ([[0.5, 1 ], [- 1, 1 ], [7, - 6 ]])
>>> y_true = np. array ([[0.5, 1 ], [- 1, 1 ], [7, - 6 ]])
>>> y_pred = np. array ([[0, 2 ], [- 1, 2 ], [8, - 5 ]])
>>> mase (y_true, y_pred, y_train = y_train) np.float64(0.18181818181818182)
>>> mase = MeanAbsoluteScaledError (multioutput = 'raw_values')
>>> mase (y_true, y_pred, y_train = y_train) array([0.10526316, 0.28571429])
>>> mase = MeanAbsoluteScaledError (multioutput = [0.3, 0.7 ])
>>> mase (y_true, y_pred, y_train = y_train) np.float64(0.21935483870967742)

References

Hyndman, R. J and Koehler, A. B. (2006). “Another look at measures of forecast accuracy”, International Journal of Forecasting, Volume 22, Issue 4.

Hyndman, R. J. (2006). “Another look at forecast accuracy metrics for intermittent demand”, Foresight, Issue 4.

Makridakis, S., Spiliotis, E. and Assimakopoulos, V. (2020) “The M4 Competition: 100,000 time series and 61 forecasting methods”, International Journal of Forecasting, Volume 3.