Metric

RelativeLoss

Calculate relative loss of forecast versus benchmark forecast.

Applies a forecasting performance metric to a set of forecasts and benchmark forecasts and reports ratio of the metric from the forecasts to the the metric from the benchmark forecasts. Relative loss output is non-negative floating point. The best value is 0.0.

If the score of the benchmark predictions for a given loss function is zero then a large value is returned.

This function allows the calculation of scale-free relative loss metrics. Unlike mean absolute scaled error (MASE) the function calculates the scale-free metric relative to a defined loss function on a benchmark method instead of the in-sample training data. Like MASE, metrics created using this function can be used to compare forecast methods on a single series and also to compare forecast accuracy between series.

This is useful when a scale-free comparison is beneficial but the training data used to generate some (or all) predictions is unknown such as when comparing the loss of 3rd party forecasts or surveys of professional forecasters.

Only metrics that do not require y_train are currently supported.

Schnellstart

python

from sktime.performance_metrics.forecasting import RelativeLoss

estimator = RelativeLoss(multioutput='uniform_average', multilevel='uniform_average', relative_loss_function=<function mean_absolute_error>, by_index=False)

Parameter(4)

relative_loss_functionfunction: Function to use in calculation relative loss.
multioutput‘uniform_average’ (default), 1D array-like, or ‘raw_values’: Whether and how to aggregate metric for multivariate (multioutput) data.
If 'uniform_average' (default), errors of all outputs are averaged with uniform weight.
If 1D array-like, errors are averaged across variables, with values used as averaging weights (same order).
If 'raw_values', does not average across variables (outputs), per-variable errors are returned.
multilevel{‘raw_values’, ‘uniform_average’, ‘uniform_average_time’}: How to aggregate the metric for hierarchical data (with levels).
If 'uniform_average' (default), errors are mean-averaged across levels.
If 'uniform_average_time', metric is applied to all data, ignoring level index.
If 'raw_values', does not average errors across levels, hierarchy is retained.
by_indexbool, default=False: Controls averaging over time points in direct call to metric object.
If False (default), direct call to the metric object averages over time points, equivalent to a call of the evaluate method.
If True, direct call to the metric object evaluates the metric at each time point, equivalent to a call of the evaluate_by_index method.

Beispiele

>>> import numpy as np
>>> from sktime.performance_metrics.forecasting import RelativeLoss
>>> from sktime.performance_metrics.forecasting import mean_squared_error
>>> y_true = np. array ([3, - 0.5, 2, 7, 2 ])
>>> y_pred = np. array ([2.5, 0.0, 2, 8, 1.25 ])
>>> y_pred_benchmark = y_pred * 1.1
>>> relative_mae = RelativeLoss ()
>>> relative_mae (y_true, y_pred, y_pred_benchmark = y_pred_benchmark) np.float64(0.8148148148148147)
>>> relative_mse = RelativeLoss (relative_loss_function = mean_squared_error)
>>> relative_mse (y_true, y_pred, y_pred_benchmark = y_pred_benchmark) np.float64(0.5178095088655261)
>>> y_true = np. array ([[0.5, 1 ], [- 1, 1 ], [7, - 6 ]])
>>> y_pred = np. array ([[0, 2 ], [- 1, 2 ], [8, - 5 ]])
>>> y_pred_benchmark = y_pred * 1.1
>>> relative_mae = RelativeLoss ()
>>> relative_mae (y_true, y_pred, y_pred_benchmark = y_pred_benchmark) np.float64(0.8490566037735847)
>>> relative_mae = RelativeLoss (multioutput = 'raw_values')
>>> relative_mae (y_true, y_pred, y_pred_benchmark = y_pred_benchmark) array([0.625, 1.03448276])
>>> relative_mae = RelativeLoss (multioutput = [0.3, 0.7 ])
>>> relative_mae (y_true, y_pred, y_pred_benchmark = y_pred_benchmark) np.float64(0.927272727272727)

Referenzen

Hyndman, R. J and Koehler, A. B. (2006). “Another look at measures of forecast accuracy”, International Journal of Forecasting, Volume 22, Issue 4.