Back to models
Metric

PinballLoss

API referenceSource
Multivariate

Pinball loss aka quantile loss for quantile/interval predictions.

Can be used for both quantile and interval predictions.

For a quantile prediction \(\widehat{y} at quantile point:math:\)alpha`, and a ground truth value \(y\), the pinball loss is defined as \(L_\alpha(y, \widehat{y}):= (y - \widehat{y}) \cdot (\alpha - H(y - \widehat{y}))\), where \(H\) is the Heaviside step function defined as \(H(x) = 1\) if \(x \ge 0\) and \(H(x) = 0\) otherwise.

For a symmetric prediction interval \(I = [\widehat{y}_{\alpha}, \widehat{y}_{1 - \alpha}]\), the pinball loss is defined as \(L_\alpha(y, I):= L_\alpha(y, \widehat{y}_{\alpha}) + L_{1 - \alpha}(y, \widehat{y}_{1 - \alpha})\), or, in terms of coverage \(c = 1 - 2\alpha\), as \(L_c(y, I):= L_{1/2 - c/2}(y, a) + L_{1/2 + c/2}(y, b)\), if we write \(I = [a, b]\).

  • evaluate computes the average test sample loss.

  • evaluate_by_index produces the loss sample by test data point.

  • multivariate controls averaging over variables.

  • score_average controls averaging over quantiles/intervals.

Quickstart

python
from sktime.performance_metrics.forecasting.probabilistic import PinballLoss

estimator = PinballLoss(multioutput='uniform_average', score_average=True, alpha=None)

Parameters(3)

multioutput‘uniform_average’ (default), 1D array-like, or ‘raw_values’

Whether and how to aggregate metric for multivariate (multioutput) data.

  • If 'uniform_average' (default), errors of all outputs are averaged with uniform weight.

  • If 1D array-like, errors are averaged across variables, with values used as averaging weights (same order).

  • If 'raw_values', does not average across variables (outputs), per-variable errors are returned.

score_averagebool, optional, default = True

specifies whether scores for each quantile should be averaged.

  • If True, metric/loss is averaged over all quantiles present in y_pred.

  • If False, metric/loss is not averaged over quantiles.

alpha (optional)float, list of float, or 1D array-like, default=None

quantiles to evaluate metric at. Can be specified if no explicit quantiles are present in the direct use of the metric, for instance in benchmarking via evaluate, or tuning via ForecastingGridSearchCV.

Examples

>>> import numpy as np
>>> import pandas as pd
>>> from sktime.performance_metrics.forecasting.probabilistic import PinballLoss
>>> y_true = pd. Series ([3, - 0.5, 2, 7, 2 ])
>>> y_pred = pd. DataFrame ({
... ('Quantiles', 0.05): [1.25, 0, 1, 4, 0.625 ],
... ('Quantiles', 0.5): [2.5, 0, 2, 8, 1.25 ],
... ('Quantiles', 0.95): [3.75, 0, 3, 12, 1.875 ],
... })
>>> pl = PinballLoss ()
>>> pl (y_true, y_pred) np.float64(0.1791666666666667)
>>> pl = PinballLoss (score_average = False)
>>> pl (y_true, y_pred). to_numpy () array([0.16625, 0.275, 0.09625])
>>> y_true = pd. DataFrame ({
... "Quantiles1": [3, - 0.5, 2, 7, 2 ],
... "Quantiles2": [4, 0.5, 3, 8, 3 ],
... })
>>> y_pred = pd. DataFrame ({
... ('Quantiles1', 0.05): [1.5, - 1, 1, 4, 0.65 ],
... ('Quantiles1', 0.5): [2.5, 0, 2, 8, 1.25 ],
... ('Quantiles1', 0.95): [3.5, 4, 3, 12, 1.85 ],
... ('Quantiles2', 0.05): [2.5, 0, 2, 8, 1.25 ],
... ('Quantiles2', 0.5): [5.0, 1, 4, 16, 2.5 ],
... ('Quantiles2', 0.95): [7.5, 2, 6, 24, 3.75 ],
... })
>>> pl = PinballLoss (multioutput = 'raw_values')
>>> pl (y_true, y_pred). to_numpy () array([0.16233333, 0.465 ])
>>> pl = PinballLoss (multioutput = np. array ([0.3, 0.7 ]))
>>> pl (y_true, y_pred) np.float64(0.3742000000000001)