PinballLoss

class PinballLoss(multioutput='uniform_average', score_average=True, alpha=None)[source]

Pinball loss aka quantile loss for quantile/interval predictions.

Can be used for both quantile and interval predictions.

For a quantile prediction \(\widehat{y} at quantile point :math:\)alpha`, and a ground truth value \(y\), the pinball loss is defined as \(L_\alpha(y, \widehat{y}) := (y - \widehat{y}) \cdot (\alpha - H(y - \widehat{y}))\), where \(H\) is the Heaviside step function defined as \(H(x) = 1\) if \(x \ge 0\) and \(H(x) = 0\) otherwise.

For a symmetric prediction interval \(I = [\widehat{y}_{\alpha}, \widehat{y}_{1 - \alpha}]\), the pinball loss is defined as \(L_\alpha(y, I) := L_\alpha(y, \widehat{y}_{\alpha}) + L_{1 - \alpha}(y, \widehat{y}_{1 - \alpha})\), or, in terms of coverage \(c = 1 - 2\alpha\), as \(L_c(y, I) := L_{1/2 - c/2}(y, a) + L_{1/2 + c/2}(y, b)\), if we write \(I = [a, b]\).

evaluate computes the average test sample loss.
evaluate_by_index produces the loss sample by test data point.
multivariate controls averaging over variables.
score_average controls averaging over quantiles/intervals.

Parameters:

multioutput‘uniform_average’ (default), 1D array-like, or ‘raw_values’

Whether and how to aggregate metric for multivariate (multioutput) data.

If 'uniform_average' (default), errors of all outputs are averaged with uniform weight.
If 1D array-like, errors are averaged across variables, with values used as averaging weights (same order).
If 'raw_values', does not average across variables (outputs), per-variable errors are returned.

score_averagebool, optional, default = True

specifies whether scores for each quantile should be averaged.

If True, metric/loss is averaged over all quantiles present in y_pred.
If False, metric/loss is not averaged over quantiles.

alpha (optional)float, list of float, or 1D array-like, default=None

quantiles to evaluate metric at. Can be specified if no explicit quantiles are present in the direct use of the metric, for instance in benchmarking via evaluate, or tuning via ForecastingGridSearchCV.

Examples

>>> import numpy as np
>>> import pandas as pd
>>> from sktime.performance_metrics.forecasting.probabilistic import PinballLoss
>>> y_true = pd.Series([3, -0.5, 2, 7, 2])
>>> y_pred = pd.DataFrame({
...     ('Quantiles', 0.05): [1.25, 0, 1, 4, 0.625],
...     ('Quantiles', 0.5): [2.5, 0, 2, 8, 1.25],
...     ('Quantiles', 0.95): [3.75, 0, 3, 12, 1.875],
... })
>>> pl = PinballLoss()
>>> pl(y_true, y_pred)
np.float64(0.1791666666666667)
>>> pl = PinballLoss(score_average=False)
>>> pl(y_true, y_pred).to_numpy()
array([0.16625, 0.275  , 0.09625])
>>> y_true = pd.DataFrame({
...     "Quantiles1": [3, -0.5, 2, 7, 2],
...     "Quantiles2": [4, 0.5, 3, 8, 3],
... })
>>> y_pred = pd.DataFrame({
...     ('Quantiles1', 0.05): [1.5, -1, 1, 4, 0.65],
...     ('Quantiles1', 0.5): [2.5, 0, 2, 8, 1.25],
...     ('Quantiles1', 0.95): [3.5, 4, 3, 12, 1.85],
...     ('Quantiles2', 0.05): [2.5, 0, 2, 8, 1.25],
...     ('Quantiles2', 0.5): [5.0, 1, 4, 16, 2.5],
...     ('Quantiles2', 0.95): [7.5, 2, 6, 24, 3.75],
... })
>>> pl = PinballLoss(multioutput='raw_values')
>>> pl(y_true, y_pred).to_numpy()
array([0.16233333, 0.465     ])
>>> pl = PinballLoss(multioutput=np.array([0.3, 0.7]))
>>> pl(y_true, y_pred)
np.float64(0.3742000000000001)

Methods

__call__(y_true, y_pred, **kwargs)

Calculate metric value using underlying metric function.

__call__(y_true, y_pred, **kwargs)[source]

Calculate metric value using underlying metric function.

Parameters:

y_truepd.Series, pd.DataFrame or np.array of shape (fh,) or (fh, n_outputs) where fh is the forecasting horizon: Ground truth (correct) target values.
y_predreturn object of probabilistic prediction method scitype:y_pred: must be at fh and for variables equal to those in y_true.

Returns:

lossfloat or 1-column pd.DataFrame with calculated metric value(s)

metric is always averaged (arithmetic) over fh values if multioutput = “raw_values”,

will have a column level corresponding to variables in y_true

if multioutput = multioutput = “uniform_average” or or array-like: entries will be averaged over output variable column
if score_average = False,: will have column levels corresponding to quantiles/intervals
if score_average = True,: entries will be averaged over quantiles/interval column