Metric

AUCalibration

Area under the calibration curve for distributional predictions.

Computes the unsigned area between the calibration curve and the diagonal.

The calibration curve is the cumulative curve of the sample of predictive cumulative distribution functions evaluated at the true values.

Mathematically, let \(d_1, \dots, d_N\) be the predictive distributions, let \(y_1, \dots, y_N\) be the true values, and let \(F_i\) be the cumulative distribution function of \(d_i\).

Define the calibration sample as \(c_i:= F_i(y_i)\), for \(i = 1, \dots, N\). For perfect predictions, the sample of \(c_i\) will be uniformly distributed on [0, 1], and i.i.d. from that uniform distribution.

Let \(c_{(i)}\) be the \(i\)-th order statistic of the sample of \(c_i\), i.e., the \(i\)-th smallest value in the sample.

The (unsigned) area under the calibration curve - or, more precisely, between the diagonal and the calibration curve - is defined as

\[\frac{1}{N} \sum_{i=1}^N \left| c_{(i)} - \frac{i}{N} \right|.\]

evaluate returns the unsigned area between the calibration curve and the diagonal, i.e., the above quantity.
evaluate_by_index returns, for the \(i\)-th test sample, the value \(\left| c_i - \frac{r_i}{N} \right|\), where \(r_i\) is the rank of \(c_i\) in the sample of \(c_i\). In case of ties, tied ranks are averaged.
multivariate controls averaging over variables.

Quickstart

python

from sktime.performance_metrics.forecasting.probabilistic import AUCalibration

estimator = AUCalibration(multioutput='uniform_average', multivariate=False)

Parameters(2)

multioutput‘uniform_average’ (default), 1D array-like, or ‘raw_values’: Whether and how to aggregate metric for multivariate (multioutput) data.
If 'uniform_average' (default), errors of all outputs are averaged with uniform weight.
If 1D array-like, errors are averaged across variables, with values used as averaging weights (same order).
If 'raw_values', does not average across variables (outputs), per-variable errors are returned.
multivariatebool, optional, default=False: if True, behaves as multivariate metric (sum of scores): the metric is computed for entire row, results one score per row
if False, is univariate metric, per variable: the metric is computed per variable marginal, results in many scores per row