AUCalibration

AUCalibration

class AUCalibration(multioutput='uniform_average', multivariate=False)[source]

Area under the calibration curve for distributional predictions.

Computes the unsigned area between the calibration curve and the diagonal.

The calibration curve is the cumulative curve of the sample of predictive cumulative distribution functions evaluated at the true values.

Mathematically, let \(d_1, \dots, d_N\) be the predictive distributions, let \(y_1, \dots, y_N\) be the true values, and let \(F_i\) be the cumulative distribution function of \(d_i\).

Define the calibration sample as \(c_i := F_i(y_i)\), for \(i = 1, \dots, N\). For perfect predictions, the sample of \(c_i\) will be uniformly distributed on [0, 1], and i.i.d. from that uniform distribution.

Let \(c_{(i)}\) be the \(i\)-th order statistic of the sample of \(c_i\), i.e., the \(i\)-th smallest value in the sample.

The (unsigned) area under the calibration curve - or, more precisely, between the diagonal and the calibration curve - is defined as

\[\frac{1}{N} \sum_{i=1}^N \left| c_{(i)} - \frac{i}{N} \right|.\]

• evaluate returns the unsigned area between the calibration curve and the diagonal, i.e., the above quantity.

• evaluate_by_index returns, for the \(i\)-th test sample, the value \(\left| c_i - \frac{r_i}{N} \right|\), where \(r_i\) is the rank of \(c_i\) in the sample of \(c_i\). In case of ties, tied ranks are averaged.

• multivariate controls averaging over variables.

Parameters:

multioutput‘uniform_average’ (default), 1D array-like, or ‘raw_values’

Whether and how to aggregate metric for multivariate (multioutput) data.

• If 'uniform_average' (default), errors of all outputs are averaged with uniform weight.

• If 1D array-like, errors are averaged across variables, with values used as averaging weights (same order).

• If 'raw_values', does not average across variables (outputs), per-variable errors are returned.

multivariatebool, optional, default=False

• if True, behaves as multivariate metric (sum of scores): the metric is computed for entire row, results one score per row

• if False, is univariate metric, per variable: the metric is computed per variable marginal, results in many scores per row

Methods

__call__(y_true, y_pred, **kwargs)

Calculate metric value using underlying metric function.

__call__(y_true, y_pred, **kwargs)[source]

Calculate metric value using underlying metric function.

Parameters:

y_truepd.Series, pd.DataFrame or np.array of shape (fh,) or (fh, n_outputs) where fh is the forecasting horizon

Ground truth (correct) target values.

y_predreturn object of probabilistic prediction method scitype:y_pred

must be at fh and for variables equal to those in y_true.

Returns:

lossfloat or 1-column pd.DataFrame with calculated metric value(s)

metric is always averaged (arithmetic) over fh values if multioutput = “raw_values”,

will have a column level corresponding to variables in y_true

if multioutput = multioutput = “uniform_average” or or array-like

entries will be averaged over output variable column

if score_average = False,

will have column levels corresponding to quantiles/intervals

if score_average = True,

entries will be averaged over quantiles/interval column