CRPS

CRPS

class CRPS(multioutput='uniform_average', multivariate=False)[source]

Continuous rank probability score for distributional predictions.

Also known as:

• integrated squared loss (ISL)

• integrated Brier loss (IBL)

• energy loss

For a predictive distribution \(d\) and a ground truth value \(y\), the CRPS is defined as \(L(y, d) := \mathbb{E}_{Y \sim d}|Y-y| - \frac{1}{2} \mathbb{E}_{X,Y \sim d}|X-Y|\).

• evaluate computes the average test sample loss.

• evaluate_by_index produces the loss sample by test data point.

• multivariate controls averaging over variables.

Parameters:

multioutput‘uniform_average’ (default), 1D array-like, or ‘raw_values’

Whether and how to aggregate metric for multivariate (multioutput) data.

• If 'uniform_average' (default), errors of all outputs are averaged with uniform weight.

• If 1D array-like, errors are averaged across variables, with values used as averaging weights (same order).

• If 'raw_values', does not average across variables (outputs), per-variable errors are returned.

multivariatebool, optional, default=False

• if True, behaves as multivariate CRPS: the score is computed for entire row, results one score per row

• if False, is univariate CRPS: the score is computed per variable marginal, results in many scores per row

Methods

__call__(y_true, y_pred, **kwargs)

Calculate metric value using underlying metric function.

__call__(y_true, y_pred, **kwargs)[source]

Calculate metric value using underlying metric function.

Parameters:

y_truepd.Series, pd.DataFrame or np.array of shape (fh,) or (fh, n_outputs) where fh is the forecasting horizon

Ground truth (correct) target values.

y_predreturn object of probabilistic prediction method scitype:y_pred

must be at fh and for variables equal to those in y_true.

Returns:

lossfloat or 1-column pd.DataFrame with calculated metric value(s)

metric is always averaged (arithmetic) over fh values if multioutput = “raw_values”,

will have a column level corresponding to variables in y_true

if multioutput = multioutput = “uniform_average” or or array-like

entries will be averaged over output variable column

if score_average = False,

will have column levels corresponding to quantiles/intervals

if score_average = True,

entries will be averaged over quantiles/interval column