Performance metrics#

The sktime.performance_metrics module contains metrics for evaluating and tuning time series models.

All parameter estimators in sktime can be listed using the sktime.registry.all_estimators utility, using estimator_types="metric", optionally filtered by tags. Valid tags can be listed using sktime.registry.all_tags.

A full table with tag based search is also available on the Estimator Search Page (select “metric” in the “Estimator type” dropdown).

Metrics for assessing model performance.

Forecasting#

Point forecasts - classes#

MeanAbsoluteScaledError([multioutput, ...])

Mean absolute scaled error (MASE).

MedianAbsoluteScaledError([multioutput, ...])

Median absolute scaled error (MdASE).

MeanSquaredScaledError([multioutput, ...])

Mean squared scaled error (MSSE) or root mean squared scaled error (RMSSE).

MedianSquaredScaledError([multioutput, ...])

Median squared scaled error (MdSSE) or root median squared scaled error (RMdSSE).

MeanAbsoluteError([multioutput, multilevel, ...])

Mean absolute error (MAE).

MeanSquaredError([multioutput, multilevel, ...])

Mean squared error (MSE) or root mean squared error (RMSE).

MedianAbsoluteError([multioutput, ...])

Median absolute error (MdAE).

MedianSquaredError([multioutput, ...])

Median squared error (MdSE) or root median squared error (RMdSE).

GeometricMeanAbsoluteError([multioutput, ...])

Geometric mean absolute error (GMAE).

GeometricMeanSquaredError([multioutput, ...])

Geometric mean squared error (GMSE) or Root geometric mean squared error (RGMSE).

MeanAbsolutePercentageError([multioutput, ...])

Mean absolute percentage error (MAPE) or symmetric MAPE.

MedianAbsolutePercentageError([multioutput, ...])

Median absolute percentage error (MdAPE) or symmetric version.

MeanSquaredPercentageError([multioutput, ...])

Mean squared percentage error (MSPE), or RMSPE, or symmetric MSPE, RMSPE.

MedianSquaredPercentageError([multioutput, ...])

Median squared percentage error (MdSPE), or RMdSPE, or symmetric MdSPE, RMDsPE.

MeanRelativeAbsoluteError([multioutput, ...])

Mean relative absolute error (MRAE).

MedianRelativeAbsoluteError([multioutput, ...])

Median relative absolute error (MdRAE).

GeometricMeanRelativeAbsoluteError([...])

Geometric mean relative absolute error (GMRAE).

GeometricMeanRelativeSquaredError([...])

Geometric mean relative squared error (GMRSE).

MeanAsymmetricError([multioutput, ...])

Calculate mean of asymmetric loss function.

MeanLinexError([a, b, multioutput, ...])

Calculate mean linex error.

RelativeLoss([multioutput, multilevel, ...])

Calculate relative loss of forecast versus benchmark forecast.

Point forecasts - functions#

make_forecasting_scorer(func[, name, ...])

Create a metric class from a metric function.

mean_absolute_scaled_error(y_true, y_pred[, ...])

Mean absolute scaled error (MASE).

median_absolute_scaled_error(y_true, y_pred)

Median absolute scaled error (MdASE).

mean_squared_scaled_error(y_true, y_pred[, ...])

Mean squared scaled error (MSSE) or root mean squared scaled error (RMSSE).

median_squared_scaled_error(y_true, y_pred)

Median squared scaled error (MdSSE) or root median squared scaled error (RMdSSE).

mean_absolute_error(y_true, y_pred[, ...])

Mean absolute error (MAE).

mean_squared_error(y_true, y_pred[, ...])

Mean squared error (MSE) or root mean squared error (RMSE).

median_absolute_error(y_true, y_pred[, ...])

Median absolute error (MdAE).

median_squared_error(y_true, y_pred[, ...])

Median squared error (MdSE) or root median squared error (RMdSE).

geometric_mean_absolute_error(y_true, y_pred)

Geometric mean absolute error (GMAE).

geometric_mean_squared_error(y_true, y_pred)

Geometric mean squared error (GMSE) or Root geometric mean squared error (RGMSE).

mean_absolute_percentage_error(y_true, y_pred)

Mean absolute percentage error (MAPE) or symmetric version.

median_absolute_percentage_error(y_true, y_pred)

Median absolute percentage error (MdAPE) or symmetric version.

mean_squared_percentage_error(y_true, y_pred)

Mean squared percentage error (MSPE) or square root version.

median_squared_percentage_error(y_true, y_pred)

Median squared percentage error (MdSPE) or square root version.

mean_relative_absolute_error(y_true, y_pred)

Mean relative absolute error (MRAE).

median_relative_absolute_error(y_true, y_pred)

Median relative absolute error (MdRAE).

geometric_mean_relative_absolute_error(...)

Geometric mean relative absolute error (GMRAE).

geometric_mean_relative_squared_error(...[, ...])

Geometric mean relative squared error (GMRSE).

mean_asymmetric_error(y_true, y_pred[, ...])

Calculate mean of asymmetric loss function.

mean_linex_error(y_true, y_pred[, a, b, ...])

Calculate mean linex error.

relative_loss(y_true, y_pred[, ...])

Relative loss of forecast versus benchmark forecast for a given metric.

Quantile and interval forecasts#

PinballLoss([multioutput, score_average, alpha])

Pinball loss aka quantile loss for quantile/interval predictions.

EmpiricalCoverage([multioutput, ...])

Empirical coverage percentage for interval predictions.

ConstraintViolation([multioutput, ...])

Percentage of interval constraint violations for interval predictions.

Distribution forecasts#

AUCalibration([multioutput, multivariate])

Area under the calibration curve for distributional predictions.

CRPS([multioutput, multivariate])

Continuous rank probability score for distributional predictions.

LogLoss([multioutput, multivariate])

Logarithmic loss for distributional predictions.

SquaredDistrLoss([multioutput, multivariate])

Squared loss for distributional predictions.

Detection tasks#

Detection metrics can be applied to compare ground truth events with detected events, and ground truth segments with detected segments.

Detection metrics are typically designed for either:

  • point events, i.e., annotated time stamps, or

  • segments, i.e., annotated time intervals.

The metrics in sktime can be used for both types of detection tasks:

  • segmentation metrics interpret point events as segment boundaries, separating consecutive segments

  • point event metrics are applied to segments by considering their boundaries as point events

Event detection - anomalies, outliers#

DirectedChamfer([normalize])

Directed Chamfer distance between event points.

DirectedHausdorff()

Directed Hausdorff distance between event points.

DetectionCount([target, excess_only])

Count of detection, possibly in excess or deviation of a target count.

WindowedF1Score([margin])

F1-score for event detection, using a margin-based match criterion.

Segment detection#

RandIndex([use_loc])

Segmentation Rand Index metric.

Legacy detection metrics#

These metrics do not follow the standard API and will be deprecated in the future.

count_error(true_change_points, ...)

Error counting the difference in the number of change points.

hausdorff_error(true_change_points, ...[, ...])

Compute the Hausdorff distance between two sets of change points.

prediction_ratio(true_change_points, ...)

Prediction ratio is the ratio of number of predicted to true change points.