ForecastingBenchmark#

class ForecastingBenchmark(id_format: str | None = None, backend=None, backend_params=None)[source]#

Forecasting benchmark.

Run a series of forecasters against a series of tasks defined via dataset loaders, cross validation splitting strategies and performance metrics, and return results as a df (as well as saving to file).

Parameters:
id_format: str, optional (default=None)

A regex used to enforce task/estimator ID to match a certain format

backend{“dask”, “loky”, “multiprocessing”, “threading”}, by default None.

Runs parallel evaluate for each task if specified.

  • “None”: executes loop sequentally, simple list comprehension

  • “loky”, “multiprocessing” and “threading”: uses joblib.Parallel loops

  • “joblib”: custom and 3rd party joblib backends, e.g., spark

  • “dask”: uses dask, requires dask package in environment

  • “dask_lazy”: same as “dask”,

but changes the return to (lazy) dask.dataframe.DataFrame.

Recommendation: Use “dask” or “loky” for parallel evaluate. “threading” is unlikely to see speed ups due to the GIL and the serialization backend (cloudpickle) for “dask” and “loky” is generally more robust than the standard pickle library used in “multiprocessing”.

backend_paramsdict, optional

additional parameters passed to the backend as config. Directly passed to utils.parallel.parallelize. Valid keys depend on the value of backend:

  • “None”: no additional parameters, backend_params is ignored

  • “loky”, “multiprocessing” and “threading”: default joblib backends

any valid keys for joblib.Parallel can be passed here, e.g., n_jobs, with the exception of backend which is directly controlled by backend. If n_jobs is not passed, it will default to -1, other parameters will default to joblib defaults. - “joblib”: custom and 3rd party joblib backends, e.g., spark. any valid keys for joblib.Parallel can be passed here, e.g., n_jobs, backend must be passed as a key of backend_params in this case. If n_jobs is not passed, it will default to -1, other parameters will default to joblib defaults. - “dask”: any valid keys for dask.compute can be passed, e.g., scheduler

Methods

add_estimator(estimator[, estimator_id])

Register an estimator to the benchmark.

add_task(dataset_loader, cv_splitter, scorers)

Register a forecasting task to the benchmark.

run(output_file)

Run the benchmark.

add_task(dataset_loader: Callable, cv_splitter: BaseSplitter, scorers: list[BaseMetric], task_id: str | None = None, error_score=nan, strategy='refit')[source]#

Register a forecasting task to the benchmark.

Parameters:
dataset_loaderCallable

A function which returns a dataset, like from sktime.datasets.

cv_splitterBaseSplitter object

Splitter used for generating validation folds.

scorersa list of BaseMetric objects

Each BaseMetric output will be included in the results.

task_idstr, optional (default=None)

Identifier for the benchmark task. If none given then uses dataset loader name combined with cv_splitter class name.

error_score“raise” or numeric, default=np.nan

Value to assign to the score if an exception occurs in estimator fitting. If set to “raise”, the exception is raised. If a numeric value is given, FitFailedWarning is raised.

strategy{“refit”, “update”, “no-update_params”}, optional, default=”refit”

defines the ingestion mode when the forecaster sees new data when window expands “refit” = forecaster is refitted to each training window “update” = forecaster is updated with training window data, in sequence provided “no-update_params” = fit to first training window, re-used without fit or update

Returns:
A dictionary of benchmark results for that forecaster
add_estimator(estimator: BaseEstimator, estimator_id: str | None = None)[source]#

Register an estimator to the benchmark.

Parameters:
estimatorDict, List or BaseEstimator object

Estimator to add to the benchmark. If Dict, keys are estimator_ids used to customise identifier ID and values are estimators. If List, each element is an estimator. estimator_ids are generated automatically using the estimator’s class name.

estimator_idstr, optional (default=None)

Identifier for estimator. If none given then uses estimator’s class name.

run(output_file: str) DataFrame[source]#

Run the benchmark.

Parameters:
output_filestr

Path to write results output file to.

Returns:
pandas DataFrame of results