ForecastingBenchmark#
- class ForecastingBenchmark(id_format: str | None = None, backend=None, backend_params=None)[source]#
Forecasting benchmark.
Run a series of forecasters against a series of tasks defined via dataset loaders, cross validation splitting strategies and performance metrics, and return results as a df (as well as saving to file).
- Parameters:
- id_format: str, optional (default=None)
A regex used to enforce task/estimator ID to match a certain format
- backend{“dask”, “loky”, “multiprocessing”, “threading”}, by default None.
Runs parallel evaluate for each task if specified.
“None”: executes loop sequentally, simple list comprehension
“loky”, “multiprocessing” and “threading”: uses
joblib.Parallel
loops“joblib”: custom and 3rd party
joblib
backends, e.g.,spark
“dask”: uses
dask
, requiresdask
package in environment“dask_lazy”: same as “dask”,
but changes the return to (lazy)
dask.dataframe.DataFrame
.Recommendation: Use “dask” or “loky” for parallel evaluate. “threading” is unlikely to see speed ups due to the GIL and the serialization backend (
cloudpickle
) for “dask” and “loky” is generally more robust than the standardpickle
library used in “multiprocessing”.- backend_paramsdict, optional
additional parameters passed to the backend as config. Directly passed to
utils.parallel.parallelize
. Valid keys depend on the value ofbackend
:“None”: no additional parameters,
backend_params
is ignored“loky”, “multiprocessing” and “threading”: default
joblib
backends
any valid keys for
joblib.Parallel
can be passed here, e.g.,n_jobs
, with the exception ofbackend
which is directly controlled bybackend
. Ifn_jobs
is not passed, it will default to-1
, other parameters will default tojoblib
defaults. - “joblib”: custom and 3rd partyjoblib
backends, e.g.,spark
. any valid keys forjoblib.Parallel
can be passed here, e.g.,n_jobs
,backend
must be passed as a key ofbackend_params
in this case. Ifn_jobs
is not passed, it will default to-1
, other parameters will default tojoblib
defaults. - “dask”: any valid keys fordask.compute
can be passed, e.g.,scheduler
Methods
add_estimator
(estimator[, estimator_id])Register an estimator to the benchmark.
add_task
(dataset_loader, cv_splitter, scorers)Register a forecasting task to the benchmark.
run
(output_file)Run the benchmark.
- add_task(dataset_loader: Callable, cv_splitter: BaseSplitter, scorers: list[BaseMetric], task_id: str | None = None)[source]#
Register a forecasting task to the benchmark.
- Parameters:
- dataset_loaderCallable
A function which returns a dataset, like from sktime.datasets.
- cv_splitterBaseSplitter object
Splitter used for generating validation folds.
- scorersa list of BaseMetric objects
Each BaseMetric output will be included in the results.
- task_idstr, optional (default=None)
Identifier for the benchmark task. If none given then uses dataset loader name combined with cv_splitter class name.
- Returns:
- A dictionary of benchmark results for that forecaster
- add_estimator(estimator: BaseEstimator, estimator_id: str | None = None)[source]#
Register an estimator to the benchmark.
- Parameters:
- estimatorDict, List or BaseEstimator object
Estimator to add to the benchmark. If Dict, keys are estimator_ids used to customise identifier ID and values are estimators. If List, each element is an estimator. estimator_ids are generated automatically using the estimator’s class name.
- estimator_idstr, optional (default=None)
Identifier for estimator. If none given then uses estimator’s class name.