ForecastingOptunaSearchCV

class ForecastingOptunaSearchCV(forecaster, cv, param_grid, scoring=None, strategy='refit', update_behaviour='full_refit', refit=True, verbose=0, return_n_best_forecasters=1, backend='loky', error_score=nan, n_evals=100, sampler=None)[source]

Perform Optuna search cross-validation to find optimal model hyperparameters.

Experimental: This feature is under development and interfaces may change.

In fit, this estimator uses the optuna base search algorithm applied to the sktime evaluate benchmarking output.

param_grid is used to parametrize the search space, over parameters of the passed forecaster, via set_params.

The remaining parameters are passed directly to evaluate, to obtain the primary optimization outcome as the aggregate scoring metric specified on the evaluation schema.

Parameters:

forecastersktime forecaster, BaseForecaster instance or interface compatible

The forecaster to tune, must implement the sktime forecaster interface. sklearn regressors can be used, but must first be converted to forecasters via one of the reduction compositors, e.g., via make_reduction

cvsktime time series splitter

Re-sampling strategy for cross-validation, must be an instance of a sktime time series splitter, e.g. SlidingWindowSplitter()

param_griddict of optuna samplers

Dictionary with parameters names as keys and lists of parameter distributions from which to sample parameter values. e.g. {“forecaster”: optuna.distributions.CategoricalDistribution( (STLForecaster(), ThetaForecaster())}

scoringsktime metric (BaseMetric), str, or callable, optional (default=MAPE)

scoring metric to use in tuning the forecaster

sktime metric objects (BaseMetric) descendants can be searched with the registry.all_estimators search utility, for instance via all_estimators("metric", as_dataframe=True)
If callable, must have signature (y_true: 1D np.ndarray, y_pred: 1D np.ndarray) -> float, with np.ndarray being of the same length, and lower being better.
If str, uses registry.resolve_alias to resolve to one of the above. Valid strings are valid registry.craft specs, which include string repr-s of any BaseMetric object, e.g., “MeanSquaredError()”; and keys of registry.ALIAS_DICT referring to metrics.
If None, defaults to MeanAbsolutePercentageError()

strategy{“refit”, “update”, “no-update_params”}, optional, default=”refit”

data ingestion strategy in fitting cv, passed to evaluate internally defines the ingestion mode when the forecaster sees new data when window expands

"refit" = a new copy of the forecaster is fitted to each training window
"update" = forecaster is updated with training window data, in sequence provided
"no-update_params" = fit to first training window, re-used without fit or update

update_behaviourstr, optional, default = “full_refit”

one of {“full_refit”, “inner_only”, “no_update”} behaviour of the forecaster when calling update

"full_refit" = both tuning parameters and inner estimator refit on all data seen
"inner_only" = tuning parameters are not re-tuned, inner estimator is updated
"no_update" = neither tuning parameters nor inner estimator are updated

refitbool, optional (default=True)

Whether to refit the forecaster with the best parameters on the entire data.

True = refit the forecaster with the best parameters on the entire data in fit
False = no refitting takes place. The forecaster cannot be used to predict. This is to be used to tune the hyperparameters, and then use the estimator as a parameter estimator, e.g., via get_fitted_params or PluginParamsForecaster.

verbose: int, optional (default=0)

Verbosity level. The higher, the more messages.

return_n_best_forecastersint, default=1

Number of best forecasters to return.

backendstr, default=”loky”

Backend to use when running the fit.

Backend must be one supported by optuna:

“loky” (default): joblib’s LokyBackend.
“threading”: joblib’s ThreadingBackend.
“multiprocessing”: joblib’s MultiprocessingBackend.

error_score‘raise’ or numeric, default=np.nan

Value to assign to the score if an error occurs in estimator fitting.

n_evalsint, default=100

Number of parameter settings that are sampled. n_iter trades off runtime vs quality of the solution.

samplerOptuna sampler, optional (default=None)

e.g. optuna.samplers.TPESampler(seed=42)

Attributes:

best_index_int
best_score_: float: Score of the best model
best_params_dict: Best parameter values across the parameter grid
best_forecaster_estimator: Fitted estimator with the best parameters
cv_results_dict: Results from grid search cross validation
n_best_forecasters_: list of tuples (“rank”, <forecaster>): The “rank” is in relation to best_forecaster_
n_best_scores_: list of float: The scores of n_best_forecasters_ sorted from best to worst score of forecasters

Examples

>>> from sktime.forecasting.model_selection import (
...     ForecastingOptunaSearchCV,
...     )
>>> from sktime.datasets import load_shampoo_sales
>>> import warnings
>>> warnings.simplefilter(action="ignore", category=FutureWarning)
>>> from sktime.forecasting.base import ForecastingHorizon
>>> from sktime.split import ExpandingWindowSplitter
>>> from sktime.split import temporal_train_test_split
>>> from sklearn.preprocessing import MinMaxScaler, RobustScaler
>>> from sktime.forecasting.compose import TransformedTargetForecaster
>>> from sktime.transformations.adapt import TabularToSeriesAdaptor
>>> from sktime.transformations.detrend import Deseasonalizer, Detrender
>>> from sktime.forecasting.naive import NaiveForecaster
>>> from sktime.forecasting.trend import STLForecaster, TrendForecaster
>>> import optuna
>>> from  optuna.distributions import CategoricalDistribution

>>> y = load_shampoo_sales()
>>> y_train, y_test = temporal_train_test_split(y=y, test_size=6)
>>> fh = ForecastingHorizon(y_test.index, is_relative=False).to_relative(
...         cutoff=y_train.index[-1]
...     )
>>> cv = ExpandingWindowSplitter(fh=fh, initial_window=24, step_length=1)
>>> forecaster = TransformedTargetForecaster(
...     steps=[
...             ("detrender", Detrender()),
...             ("scaler", RobustScaler()),
...             ("minmax2", MinMaxScaler((1, 10))),
...             ("forecaster", NaiveForecaster()),
...         ]
...     )
>>> param_grid = {
...     "scaler__with_scaling": CategoricalDistribution(
...             (True, False)
...         ),
...     "forecaster": CategoricalDistribution(
...             (NaiveForecaster(), TrendForecaster())
...         ),
...     }
>>> gscv = ForecastingOptunaSearchCV(
...         forecaster=forecaster,
...         param_grid=param_grid,
...         cv=cv,
...         n_evals=10,
...     )
>>> gscv.fit(y)
ForecastingOptunaSearchCV(...)
>>> print(f"{gscv.best_params_=}")

Methods

`check_is_fitted`([method_name])	Check if the estimator has been fitted.
`clone`()	Obtain a clone of the object with same hyper-parameters and config.
`clone_tags`(estimator[, tag_names])	Clone tags from another object as dynamic override.
`create_test_instance`([parameter_set])	Construct an instance of the class, using first test parameter set.
`create_test_instances_and_names`([parameter_set])	Create list of all test instances and a list of names for them.
`fit`(y[, X, fh])	Fit forecaster to training data.
`fit_predict`(y[, X, fh, X_pred])	Fit and forecast time series at future horizon.
`get_class_tag`(tag_name[, tag_value_default])	Get class tag value from class, with tag level inheritance from parents.
`get_class_tags`()	Get class tags from class, with tag level inheritance from parent classes.
`get_config`()	Get config flags for self.
`get_fitted_params`([deep])	Get fitted parameters.
`get_param_defaults`()	Get object's parameter defaults.
`get_param_names`([sort])	Get object's parameter names.
`get_params`([deep])	Get a dict of parameters values for this object.
`get_pretrained_params`([deep])	Get pretrained parameters of this estimator.
`get_tag`(tag_name[, tag_value_default, ...])	Get tag value from instance, with tag level inheritance and overrides.
`get_tags`()	Get tags from instance, with tag level inheritance and overrides.
`get_test_params`([parameter_set])	Return testing parameter settings for the estimator.
`is_composite`()	Check if the object is composed of other BaseObjects.
`load_from_path`(serial)	Load object from file location.
`load_from_serial`(serial)	Load object from serialized memory container.
`predict`([fh, X])	Forecast time series at future horizon.
`predict_interval`([fh, X, coverage])	Compute/return prediction interval forecasts.
`predict_proba`([fh, X, marginal])	Compute/return fully probabilistic forecasts.
`predict_quantiles`([fh, X, alpha])	Compute/return quantile forecasts.
`predict_residuals`([y, X])	Return residuals of time series forecasts.
`predict_var`([fh, X, cov])	Compute/return variance forecasts.
`pretrain`(y[, X, fh])	Pre-train forecaster on panel (global) data.
`reset`()	Reset the object to a clean post-init state.
`save`([path, serialization_format])	Save serialized self to bytes-like object or to (.zip) file.
`score`(y[, X, fh])	Scores forecast against ground truth, using MAPE (non-symmetric).
`set_config`(**config_dict)	Set config flags to given values.
`set_params`(**params)	Set the parameters of this object.
`set_random_state`([random_state, deep, ...])	Set random_state pseudo-random seed parameters for self.
`set_tags`(**tag_dict)	Set instance level tag overrides to given values.
`update`(y[, X, update_params])	Update cutoff value and, optionally, fitted parameters.
`update_predict`(y[, cv, X, update_params, ...])	Make predictions and update model iteratively over the test set.
`update_predict_single`([y, fh, X, update_params])	Update model with new data and make forecasts.