ForecastingRandomizedSearchCV

ForecastingRandomizedSearchCV

class ForecastingRandomizedSearchCV(forecaster, cv, param_distributions, n_iter=10, scoring=None, strategy='refit', update_behaviour='full_refit', refit=True, tune_by_instance=False, tune_by_variable=False, verbose=0, return_n_best_forecasters=1, error_score=nan, backend='loky', backend_params=None, random_state=None, n_jobs='deprecated')[source]

Perform randomized-search cross-validation to find optimal model parameters.

The forecaster is fit on the initial window and then temporal cross-validation is used to find the optimal parameter

Randomized cross-validation is performed based on a cross-validation iterator encoding the cross-validation scheme, the parameter distributions to search over, and (optionally) the evaluation metric for comparing model performance. As in scikit-learn, tuning works through the common hyper-parameter interface which allows to repeatedly fit and evaluate the same forecaster with different hyper-parameters.

Parameters:

forecastersktime forecaster, BaseForecaster instance or interface compatible

The forecaster to tune, must implement the sktime forecaster interface. sklearn regressors can be used, but must first be converted to forecasters via one of the reduction compositors, e.g., via make_reduction

cvsktime time series splitter

Re-sampling strategy for cross-validation, must be an instance of a sktime time series splitter, e.g. SlidingWindowSplitter()

param_distributionsdict or list of dicts

Dictionary with parameters names (str) as keys and distributions or lists of parameters to try. Distributions must provide a rvs method for sampling (such as those from scipy.stats.distributions).

• If a list is given, it is sampled uniformly.

• If a list of dicts is given, first a dict is sampled uniformly, and then a parameter is sampled using that dict as above.

n_iterint, default=10

Number of parameter settings that are sampled. n_iter trades off runtime vs quality of the solution.

scoringsktime metric (BaseMetric), str, or callable, optional (default=None)

scoring metric to use in tuning the forecaster

• sktime metric objects (BaseMetric) descendants can be searched with the registry.all_estimators search utility, for instance via all_estimators("metric", as_dataframe=True)

• If callable, must have signature (y_true: 1D np.ndarray, y_pred: 1D np.ndarray) -> float, with np.ndarray being of the same length, and lower being better.

• If str, uses registry.resolve_alias to resolve to one of the above. Valid strings are valid registry.craft specs, which include string repr-s of any BaseMetric object, e.g., “MeanSquaredError()”; and keys of registry.ALIAS_DICT referring to metrics.

• If None, defaults to MeanAbsolutePercentageError()

strategy{“refit”, “update”, “no-update_params”}, optional, default=”refit”

data ingestion strategy in fitting cv, passed to evaluate internally defines the ingestion mode when the forecaster sees new data when window expands

• "refit" = a new copy of the forecaster is fitted to each training window

• "update" = forecaster is updated with training window data, in sequence provided

• "no-update_params" = fit to first training window, re-used without fit or update

update_behaviourstr, optional, default = “full_refit”

one of {“full_refit”, “inner_only”, “no_update”} behaviour of the forecaster when calling update

• "full_refit" = both tuning parameters and inner estimator refit on all data seen

• "inner_only" = tuning parameters are not re-tuned, inner estimator is updated

• "no_update" = neither tuning parameters nor inner estimator are updated

refitbool, optional (default=True)

Whether to refit the forecaster with the best parameters on the entire data.

• True = refit the forecaster with the best parameters on the entire data in fit

• False = no refitting takes place. The forecaster cannot be used to predict. This is to be used to tune the hyperparameters, and then use the estimator as a parameter estimator, e.g., via get_fitted_params or PluginParamsForecaster.

tune_by_instancebool, optional (default=False)

Whether to tune parameter by each time series instance separately, in case of Panel or Hierarchical data passed to the tuning estimator. Only applies if time series passed are Panel or Hierarchical.

• If True, clones of the forecaster will be fit to each instance separately, and are available in fields of the forecasters_ attribute. Has the same effect as applying ForecastByLevel wrapper to self.

• If False, the same best parameter is selected for all instances.

tune_by_variablebool, optional (default=False)

Whether to tune parameter by each time series variable separately, in case of multivariate data passed to the tuning estimator. Only applies if time series passed are strictly multivariate.

• If True, clones of the forecaster will be fit to each variable separately, and are available in fields of the forecasters_ attribute. Has the same effect as applying ColumnEnsembleForecaster wrapper to self.

• If False, the same best parameter is selected for all variables.

verbose: int, optional (default=0)

Verbosity level. The higher, the more messages.

return_n_best_forecasters: int, default=1

In case the n best forecaster should be returned, this value can be set and the n best forecasters will be assigned to n_best_forecasters_. Set return_n_best_forecasters to -1 to return all forecasters.

error_score“raise” or numeric, default=np.nan

Value to assign to the score if an exception occurs in estimator fitting. If set to “raise”, the exception is raised. If a numeric value is given, FitFailedWarning is raised.

backend{“dask”, “loky”, “multiprocessing”, “threading”}, by default “loky”.

Runs parallel evaluate if specified and strategy is set as “refit”.

• “None”: executes loop sequentally, simple list comprehension

• “loky”, “multiprocessing” and “threading”: uses joblib.Parallel loops

• “joblib”: custom and 3rd party joblib backends, e.g., spark

• “dask”: uses dask, requires dask package in environment

• “ray”: uses ray, requires ray package in environment

Recommendation: Use “dask” or “loky” for parallel evaluate. “threading” is unlikely to see speed ups due to the GIL and the serialization backend (cloudpickle) for “dask” and “loky” is generally more robust than the standard pickle library used in “multiprocessing”.

backend_paramsdict, optional

additional parameters passed to the backend as config. Directly passed to utils.parallel.parallelize. Valid keys depend on the value of backend:

• “None”: no additional parameters, backend_params is ignored

• “loky”, “multiprocessing” and “threading”: default joblib backends any valid keys for joblib.Parallel can be passed here, e.g., n_jobs, with the exception of backend which is directly controlled by backend. If n_jobs is not passed, it will default to -1, other parameters will default to joblib defaults.

• “joblib”: custom and 3rd party joblib backends, e.g., spark. any valid keys for joblib.Parallel can be passed here, e.g., n_jobs, backend must be passed as a key of backend_params in this case. If n_jobs is not passed, it will default to -1, other parameters will default to joblib defaults.

• “dask”: any valid keys for dask.compute can be passed, e.g., scheduler

• “ray”: The following keys can be passed:

  • “ray_remote_args”: dictionary of valid keys for ray.init

  • “shutdown_ray”: bool, default=True; False prevents ray from shutting

    down after parallelization.

  • “logger_name”: str, default=”ray”; name of the logger to use.

  • “mute_warnings”: bool, default=False; if True, suppresses warnings

random_stateint, RandomState instance or None, default=None

Pseudo random number generator state used for random uniform sampling from lists of possible values instead of scipy.stats distributions. Pass an int for reproducible output across multiple function calls.

Attributes:

best_index_int

best_score_: float

Score of the best model

best_params_dict

Best parameter values across the parameter grid

best_forecaster_estimator

Fitted estimator with the best parameters

cv_results_dict

Results from grid search cross validation

n_best_forecasters_: list of tuples (“rank”, <forecaster>)

The “rank” is in relation to best_forecaster_

n_best_scores_: list of float

The scores of n_best_forecasters_ sorted from best to worst score of forecasters

forecasters_pd.DataFramee

DataFrame with all fitted forecasters and their parameters. Only present if tune_by_instance=True or tune_by_variable=True, and at least one of the two is applicable. In this case, the other attributes are not present in self, only in the fields of forecasters_.

Examples

>>> from sktime.datasets import load_shampoo_sales
>>> from sktime.forecasting.model_selection import ForecastingRandomizedSearchCV
>>> from sktime.split import ExpandingWindowSplitter
>>> from sktime.forecasting.naive import NaiveForecaster
>>> y = load_shampoo_sales()
>>> fh = [1, 2, 3]
>>> cv = ExpandingWindowSplitter(fh=fh)
>>> forecaster = NaiveForecaster()
>>> param_distributions = {"strategy": ["last", "mean", "drift"]}
>>> rscv = ForecastingRandomizedSearchCV(
...     forecaster=forecaster,
...     param_distributions=param_distributions,
...     cv=cv,
...     n_iter=3,
...     random_state=42)
>>> rscv.fit(y)
ForecastingRandomizedSearchCV(...)
>>> y_pred = rscv.predict(fh)

Advanced randomized search with a scipy.stats distribution for a continuous hyperparameter, on a pipeline forecaster:

>>> from scipy.stats import randint
>>> from sktime.datasets import load_shampoo_sales
>>> from sktime.forecasting.compose import TransformedTargetForecaster
>>> from sktime.forecasting.model_selection import ForecastingRandomizedSearchCV
>>> from sktime.forecasting.naive import NaiveForecaster
>>> from sktime.split import ExpandingWindowSplitter
>>> from sktime.transformations.detrend import Detrender
>>> y = load_shampoo_sales()
>>> pipe = TransformedTargetForecaster(steps=[
...     ("detrender", Detrender()),
...     ("forecaster", NaiveForecaster(strategy="mean"))])
>>> param_distributions = {
...     "forecaster__window_length": randint(2, 12),
...     "forecaster__strategy": ["mean", "last", "drift"],
... }
>>> cv = ExpandingWindowSplitter(
...     initial_window=18, step_length=6, fh=[1, 2, 3])
>>> rscv = ForecastingRandomizedSearchCV(
...     forecaster=pipe,
...     param_distributions=param_distributions,
...     cv=cv,
...     n_iter=5,
...     random_state=42)
>>> rscv.fit(y)
ForecastingRandomizedSearchCV(...)
>>> y_pred = rscv.predict(fh=[1, 2, 3])

Methods

check_is_fitted([method_name])	Check if the estimator has been fitted.
clone()	Obtain a clone of the object with same hyper-parameters and config.
clone_tags(estimator[, tag_names])	Clone tags from another object as dynamic override.
create_test_instance([parameter_set])	Construct an instance of the class, using first test parameter set.
create_test_instances_and_names([parameter_set])	Create list of all test instances and a list of names for them.
fit(y[, X, fh])	Fit forecaster to training data.
fit_predict(y[, X, fh, X_pred])	Fit and forecast time series at future horizon.
get_class_tag(tag_name[, tag_value_default])	Get class tag value from class, with tag level inheritance from parents.
get_class_tags()	Get class tags from class, with tag level inheritance from parent classes.
get_config()	Get config flags for self.
get_fitted_params([deep])	Get fitted parameters.
get_param_defaults()	Get object's parameter defaults.
get_param_names([sort])	Get object's parameter names.
get_params([deep])	Get a dict of parameters values for this object.
get_pretrained_params([deep])	Get pretrained parameters of this estimator.
get_tag(tag_name[, tag_value_default, ...])	Get tag value from instance, with tag level inheritance and overrides.
get_tags()	Get tags from instance, with tag level inheritance and overrides.
get_test_params([parameter_set])	Return testing parameter settings for the estimator.
is_composite()	Check if the object is composed of other BaseObjects.
load_from_path(serial)	Load object from file location.
load_from_serial(serial)	Load object from serialized memory container.
predict([fh, X])	Forecast time series at future horizon.
predict_interval([fh, X, coverage])	Compute/return prediction interval forecasts.
predict_proba([fh, X, marginal])	Compute/return fully probabilistic forecasts.
predict_quantiles([fh, X, alpha])	Compute/return quantile forecasts.
predict_residuals([y, X])	Return residuals of time series forecasts.
predict_var([fh, X, cov])	Compute/return variance forecasts.
pretrain(y[, X, fh])	Pre-train forecaster on panel (global) data.
reset()	Reset the object to a clean post-init state.
save([path, serialization_format])	Save serialized self to bytes-like object or to (.zip) file.
score(y[, X, fh])	Scores forecast against ground truth, using MAPE (non-symmetric).
set_config(**config_dict)	Set config flags to given values.
set_params(**params)	Set the parameters of this object.
set_random_state([random_state, deep, ...])	Set random_state pseudo-random seed parameters for self.
set_tags(**tag_dict)	Set instance level tag overrides to given values.
update(y[, X, update_params])	Update cutoff value and, optionally, fitted parameters.
update_predict(y[, cv, X, update_params, ...])	Make predictions and update model iteratively over the test set.
update_predict_single([y, fh, X, update_params])	Update model with new data and make forecasts.