TSCOptCV
Tune an sktime classifier via any optimizer in the hyperactive toolbox.
TSCOptCV uses any available tuning engine from hyperactive to tune a classifier by backtesting.
It passes backtesting results as scores to the tuning engine, which identifies the best hyperparameters.
Any available tuning engine from hyperactive can be used, for example:
grid search -
from hyperactive.opt import GridSearchSk as GridSearch, this results in the same algorithm asTSCGridSearchCVhill climbing -
from hyperactive.opt import HillClimbingoptuna parzen-tree search -
from hyperactive.opt.optuna import TPEOptimizer
Configuration of the tuning engine is as per the respective documentation.
Formally, TSCOptCV does the following:
In fit:
wraps the
estimator,scoring, and other parameters into aSktimeClassificationExperimentinstance, which is passed to the optimizeroptimizeras theexperimentargument.Optimal parameters are then obtained from
optimizer.solve, and set asbest_params_andbest_estimator_attributes.If
refit=True,best_estimator_is fitted to the entireyandX.
In predict and predict-like methods, calls the respective method of the best_estimator_ if refit=True.
Schnellstart
from sktime.classification.model_selection import TSCOptCV
estimator = TSCOptCV(estimator, optimizer, cv=None, scoring=None, refit=True, error_score=nan, backend=None, backend_params=None)Parameter(8)
- estimatorsktime classifier, BaseClassifier instance or interface compatible
- The classifier to tune, must implement the sktime classifier interface.
- optimizerhyperactive BaseOptimizer
- The optimizer to be used for hyperparameter search.
- cvint, sklearn cross-validation generator or an iterable, default=3-fold CV
Determines the cross-validation splitting strategy. Possible inputs for cv are:
None = default =
KFold(n_splits=3, shuffle=True)integer, number of folds folds in a
KFoldsplitter,shuffle=TrueAn iterable yielding (train, test) splits as arrays of indices.
For integer/None inputs, if the estimator is a classifier and
yis either binary or multiclass,StratifiedKFoldis used. In all other cases,KFoldis used. These splitters are instantiated withshuffle=Falseso the splits will be the same across calls.- scoringstr, callable, default=None
Strategy to evaluate the performance of the cross-validated model on the test set. Can be:
a single string resolvable to an sklearn scorer
a callable that returns a single value;
None= default =accuracy_score
- refitbool, optional (default=True)
- True = refit the forecaster with the best parameters on the entire data in fit False = no refitting takes place. The forecaster cannot be used to predict. This is to be used to tune the hyperparameters, and then use the estimator as a parameter estimator, e.g., via get_fitted_params or PluginParamsForecaster.
- error_score“raise” or numeric, default=np.nan
- Value to assign to the score if an exception occurs in estimator fitting. If set to “raise”, the exception is raised. If a numeric value is given, FitFailedWarning is raised.
- backendstring, by default “None”.
Parallelization backend to use for runs. Runs parallel evaluate if specified and
strategy="refit".“None”: executes loop sequentially, simple list comprehension
“loky”, “multiprocessing” and “threading”: uses
joblib.Parallelloops“joblib”: custom and 3rd party
joblibbackends, e.g.,spark“dask”: uses
dask, requiresdaskpackage in environment“dask_lazy”: same as “dask”, but changes the return to (lazy)
dask.dataframe.DataFrame.“ray”: uses
ray, requiresraypackage in environment
Recommendation: Use “dask” or “loky” for parallel evaluate. “threading” is unlikely to see speed ups due to the GIL and the serialization backend (
cloudpickle) for “dask” and “loky” is generally more robust than the standardpicklelibrary used in “multiprocessing”.- backend_paramsdict, optional
additional parameters passed to the backend as config. Directly passed to
utils.parallel.parallelize. Valid keys depend on the value ofbackend:“None”: no additional parameters,
backend_paramsis ignored“loky”, “multiprocessing” and “threading”: default
joblibbackends any valid keys forjoblib.Parallelcan be passed here, e.g.,n_jobs, with the exception ofbackendwhich is directly controlled bybackend. Ifn_jobsis not passed, it will default to-1, other parameters will default tojoblibdefaults.“joblib”: custom and 3rd party
joblibbackends, e.g.,spark. any valid keys forjoblib.Parallelcan be passed here, e.g.,n_jobs,backendmust be passed as a key ofbackend_paramsin this case. Ifn_jobsis not passed, it will default to-1, other parameters will default tojoblibdefaults.“dask”: any valid keys for
dask.computecan be passed, e.g.,scheduler“ray”: The following keys can be passed:
“ray_remote_args”: dictionary of valid keys for
ray.init- “shutdown_ray”: bool, default=True; False prevents
rayfrom shutting down after parallelization.
- “shutdown_ray”: bool, default=True; False prevents