HFTransformersForecaster

Quickstart

from sktime.forecasting.hf_transformers import HFTransformersForecaster

estimator = HFTransformersForecaster(model_path: str=None, fit_strategy='minimal', validation_split=0.2, config=None, training_args=None, compute_metrics=None, deterministic=False, callbacks=None, peft_config=None, device=None)

Parameters(10)

model_pathstr or PreTrainedModel

Path to the huggingface model to use for forecasting. Currently, Informer, Autoformer, and TimeSeriesTransformer are supported. This can be one of the following: - A string specifying the Hugging Face model name or path

(e.g., “huggingface/autoformer-tourism-monthly”).

• An instance of a PreTrainedModel, allowing manual initialization and configuration.

fit_strategystr, default=”minimal”

Strategy to use for fitting (fine-tuning) the model. This can be one of the following:

• “minimal”: Fine-tunes only a small subset of the model parameters, allowing for quick adaptation with limited computational resources.

• “full”: Fine-tunes all model parameters, which may result in better performance but requires more computational power and time.

• “peft”: Applies Parameter-Efficient Fine-Tuning (PEFT) techniques to adapt the model with fewer trainable parameters, saving computational resources.

  Note: If the ‘peft’ package is not available, a ModuleNotFoundError will be raised, indicating that the ‘peft’ package is required. Please install it using pip install peft to use this fit strategy.

validation_splitfloat, default=0.2

Fraction of the data to use for validation

configdict, default={}

Configuration to use for the model. Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs and architecture. Refer to the individual model config for particular model-specific config params.

PreTrainedConfig is the base class for all configuration classes. It handles a few parameters common to all models’ configurations as well as methods for loading/downloading/saving configurations. A configuration file can be loaded and saved to disk. Loading the configuration file and using this file to initialize a model does not load the model weights. It only affects the model’s configuration.

Keys supported by PreTrainedConfig include:

name_or_pathstr, optional, default=””

Store the string that was passed to PreTrainedModel.from_pretrained() as pretrained_model_name_or_path if the configuration was created with such a method.

training_argsdict, default={}

Training arguments to use for the model. See transformers.TrainingArguments for details [1]. Note that the output_dir argument is required.

compute_metricslist, default=None

List of metrics to compute during training. See transformers.Trainer for details.

deterministicbool, default=False

Whether the predictions should be deterministic or not.

callbackslist, default=[]

List of callbacks to use during training. See transformers.Trainer

peft_configpeft.PeftConfig, default=None

Configuration for Parameter-Efficient Fine-Tuning. When fit_strategy is set to “peft”, this will be used to set up PEFT parameters for the model. See the peft documentation for details [2].

devicestr, optional (default=None)

Device on which to load the model, passed to the transformers device_map, for example "cpu", "cuda", or "auto". "auto" selects an available accelerator. If None, the transformers default placement is used. Ignored when model_path is an already initialized model object, which keeps its own device. Setting device requires the accelerate package, which is not part of the base transformers install. Install it with pip install accelerate or pip install "transformers[torch]".

Examples

Using a Pretrained Model from Hugging Face
>>> from sktime.forecasting.hf_transformers import HFTransformersForecaster
>>> from sktime.datasets import load_airline
>>> y = load_airline ()
>>> forecaster = HFTransformersForecaster (
... model_path = "huggingface/autoformer-tourism-monthly",
... training_args = {
... "num_train_epochs": 20,
... "output_dir": "test_output",
... "per_device_train_batch_size": 32,
... },
... config = {
... "lags_sequence": [1, 2, 3 ],
... "context_length": 2,
... "prediction_length": 4,
... "use_cpu": True,
... "label_length": 2,
... },
... )
>>> forecaster. fit (y)
>>> fh = [1, 2, 3 ]
>>> y_pred = forecaster. predict (fh) Using PEFT for Fine-Tuning
>>> from sktime.forecasting.hf_transformers import HFTransformersForecaster
>>> from sktime.datasets import load_airline
>>> from peft import LoraConfig
>>> y = load_airline ()
>>> forecaster = HFTransformersForecaster (
... model_path = "huggingface/autoformer-tourism-monthly",
... fit_strategy = "peft",
... training_args = {
... "num_train_epochs": 20,
... "output_dir": "test_output",
... "per_device_train_batch_size": 32,
... },
... config = {
... "lags_sequence": [1, 2, 3 ],
... "context_length": 2,
... "prediction_length": 4,
... "use_cpu": True,
... "label_length": 2,
... },
... peft_config = LoraConfig (
... r = 8,
... lora_alpha = 32,
... target_modules = ["q_proj", "v_proj" ],
... lora_dropout = 0.01,
... )
... )
>>> forecaster. fit (y)
>>> fh = [1, 2, 3 ]
>>> y_pred = forecaster. predict (fh) Using an Initialized Model
>>> from sktime.datasets import load_airline
>>> from transformers import AutoformerConfig, AutoformerForPrediction
>>> from sktime.forecasting.hf_transformers import HFTransformersForecaster
>>> y = load_airline ()
>>> # Define model configuration
>>> config = AutoformerConfig (
... num_dynamic_real_features = 0,
... num_static_real_features = 0,
... num_static_categorical_features = 0,
... num_time_features = 0,
... context_length = 32,
... prediction_length = 8,
... lags_sequence = [1, 2, 3 ],
... )
>>> # Initialize the model
>>> model = AutoformerForPrediction (config)
>>> # Initialize the forecaster with the model
>>> forecaster = HFTransformersForecaster (
... model_path = model,
... fit_strategy = "minimal",
... training_args = {
... "num_train_epochs": 10,
... "output_dir": "output",
... "per_device_train_batch_size": 4
... },
... )
>>> forecaster. fit (y)
>>> fh = [1, 2, 3 ]
>>> y_pred = forecaster. predict (fh)

References