PatchTSTForecaster

PatchTSTForecaster

class PatchTSTForecaster(model_path=None, fit_strategy='full', validation_split=0.2, config=None, training_args=None, compute_metrics=None, callbacks=None, device=None)[source]

Interface for the PatchTST forecaster.

This forecaster interfaces the Huggingface library’s PatchTST model for time series forecasting. The model was originally designed by Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong and Jayant Kalagnanam. It utilizes a transformer model architecture and splits the time series data into patches that are then processed by the model. Visit [1] for more information on the model architecture and its authors. For tips on how to construct your own PatchTST config, see [2].

The PatchTST forecaster can be used in three ways:

1) Full training via a new model initialized from a config or a loaded model with pre-trained weights 2) Minimal fine-tuning with a pre-trained model and an altered config 3) Zero-shot forecasting with a pre-trained model

For more details, please visit the fit_strategy parameter

Parameters:

model_pathstr or PatchTSTModel, optional

Path to the Huggingface model to use for global forecasting. If model_path is passed, the remaining model config parameters will be ignored except for specific training or dataset parameters. This has 3 options:

• model id to an online pretrained PatchTST Model hosted on HuggingFace

• A path or url to a saved configuration JSON file

• A path to a directory containing a configuration file saved

using the ~PretrainedConfig.save_pretrained method or the ~PreTrainedModel.save_pretrained method

fit_strategystr, values = [“full”,”minimal”,”zero-shot”], default = “full”

String to set the fit_strategy of the model.

• This strategy is used to create and train a new model from scratch

(pre-pretraining) or to update all of the weights in a pre-trained model (also known as full fine-tuning). If fit_strategy is set to full, requires either the model_path parameter or the config` parameter to be passed in, but not both. If only config is passed, it will initialize an new model with untrained weights with the specified config arguments. If only model_path is passed, it will fine-tune ALL of the pre-trained weights of the model.

• If fit_strategy is set to “minimal” requires both the model_path

and config parameter. We will use the model_path and the specified config to compare the weight shapes of the passed pre-trained model to those in the config. If there are weight size mismatches, the model will reinitialize new weights to match the weight shapes inside the config. The y argument will then be fit to fine-tune the model. In the case where there are no newly initialized weights (i.e the config weight shapes match the pretrained model weight shapes), it will behave the same as the “full” strategy where only the model_path is passed in.

• If fit_strategy is set to “zero-shot”, requires only the model_path

parameter. It will load the model via the fit function with the argument model_path and ignore any passed y.

validation_splitfloat, optional, default = 0.2

Fraction of the data to use for validation.

devicestr, optional (default=None)

Device on which to load the model, passed to the transformers device_map, for example "cpu", "cuda", or "auto". "auto" selects an available accelerator. If None, the transformers default placement is used. Ignored when model_path is an already initialized PatchTSTModel, which keeps its own device.

configdict, optional, default = {}

A config dict specifying parameters to initialize a full PatchTST model. Missing parameters in the config will be automatically replaced by their default values. Valid keys include:

num_input_channelsint, optional, default=1

The number of input channels.

context_lengthint, optional, default=32

The context length of the input sequence.

distribution_outputstr, optional, default=”student_t”

The distribution emission head for the model when loss is "nll". Could be either "student_t", "normal" or "negative_binomial".

lossstr, optional, default=”mse”

The loss function for the model corresponding to the distribution_output head. For parametric distributions it is the negative log likelihood ("nll") and for point estimates it is the mean squared error "mse".

patch_lengthint, optional, default=1

Define the patch length of the patchification process.

patch_strideint, optional, default=1

Define the stride of the patchification process.

num_hidden_layersint, optional, default=3

Number of hidden layers in the Transformer decoder.

d_modelint, optional, default=128

Size of the encoder layers and the pooler layer.

num_attention_headsint, optional, default=4

Number of attention heads for each attention layer in the Transformer encoder.

share_embeddingbool, optional, default=True

Sharing the input embedding across all channels.

channel_attentionbool, optional, default=False

Activate channel attention block in the Transformer to allow channels to attend each other.

ffn_dimint, optional, default=512

Dimension of the “intermediate” (often named feed-forward) layer in the Transformer encoder.

norm_typestr, optional, default=”batchnorm”

Normalization at each Transformer layer. Can be "batchnorm" or "layernorm".

norm_epsfloat, optional, default=1e-05

A value added to the denominator for numerical stability of normalization.

attention_dropoutfloat or int, optional, default=0.0

The dropout ratio for the attention probabilities.

positional_dropoutfloat, optional, default=0.0

The dropout probability in the positional embedding layer.

path_dropoutfloat, optional, default=0.0

The dropout path in the residual block.

ff_dropoutfloat, optional, default=0.0

The dropout probability used between the two layers of the feed-forward networks.

biasbool, optional, default=True

Whether to add bias in the feed-forward networks.

activation_functionstr, optional, default=”gelu”

The non-linear activation function (string) in the Transformer. "gelu" and "relu" are supported.

pre_normbool, optional, default=True

Normalization is applied before self-attention if pre_norm is set to True. Otherwise, normalization is applied after residual block.

positional_encoding_typestr, optional, default=”sincos”

Positional encodings. Options "random" and "sincos" are supported.

use_cls_tokenbool, optional, default=False

Whether cls token is used.

init_stdfloat, optional, default=0.02

The standard deviation of the truncated normal weight initialization distribution.

share_projectionbool, optional, default=True

Sharing the projection layer across different channels in the forecast head.

scalingstr, bool, or None, optional, default=”std”

Whether to scale the input targets via "mean" scaler, "std" scaler or no scaler if None. If True, the scaler is set to "mean".

do_mask_inputbool, optional

Apply masking during the pretraining.

mask_typestr, optional, default=”random”

Masking type. Only "random" and "forecast" are currently supported.

random_mask_ratiofloat, optional, default=0.5

Masking ratio applied to mask the input data during random pretraining.

num_forecast_mask_patchesint or list, optional, default=[2]

Number of patches to be masked at the end of each batch sample. If it is an integer, all the samples in the batch will have the same number of masked patches. If it is a list, samples in the batch will be randomly masked by numbers defined in the list. This argument is only used for forecast pretraining.

channel_consistent_maskingbool, optional, default=False

If channel consistent masking is True, all the channels will have the same masking pattern.

unmasked_channel_indiceslist, optional

Indices of channels that are not masked during pretraining. Values in the list are numbers between 1 and num_input_channels.

mask_valueint, optional, default=0

Values in the masked patches will be filled by mask_value.

pooling_typestr, optional, default=”mean”

Pooling of the embedding. "mean", "max" and None are supported.

head_dropoutfloat, optional, default=0.0

The dropout probability for head.

prediction_lengthint, optional, default=24

The prediction horizon that the model will output.

num_targetsint, optional, default=1

Number of targets for regression and classification tasks. For classification, it is the number of classes.

output_rangelist, optional

Output range for regression task. The range of output values can be set to enforce the model to produce values within a range.

num_parallel_samplesint, optional, default=100

The number of samples generated in parallel for probabilistic prediction.

Note: if prediction_length is passed as larger than the passed fh in the fit function, the prediction_length will be used to train the model. If prediction_length is passed as smaller than the passed fh in the fit function, the passed fh will be used to train the model.

training_argsdict, optional, default = None

Training arguments to use for the model. If this is passed, the remaining applicable training arguments will be ignored. See [3]_ for details.

compute_metricslist or function, default = None

List of metrics or function to use during training

callbacks: list or function, default = None

List of callbacks or callback function to use during training

Attributes:

cutoff

Cut-off = “present time” state of forecaster.

fh

Forecasting horizon that was passed.

is_fitted

Whether fit has been called.

state

State of the estimator.

References

[1] A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, Jayant Kalagnanam Paper: https://arxiv.org/abs/2211.14730

[2] HuggingFace PatchTST Page:

https://huggingface.co/docs/transformers/en/model_doc/patchtst

[3] https://huggingface.co/docs/transformers/v5.14.0/en/main_classes/trainer#transformers.TrainingArguments

Examples

>>> #Example with a new model initialized from config only
>>> from sktime.forecasting.patch_tst import PatchTSTForecaster
>>> from sktime.datasets import load_airline
>>> y = load_airline()
>>> forecaster = PatchTSTForecaster(
... config = {
...     "patch_length": 1,
...      "context_length": 2,
...      "patch_stride": 1,
...      "d_model": 64,
...      "num_attention_heads": 2,
...      "ffn_dim": 32,
...      "head_dropout": 0.3,
...    },
...    training_args = {
...         "output_dir":"test_output",
...         "overwrite_output_dir":True,
...         "learning_rate":1e-4,
...         "num_train_epochs":1,
...         "per_device_train_batch_size":16,
...    }
... ) #initialize an full model
>>> forecaster.fit(y, fh=[1, 2, 3])
>>> y_pred = forecaster.predict()

>>> #Example full fine-tuning with a pre-trained model
>>> from sktime.forecasting.patch_tst import PatchTSTForecaster
>>> import pandas as pd
>>> dataset_path = pd.read_csv(
...     "https://raw.githubusercontent.com/zhouhaoyi/ETDataset/main/ETT-small/ETTh1.csv"
...     ).drop(columns = ["date"]
... )
>>> from sklearn.preprocessing import StandardScaler
>>> scaler = StandardScaler()
>>> scaler.set_output(transform="pandas")
>>> scaler = scaler.fit(dataset_path.values)
>>> df = scaler.transform(dataset_path)
>>> df.columns = dataset_path.columns
>>> forecaster = PatchTSTForecaster(
...     model_path="namctin/patchtst_etth1_forecast",
...     fit_strategy = "full",
...     training_args = {
...         "output_dir":"test_output",
...         "overwrite_output_dir":True,
...         "learning_rate":1e-4,
...         "num_train_epochs":1,
...         "per_device_train_batch_size":16,
...     }
... )
>>> forecaster.fit(y = df, fh = list(range(1,4)))
>>> y_pred = forecaster.predict()

>>> #Example of minimal fine-tuning with a pre-trained model and an altered config
>>> from sktime.forecasting.patch_tst import PatchTSTForecaster
>>> import pandas as pd
>>> dataset_path = pd.read_csv(
...     "https://raw.githubusercontent.com/zhouhaoyi/ETDataset/main/ETT-small/ETTh1.csv"
...     ).drop(columns = ["date"]
... )
>>> from sklearn.preprocessing import StandardScaler
>>> scaler = StandardScaler()
>>> scaler.set_output(transform="pandas")
>>> scaler = scaler.fit(dataset_path.values)
>>> df = scaler.transform(dataset_path)
>>> df.columns = dataset_path.columns
>>> forecaster = PatchTSTForecaster(
...     model_path="namctin/patchtst_etth1_forecast",
...     config = {
...         "patch_length": 8,
...         "context_length": 512,
...         "patch_stride": 8,
...         "d_model": 128,
...         "num_attention_heads": 2,
...         "ffn_dim": 512,
...         "head_dropout": 0.3,
...         "prediction_length": 64
...     },
...     fit_strategy = "minimal",
...     training_args = {
...         "output_dir":"test_output",
...         "overwrite_output_dir":True,
...         "learning_rate":1e-4,
...         "num_train_epochs":1,
...         "per_device_train_batch_size":16,
...     }
... )
>>> forecaster.fit(y = df, fh = list(range(1,63)))
>>> y_pred = forecaster.predict()

>>> #Example with a pre-trained model to do zero-shot forecasting
>>> from sktime.forecasting.patch_tst import PatchTSTForecaster
>>> import pandas as pd
>>> dataset_path = pd.read_csv(
...     "https://raw.githubusercontent.com/zhouhaoyi/ETDataset/main/ETT-small/ETTh1.csv"
...     ).drop(columns = ["date"]
... )
>>> from sklearn.preprocessing import StandardScaler
>>> scaler = StandardScaler()
>>> scaler.set_output(transform="pandas")
>>> scaler = scaler.fit(dataset_path.values)
>>> df = scaler.transform(dataset_path)
>>> df.columns = dataset_path.columns
>>> forecaster = PatchTSTForecaster(
...     model_path="namctin/patchtst_etth1_forecast",
...     fit_strategy = "zero-shot",
...     training_args = {
...         "output_dir":"test_output",
...         "overwrite_output_dir":True,
...         "learning_rate":1e-4,
...         "num_train_epochs":1,
...         "per_device_train_batch_size":16,
...     }
... )
>>> forecaster.fit(y = df, fh = [1,2,3,4,5])
>>> y_pred = forecaster.predict()

Methods

check_is_fitted([method_name])	Check if the estimator has been fitted.
clone()	Obtain a clone of the object with same hyper-parameters and config.
clone_tags(estimator[, tag_names])	Clone tags from another object as dynamic override.
create_test_instance([parameter_set])	Construct an instance of the class, using first test parameter set.
create_test_instances_and_names([parameter_set])	Create list of all test instances and a list of names for them.
fit(y[, X, fh])	Fit forecaster to training data.
fit_predict(y[, X, fh, X_pred])	Fit and forecast time series at future horizon.
get_class_tag(tag_name[, tag_value_default])	Get class tag value from class, with tag level inheritance from parents.
get_class_tags()	Get class tags from class, with tag level inheritance from parent classes.
get_config()	Get config flags for self.
get_fitted_params([deep])	Get fitted parameters.
get_param_defaults()	Get object's parameter defaults.
get_param_names([sort])	Get object's parameter names.
get_params([deep])	Get a dict of parameters values for this object.
get_pretrained_params([deep])	Get pretrained parameters of this estimator.
get_tag(tag_name[, tag_value_default, ...])	Get tag value from instance, with tag level inheritance and overrides.
get_tags()	Get tags from instance, with tag level inheritance and overrides.
get_test_params([parameter_set])	Return testing parameter settings for the estimator.
is_composite()	Check if the object is composed of other BaseObjects.
load_from_path(serial)	Load object from file location.
load_from_serial(serial)	Load object from serialized memory container.
predict([fh, X])	Forecast time series at future horizon.
predict_interval([fh, X, coverage])	Compute/return prediction interval forecasts.
predict_proba([fh, X, marginal])	Compute/return fully probabilistic forecasts.
predict_quantiles([fh, X, alpha])	Compute/return quantile forecasts.
predict_residuals([y, X])	Return residuals of time series forecasts.
predict_var([fh, X, cov])	Compute/return variance forecasts.
pretrain(y[, X, fh])	Pre-train forecaster on panel (global) data.
reset()	Reset the object to a clean post-init state.
save([path, serialization_format])	Save serialized self to bytes-like object or to (.zip) file.
score(y[, X, fh])	Scores forecast against ground truth, using MAPE (non-symmetric).
set_config(**config_dict)	Set config flags to given values.
set_params(**params)	Set the parameters of this object.
set_random_state([random_state, deep, ...])	Set random_state pseudo-random seed parameters for self.
set_tags(**tag_dict)	Set instance level tag overrides to given values.
update(y[, X, update_params])	Update cutoff value and, optionally, fitted parameters.
update_predict(y[, cv, X, update_params, ...])	Make predictions and update model iteratively over the test set.
update_predict_single([y, fh, X, update_params])	Update model with new data and make forecasts.