Ctrl+K

binder

Early time series classification with sktime#

Early time series classification (eTSC) is the problem of classifying a time series after as few measurements as possible with the highest possible accuracy. The most critical issue of any eTSC method is to decide when enough data of a time series has been seen to take a decision: Waiting for more data points usually makes the classification problem easier but delays the time in which a classification is made; in contrast, earlier classification has to cope with less input data, often leading to inferior accuracy.

This notebook gives a quick guide to get you started with running eTSC algorithms in sktime.

References:#

[1] Schäfer, P., & Leser, U. (2020). TEASER: early and accurate time series classification. Data mining and knowledge discovery, 34(5), 1336-1362

Data sets and problem types#

The UCR/UEA time series classification archive contains a large number of example TSC problems that have been used thousands of times in the literature to assess TSC algorithms. Read the data loading documentation and notebooks for details on the sktime data formats and loading data for sktime.

[ ]:

# Imports used in this notebook
import numpy as np

from sktime.classification.early_classification._teaser import TEASER
from sktime.classification.interval_based import TimeSeriesForestClassifier
from sktime.datasets import load_arrow_head

[ ]:

# Load default train/test splits from sktime/datasets/data
arrow_train_X, arrow_train_y = load_arrow_head(split="train", return_type="numpy3d")
arrow_test_X, arrow_test_y = load_arrow_head(split="test", return_type="numpy3d")

arrow_test_X.shape

Building the TEASER classifier#

TEASER [1] is a two-tier model using a base classifier to make predictions and a decision making estimator to decide whether these predictions are safe. As a first tier, TEASER requires a TSC algorithm, such as WEASEL, which produces class probabilities as output. As a second tier an anomaly detector is required, such as a one-class SVM.

[ ]:

teaser = TEASER(
    random_state=0,
    classification_points=[25, 50, 75, 100, 125, 150, 175, 200, 251],
    estimator=TimeSeriesForestClassifier(n_estimators=10, random_state=0),
)
teaser.fit(arrow_train_X, arrow_train_y)

Determine the accuracy and earliness on the test data#

Commonly accuracy is used to determine the correctness of the predictions, while earliness is used to determine how much of the series is required on average to obtain said accuracy. I.e. for the below values, using just 43% of the full test data, we were able to get an accuracy of 69%.

[ ]:

hm, acc, earl = teaser.score(arrow_test_X, arrow_test_y)
print("Earliness on Test Data %2.2f" % earl)
print("Accuracy on Test Data %2.2f" % acc)
print("Harmonic Mean on Test Data %2.2f" % hm)

Determine the accuracy and earliness on the train data#

[ ]:

print("Earliness on Train Data %2.2f" % teaser._train_earliness)
print("Accuracy on Train Data %2.2f" % teaser._train_accuracy)

Comparison to Classification on full Test Data#

With the full test data, we would obtain 68% accuracy with the same classifier.

[ ]:

accuracy = (
    TimeSeriesForestClassifier(n_estimators=10, random_state=0)
    .fit(arrow_train_X, arrow_train_y)
    .score(arrow_test_X, arrow_test_y)
)
print("Accuracy on the full Test Data %2.2f" % accuracy)

Classifying with incomplete time series#

The main draw of eTSC is the capabilility to make classifications with incomplete time series. sktime eTSC algorithms accept inputs with less time points than the full series length, and output two items: The prediction made and whether the algorithm thinks the prediction is safe. Information about the decision such as the time stamp it was made at can be obtained from the state_info attribute.

First test with only 50 datapoints (out of 251)#

[ ]:

X = arrow_test_X[:, :, :50]
probas, _ = teaser.predict_proba(X)
idx = (probas >= 0).all(axis=1)
print("First 10 Finished prediction\n", np.argwhere(idx).flatten()[:10])
print("First 10 Probabilities of finished predictions\n", probas[idx][:10])

[ ]:

_, acc, _ = teaser.score(X, arrow_test_y)
print("Accuracy with 50 points on Test Data %2.2f" % acc)

We may also do predictions in a streaming scenario where more data becomes available from time to time#

The rationale is to keep the state info from the previous predictions in the TEASER object and use it whenever new data is available.

[ ]:

test_points = [25, 50, 75, 100, 125, 150, 175, 200, 251]
final_states = np.zeros((arrow_test_X.shape[0], 4), dtype=int)
final_decisions = np.zeros(arrow_test_X.shape[0], dtype=int)
open_idx = np.arange(0, arrow_test_X.shape[0])
teaser.reset_state_info()

for i in test_points:
    probas, decisions = teaser.update_predict_proba(arrow_test_X[:, :, :i])
    final_states[open_idx] = teaser.get_state_info()

    arrow_test_X, open_idx, final_idx = teaser.split_indices_and_filter(
        arrow_test_X, open_idx, decisions
    )
    final_decisions[final_idx] = i

    (
        hm,
        acc,
        earliness,
    ) = teaser._compute_harmonic_mean(final_states, arrow_test_y)

    print("Earliness on length %2i is %2.2f" % (i, earliness))
    print("Accuracy on length %2i is %2.2f" % (i, acc))
    print("Harmonic Mean on length %2i is %2.2f" % (i, hm))

    print("...........")

print("Time Stamp of final decisions", final_decisions)

Generated using nbsphinx. The Jupyter notebook can be found here.

On this page