MedianSquaredScaledError#
- class MedianSquaredScaledError(multioutput='uniform_average', multilevel='uniform_average', sp=1, square_root=False)[source]#
Median squared scaled error (MdSSE) or root median squared scaled error (RMdSSE).
If
square_root
is False then calculates MdSSE, otherwise calculates RMdSSE ifsquare_root
is True. Both MdSSE and RMdSSE output is non-negative floating point. The best value is 0.0.This is a squared variant of the MdASE loss metric. Like MASE and other scaled performance metrics this scale-free metric can be used to compare forecast methods on a single series or between series.
This metric is also suited for intermittent-demand series because it will not give infinite or undefined values unless the training data is a flat timeseries. In this case the function returns a large value instead of inf.
Works with multioutput (multivariate) timeseries data with homogeneous seasonal periodicity.
- Parameters:
- spint, default = 1
Seasonal periodicity of data.
- square_rootbool, default = False
Whether to take the square root of the metric
- multioutput{‘raw_values’, ‘uniform_average’} or array-like of shape (n_outputs,), default=’uniform_average’
Defines how to aggregate metric for multivariate (multioutput) data. If array-like, values used as weights to average the errors. If ‘raw_values’, returns a full set of errors in case of multioutput input. If ‘uniform_average’, errors of all outputs are averaged with uniform weight.
- multilevel{‘raw_values’, ‘uniform_average’, ‘uniform_average_time’}
Defines how to aggregate metric for hierarchical data (with levels). If ‘uniform_average’ (default), errors are mean-averaged across levels. If ‘uniform_average_time’, metric is applied to all data, ignoring level index. If ‘raw_values’, does not average errors across levels, hierarchy is retained.
References
M5 Competition Guidelines.
https://mofc.unic.ac.cy/wp-content/uploads/2020/03/M5-Competitors-Guide-Final-10-March-2020.docx
Hyndman, R. J and Koehler, A. B. (2006). “Another look at measures of forecast accuracy”, International Journal of Forecasting, Volume 22, Issue 4.
Examples
>>> import numpy as np >>> from sktime.performance_metrics.forecasting import MedianSquaredScaledError >>> y_train = np.array([5, 0.5, 4, 6, 3, 5, 2]) >>> y_true = np.array([3, -0.5, 2, 7, 2]) >>> y_pred = np.array([2.5, 0.0, 2, 8, 1.25]) >>> rmdsse = MedianSquaredScaledError(square_root=True) >>> rmdsse(y_true, y_pred, y_train=y_train) 0.16666666666666666 >>> y_train = np.array([[0.5, 1], [-1, 1], [7, -6]]) >>> y_true = np.array([[0.5, 1], [-1, 1], [7, -6]]) >>> y_pred = np.array([[0, 2], [-1, 2], [8, -5]]) >>> rmdsse(y_true, y_pred, y_train=y_train) 0.1472819539849714 >>> rmdsse = MedianSquaredScaledError(multioutput='raw_values', square_root=True) >>> rmdsse(y_true, y_pred, y_train=y_train) array([0.08687445, 0.20203051]) >>> rmdsse = MedianSquaredScaledError(multioutput=[0.3, 0.7], square_root=True) >>> rmdsse(y_true, y_pred, y_train=y_train) 0.16914781383660782
Methods
__call__
(y_true, y_pred, **kwargs)Calculate metric value using underlying metric function.
clone
()Obtain a clone of the object with same hyper-parameters.
clone_tags
(estimator[, tag_names])Clone tags from another estimator as dynamic override.
create_test_instance
([parameter_set])Construct Estimator instance if possible.
create_test_instances_and_names
([parameter_set])Create list of all test instances and a list of names for them.
evaluate
(y_true, y_pred, **kwargs)Evaluate the desired metric on given inputs.
evaluate_by_index
(y_true, y_pred, **kwargs)Return the metric evaluated at each time point.
func
(y_pred[, sp, horizon_weight, ...])Median squared scaled error (MdSSE) or root median squared scaled error (RMdSSE).
get_class_tag
(tag_name[, tag_value_default])Get a class tag's value.
Get class tags from the class and all its parent classes.
Get config flags for self.
Get object's parameter defaults.
Get object's parameter names.
get_params
([deep])Get a dict of parameters values for this object.
get_tag
(tag_name[, tag_value_default, ...])Get tag value from estimator class and dynamic tag overrides.
get_tags
()Get tags from estimator class and dynamic tag overrides.
get_test_params
([parameter_set])Return testing parameter settings for the estimator.
Check if the object is composed of other BaseObjects.
load_from_path
(serial)Load object from file location.
load_from_serial
(serial)Load object from serialized memory container.
reset
()Reset the object to a clean post-init state.
save
([path, serialization_format])Save serialized self to bytes-like object or to (.zip) file.
set_config
(**config_dict)Set config flags to given values.
set_params
(**params)Set the parameters of this object.
set_random_state
([random_state, deep, ...])Set random_state pseudo-random seed parameters for self.
set_tags
(**tag_dict)Set dynamic tags to given values.
- func(y_pred, sp=1, horizon_weight=None, multioutput='uniform_average', square_root=False, **kwargs)[source]#
Median squared scaled error (MdSSE) or root median squared scaled error (RMdSSE).
If
square_root
is False then calculates MdSSE, otherwise calculates RMdSSE ifsquare_root
is True. Both MdSSE and RMdSSE output is non-negative floating point. The best value is 0.0.This is a squared variant of the MdASE loss metric. Like MASE and other scaled performance metrics this scale-free metric can be used to compare forecast methods on a single series or between series.
This metric is also suited for intermittent-demand series because it will not give infinite or undefined values unless the training data is a flat timeseries. In this case the function returns a large value instead of inf.
Works with multioutput (multivariate) timeseries data with homogeneous seasonal periodicity.
- Parameters:
- y_truepd.Series, pd.DataFrame or np.array of shape (fh,) or (fh, n_outputs) where fh is the forecasting horizon
Ground truth (correct) target values.
- y_predpd.Series, pd.DataFrame or np.array of shape (fh,) or (fh, n_outputs) where fh is the forecasting horizon
Forecasted values.
- y_trainpd.Series, pd.DataFrame or np.array of shape (n_timepoints,) or (n_timepoints, n_outputs), default = None
Observed training values.
- spint
Seasonal periodicity of training data.
- horizon_weightarray-like of shape (fh,), default=None
Forecast horizon weights.
- multioutput{‘raw_values’, ‘uniform_average’} or array-like of shape (n_outputs,), default=’uniform_average’
Defines how to aggregate metric for multivariate (multioutput) data. If array-like, values used as weights to average the errors. If ‘raw_values’, returns a full set of errors in case of multioutput input. If ‘uniform_average’, errors of all outputs are averaged with uniform weight.
- Returns:
- lossfloat
RMdSSE loss. If multioutput is ‘raw_values’, then RMdSSE is returned for each output separately. If multioutput is ‘uniform_average’ or an ndarray of weights, then the weighted average RMdSSE of all output errors is returned.
References
M5 Competition Guidelines.
https://mofc.unic.ac.cy/wp-content/uploads/2020/03/M5-Competitors-Guide-Final-10-March-2020.docx
Hyndman, R. J and Koehler, A. B. (2006). “Another look at measures of forecast accuracy”, International Journal of Forecasting, Volume 22, Issue 4.
Examples
>>> from sktime.performance_metrics.forecasting import median_squared_scaled_error >>> y_train = np.array([5, 0.5, 4, 6, 3, 5, 2]) >>> y_true = np.array([3, -0.5, 2, 7, 2]) >>> y_pred = np.array([2.5, 0.0, 2, 8, 1.25]) >>> median_squared_scaled_error(y_true, y_pred, y_train=y_train, square_root=True) 0.16666666666666666 >>> y_train = np.array([[0.5, 1], [-1, 1], [7, -6]]) >>> y_true = np.array([[0.5, 1], [-1, 1], [7, -6]]) >>> y_pred = np.array([[0, 2], [-1, 2], [8, -5]]) >>> median_squared_scaled_error(y_true, y_pred, y_train=y_train, square_root=True) 0.1472819539849714 >>> median_squared_scaled_error(y_true, y_pred, y_train=y_train, multioutput='raw_values', square_root=True) array([0.08687445, 0.20203051]) >>> median_squared_scaled_error(y_true, y_pred, y_train=y_train, multioutput=[0.3, 0.7], square_root=True) 0.16914781383660782
- classmethod get_test_params(parameter_set='default')[source]#
Return testing parameter settings for the estimator.
- Parameters:
- parameter_setstr, default=”default”
Name of the set of test parameters to return, for use in tests. If no special parameters are defined for a value, will return
"default"
set.
- Returns:
- paramsdict or list of dict, default = {}
Parameters to create testing instances of the class Each dict are parameters to construct an “interesting” test instance, i.e.,
MyClass(**params)
orMyClass(**params[i])
creates a valid test instance.create_test_instance
uses the first (or only) dictionary inparams
- __call__(y_true, y_pred, **kwargs)[source]#
Calculate metric value using underlying metric function.
- Parameters:
- y_truetime series in sktime compatible data container format
Ground truth (correct) target values y can be in one of the following formats: Series scitype: pd.Series, pd.DataFrame, or np.ndarray (1D or 2D) Panel scitype: pd.DataFrame with 2-level row MultiIndex,
3D np.ndarray, list of Series pd.DataFrame, or nested pd.DataFrame
Hierarchical scitype: pd.DataFrame with 3 or more level row MultiIndex
- y_pred :time series in sktime compatible data container format
Forecasted values to evaluate must be of same format as y_true, same indices and columns if indexed
- Returns:
- lossfloat, np.ndarray, or pd.DataFrame
Calculated metric, averaged or by variable. float if self.multioutput=”uniform_average” or array-like
and self.multilevel=”uniform_average” or “uniform_average_time” value is metric averaged over variables and levels (see class docstring)
- np.ndarray of shape (y_true.columns,) if self.multioutput=”raw_values”
and self.multilevel=”uniform_average” or “uniform_average_time” i-th entry is metric calculated for i-th variable
- pd.DataFrame if self.multilevel=raw.values
of shape (n_levels, ) if self.multioutput = “uniform_average” or array of shape (n_levels, y_true.columns) if self.multioutput=”raw_values” metric is applied per level, row averaging (yes/no) as in multioutput
- clone()[source]#
Obtain a clone of the object with same hyper-parameters.
A clone is a different object without shared references, in post-init state. This function is equivalent to returning sklearn.clone of self.
- Raises:
- RuntimeError if the clone is non-conforming, due to faulty
__init__
.
- RuntimeError if the clone is non-conforming, due to faulty
Notes
If successful, equal in value to
type(self)(**self.get_params(deep=False))
.
- clone_tags(estimator, tag_names=None)[source]#
Clone tags from another estimator as dynamic override.
- Parameters:
- estimatorestimator inheriting from :class:BaseEstimator
- tag_namesstr or list of str, default = None
Names of tags to clone. If None then all tags in estimator are used as tag_names.
- Returns:
- Self
Reference to self.
Notes
Changes object state by setting tag values in tag_set from estimator as dynamic tags in self.
- classmethod create_test_instance(parameter_set='default')[source]#
Construct Estimator instance if possible.
- Parameters:
- parameter_setstr, default=”default”
Name of the set of test parameters to return, for use in tests. If no special parameters are defined for a value, will return “default” set.
- Returns:
- instanceinstance of the class with default parameters
Notes
get_test_params can return dict or list of dict. This function takes first or single dict that get_test_params returns, and constructs the object with that.
- classmethod create_test_instances_and_names(parameter_set='default')[source]#
Create list of all test instances and a list of names for them.
- Parameters:
- parameter_setstr, default=”default”
Name of the set of test parameters to return, for use in tests. If no special parameters are defined for a value, will return “default” set.
- Returns:
- objslist of instances of cls
i-th instance is cls(**cls.get_test_params()[i])
- nameslist of str, same length as objs
i-th element is name of i-th instance of obj in tests convention is {cls.__name__}-{i} if more than one instance otherwise {cls.__name__}
- evaluate(y_true, y_pred, **kwargs)[source]#
Evaluate the desired metric on given inputs.
- Parameters:
- y_truetime series in sktime compatible data container format
Ground truth (correct) target values y can be in one of the following formats: Series scitype: pd.Series, pd.DataFrame, or np.ndarray (1D or 2D) Panel scitype: pd.DataFrame with 2-level row MultiIndex,
3D np.ndarray, list of Series pd.DataFrame, or nested pd.DataFrame
Hierarchical scitype: pd.DataFrame with 3 or more level row MultiIndex
- y_pred :time series in sktime compatible data container format
Forecasted values to evaluate must be of same format as y_true, same indices and columns if indexed
- Returns:
- lossfloat, np.ndarray, or pd.DataFrame
Calculated metric, averaged or by variable. float if self.multioutput=”uniform_average” or array-like
and self.multilevel=”uniform_average” or “uniform_average_time” value is metric averaged over variables and levels (see class docstring)
- np.ndarray of shape (y_true.columns,) if self.multioutput=”raw_values”
and self.multilevel=”uniform_average” or “uniform_average_time” i-th entry is metric calculated for i-th variable
- pd.DataFrame if self.multilevel=raw.values
of shape (n_levels, ) if self.multioutput = “uniform_average” or array of shape (n_levels, y_true.columns) if self.multioutput=”raw_values” metric is applied per level, row averaging (yes/no) as in multioutput
- evaluate_by_index(y_true, y_pred, **kwargs)[source]#
Return the metric evaluated at each time point.
- Parameters:
- y_truetime series in sktime compatible data container format
Ground truth (correct) target values y can be in one of the following formats: Series scitype: pd.Series, pd.DataFrame, or np.ndarray (1D or 2D) Panel scitype: pd.DataFrame with 2-level row MultiIndex,
3D np.ndarray, list of Series pd.DataFrame, or nested pd.DataFrame
Hierarchical scitype: pd.DataFrame with 3 or more level row MultiIndex
- y_pred :time series in sktime compatible data container format
Forecasted values to evaluate must be of same format as y_true, same indices and columns if indexed
- Returns:
- losspd.Series or pd.DataFrame
Calculated metric, by time point (default=jackknife pseudo-values). pd.Series if self.multioutput=”uniform_average” or array-like
index is equal to index of y_true entry at index i is metric at time i, averaged over variables
- pd.DataFrame if self.multioutput=”raw_values”
index and columns equal to those of y_true i,j-th entry is metric at time i, at variable j
- classmethod get_class_tag(tag_name, tag_value_default=None)[source]#
Get a class tag’s value.
Does not return information from dynamic tags (set via set_tags or clone_tags) that are defined on instances.
- Parameters:
- tag_namestr
Name of tag value.
- tag_value_defaultany
Default/fallback value if tag is not found.
- Returns:
- tag_value
Value of the tag_name tag in self. If not found, returns tag_value_default.
- classmethod get_class_tags()[source]#
Get class tags from the class and all its parent classes.
Retrieves tag: value pairs from _tags class attribute. Does not return information from dynamic tags (set via set_tags or clone_tags) that are defined on instances.
- Returns:
- collected_tagsdict
Dictionary of class tag name: tag value pairs. Collected from _tags class attribute via nested inheritance.
- get_config()[source]#
Get config flags for self.
- Returns:
- config_dictdict
Dictionary of config name : config value pairs. Collected from _config class attribute via nested inheritance and then any overrides and new tags from _onfig_dynamic object attribute.
- classmethod get_param_defaults()[source]#
Get object’s parameter defaults.
- Returns:
- default_dict: dict[str, Any]
Keys are all parameters of cls that have a default defined in __init__ values are the defaults, as defined in __init__.
- classmethod get_param_names()[source]#
Get object’s parameter names.
- Returns:
- param_names: list[str]
Alphabetically sorted list of parameter names of cls.
- get_params(deep=True)[source]#
Get a dict of parameters values for this object.
- Parameters:
- deepbool, default=True
Whether to return parameters of components.
If True, will return a dict of parameter name : value for this object, including parameters of components (= BaseObject-valued parameters).
If False, will return a dict of parameter name : value for this object, but not include parameters of components.
- Returns:
- paramsdict with str-valued keys
Dictionary of parameters, paramname : paramvalue keys-value pairs include:
always: all parameters of this object, as via get_param_names values are parameter value for that key, of this object values are always identical to values passed at construction
if deep=True, also contains keys/value pairs of component parameters parameters of components are indexed as [componentname]__[paramname] all parameters of componentname appear as paramname with its value
if deep=True, also contains arbitrary levels of component recursion, e.g., [componentname]__[componentcomponentname]__[paramname], etc
- get_tag(tag_name, tag_value_default=None, raise_error=True)[source]#
Get tag value from estimator class and dynamic tag overrides.
- Parameters:
- tag_namestr
Name of tag to be retrieved
- tag_value_defaultany type, optional; default=None
Default/fallback value if tag is not found
- raise_errorbool
whether a ValueError is raised when the tag is not found
- Returns:
- tag_valueAny
Value of the tag_name tag in self. If not found, returns an error if raise_error is True, otherwise it returns tag_value_default.
- Raises:
- ValueError if raise_error is True i.e. if tag_name is not in
- self.get_tags().keys()
- get_tags()[source]#
Get tags from estimator class and dynamic tag overrides.
- Returns:
- collected_tagsdict
Dictionary of tag name : tag value pairs. Collected from _tags class attribute via nested inheritance and then any overrides and new tags from _tags_dynamic object attribute.
- is_composite()[source]#
Check if the object is composed of other BaseObjects.
A composite object is an object which contains objects, as parameters. Called on an instance, since this may differ by instance.
- Returns:
- composite: bool
Whether an object has any parameters whose values are BaseObjects.
- classmethod load_from_path(serial)[source]#
Load object from file location.
- Parameters:
- serialresult of ZipFile(path).open(“object)
- Returns:
- deserialized self resulting in output at
path
, ofcls.save(path)
- deserialized self resulting in output at
- classmethod load_from_serial(serial)[source]#
Load object from serialized memory container.
- Parameters:
- serial1st element of output of
cls.save(None)
- serial1st element of output of
- Returns:
- deserialized self resulting in output
serial
, ofcls.save(None)
- deserialized self resulting in output
- reset()[source]#
Reset the object to a clean post-init state.
Using reset, runs __init__ with current values of hyper-parameters (result of get_params). This Removes any object attributes, except:
hyper-parameters = arguments of __init__
object attributes containing double-underscores, i.e., the string “__”
Class and object methods, and class attributes are also unaffected.
- Returns:
- self
Instance of class reset to a clean post-init state but retaining the current hyper-parameter values.
Notes
Equivalent to sklearn.clone but overwrites self. After self.reset() call, self is equal in value to type(self)(**self.get_params(deep=False))
- save(path=None, serialization_format='pickle')[source]#
Save serialized self to bytes-like object or to (.zip) file.
Behaviour: if
path
is None, returns an in-memory serialized self ifpath
is a file location, stores self at that location as a zip filesaved files are zip files with following contents: _metadata - contains class of self, i.e., type(self) _obj - serialized self. This class uses the default serialization (pickle).
- Parameters:
- pathNone or file location (str or Path)
if None, self is saved to an in-memory object if file location, self is saved to that file location. If:
path=”estimator” then a zip file
estimator.zip
will be made at cwd. path=”/home/stored/estimator” then a zip fileestimator.zip
will be stored in/home/stored/
.- serialization_format: str, default = “pickle”
Module to use for serialization. The available options are “pickle” and “cloudpickle”. Note that non-default formats might require installation of other soft dependencies.
- Returns:
- if
path
is None - in-memory serialized self - if
path
is file location - ZipFile with reference to the file
- if
- set_config(**config_dict)[source]#
Set config flags to given values.
- Parameters:
- config_dictdict
Dictionary of config name : config value pairs. Valid configs, values, and their meaning is listed below:
- displaystr, “diagram” (default), or “text”
how jupyter kernels display instances of self
“diagram” = html box diagram representation
“text” = string printout
- print_changed_onlybool, default=True
whether printing of self lists only self-parameters that differ from defaults (False), or all parameter names and values (False). Does not nest, i.e., only affects self and not component estimators.
- warningsstr, “on” (default), or “off”
whether to raise warnings, affects warnings from sktime only
“on” = will raise warnings from sktime
“off” = will not raise warnings from sktime
- backend:parallelstr, optional, default=”None”
backend to use for parallelization when broadcasting/vectorizing, one of
“None”: executes loop sequentally, simple list comprehension
“loky”, “multiprocessing” and “threading”: uses
joblib.Parallel
“joblib”: custom and 3rd party
joblib
backends, e.g.,spark
“dask”: uses
dask
, requiresdask
package in environment
- backend:parallel:paramsdict, optional, default={} (no parameters passed)
additional parameters passed to the parallelization backend as config. Valid keys depend on the value of
backend:parallel
:“None”: no additional parameters,
backend_params
is ignored“loky”, “multiprocessing” and “threading”: default
joblib
backends any valid keys forjoblib.Parallel
can be passed here, e.g.,n_jobs
, with the exception ofbackend
which is directly controlled bybackend
. Ifn_jobs
is not passed, it will default to-1
, other parameters will default tojoblib
defaults.“joblib”: custom and 3rd party
joblib
backends, e.g.,spark
. Any valid keys forjoblib.Parallel
can be passed here, e.g.,n_jobs
,backend
must be passed as a key ofbackend_params
in this case. Ifn_jobs
is not passed, it will default to-1
, other parameters will default tojoblib
defaults.“dask”: any valid keys for
dask.compute
can be passed, e.g.,scheduler
- Returns:
- selfreference to self.
Notes
Changes object state, copies configs in config_dict to self._config_dynamic.
- set_params(**params)[source]#
Set the parameters of this object.
The method works on simple estimators as well as on composite objects. Parameter key strings
<component>__<parameter>
can be used for composites, i.e., objects that contain other objects, to access<parameter>
in the component<component>
. The string<parameter>
, without<component>__
, can also be used if this makes the reference unambiguous, e.g., there are no two parameters of components with the name<parameter>
.- Parameters:
- **paramsdict
BaseObject parameters, keys must be
<component>__<parameter>
strings. __ suffixes can alias full strings, if unique among get_params keys.
- Returns:
- selfreference to self (after parameters have been set)
- set_random_state(random_state=None, deep=True, self_policy='copy')[source]#
Set random_state pseudo-random seed parameters for self.
Finds
random_state
named parameters viaestimator.get_params
, and sets them to integers derived fromrandom_state
viaset_params
. These integers are sampled from chain hashing viasample_dependent_seed
, and guarantee pseudo-random independence of seeded random generators.Applies to
random_state
parameters inestimator
depending onself_policy
, and remaining component estimators if and only ifdeep=True
.Note: calls
set_params
even ifself
does not have arandom_state
, or none of the components have arandom_state
parameter. Therefore,set_random_state
will reset anyscikit-base
estimator, even those without arandom_state
parameter.- Parameters:
- random_stateint, RandomState instance or None, default=None
Pseudo-random number generator to control the generation of the random integers. Pass int for reproducible output across multiple function calls.
- deepbool, default=True
Whether to set the random state in sub-estimators. If False, will set only
self
’srandom_state
parameter, if exists. If True, will setrandom_state
parameters in sub-estimators as well.- self_policystr, one of {“copy”, “keep”, “new”}, default=”copy”
“copy” :
estimator.random_state
is set to inputrandom_state
“keep” :
estimator.random_state
is kept as is“new” :
estimator.random_state
is set to a new random state,
derived from input
random_state
, and in general different from it
- Returns:
- selfreference to self