TwoSampleTTest

class TwoSampleTTest(metric=None, lower_is_better=True, correction='none', alpha=0.05)[source]

Independent two-sample t-test on each ordered pair of estimators.

Reproduces the legacy Evaluator.t_test (scipy.stats.ttest_ind) and, via correction="bonferroni", the legacy Evaluator.t_test_with_bonferroni_correction: a "significant" column is added that flags pairs whose p-value falls below the Bonferroni-corrected threshold alpha / n_pairs.

Parameters:

metricstr, optional (default=None): Metric to analyse; see BaseBenchmarkAnalyzer.
lower_is_betterbool, optional (default=True): Direction of the metric; not used by the t-test itself but kept for a consistent evaluator interface.
correctionstr, optional (default=”none”): Multiple-comparison correction. One of "none" or "bonferroni".
alphafloat, optional (default=0.05): Family-wise significance level, used only when correction="bonferroni".

Returns:

pandas.DataFrame: Columns ["estimator_1", "estimator_2", "statistic", "p_val"], plus a boolean "significant" column when correction="bonferroni".

Methods

`clone`()	Obtain a clone of the object with same hyper-parameters and config.
`clone_tags`(estimator[, tag_names])	Clone tags from another object as dynamic override.
`create_test_instance`([parameter_set])	Construct an instance of the class, using first test parameter set.
`create_test_instances_and_names`([parameter_set])	Create list of all test instances and a list of names for them.
`evaluate`(results)	Compute results of the benchmark analysis, on benchmark results.
`get_class_tag`(tag_name[, tag_value_default])	Get class tag value from class, with tag level inheritance from parents.
`get_class_tags`()	Get class tags from class, with tag level inheritance from parent classes.
`get_config`()	Get config flags for self.
`get_param_defaults`()	Get object's parameter defaults.
`get_param_names`([sort])	Get object's parameter names.
`get_params`([deep])	Get a dict of parameters values for this object.
`get_tag`(tag_name[, tag_value_default, ...])	Get tag value from instance, with tag level inheritance and overrides.
`get_tags`()	Get tags from instance, with tag level inheritance and overrides.
`get_test_params`([parameter_set])	Return testing parameter settings for the evaluator.
`is_composite`()	Check if the object is composed of other BaseObjects.
`load_from_path`(serial)	Load object from file location.
`load_from_serial`(serial)	Load object from serialized memory container.
`reset`()	Reset the object to a clean post-init state.
`save`([path, serialization_format])	Save serialized self to bytes-like object or to (.zip) file.
`set_config`(**config_dict)	Set config flags to given values.
`set_params`(**params)	Set the parameters of this object.
`set_random_state`([random_state, deep, ...])	Set random_state pseudo-random seed parameters for self.
`set_tags`(**tag_dict)	Set instance level tag overrides to given values.