BaseBenchmarkAnalyzer

class BaseBenchmarkAnalyzer(metric=None, lower_is_better=True)[source]

Base class for benchmark analyzers.

Benchmark analyzers consume the flat results table produced by the v2 benchmarking framework (BaseBenchmark.run() / ResultObject.to_dataframe) and compute a benchmark analysis (ranking, omnibus / pairwise significance tests, critical-difference diagrams).

The class is stateless: evaluate does not mutate the instance, so a single evaluator can be reused across multiple result tables.

Concrete evaluators implement _evaluate(scores), where scores is a (n_datasets, n_estimators) pandas DataFrame (index = validation_id, columns = model_id) of the per-task mean score.

Parameters:

metricstr, optional (default=None): Name of the metric to analyse, i.e. the {metric} whose {metric}_mean column is used from the results table. If None, the metric is inferred when exactly one metric column is present; otherwise a ValueError is raised.
lower_is_betterbool, optional (default=True): Whether lower metric values indicate better performance (e.g. error metrics). Controls direction of comparison or ranking.

Methods

`clone`()	Obtain a clone of the object with same hyper-parameters and config.
`clone_tags`(estimator[, tag_names])	Clone tags from another object as dynamic override.
`create_test_instance`([parameter_set])	Construct an instance of the class, using first test parameter set.
`create_test_instances_and_names`([parameter_set])	Create list of all test instances and a list of names for them.
`evaluate`(results)	Compute results of the benchmark analysis, on benchmark results.
`get_class_tag`(tag_name[, tag_value_default])	Get class tag value from class, with tag level inheritance from parents.
`get_class_tags`()	Get class tags from class, with tag level inheritance from parent classes.
`get_config`()	Get config flags for self.
`get_param_defaults`()	Get object's parameter defaults.
`get_param_names`([sort])	Get object's parameter names.
`get_params`([deep])	Get a dict of parameters values for this object.
`get_tag`(tag_name[, tag_value_default, ...])	Get tag value from instance, with tag level inheritance and overrides.
`get_tags`()	Get tags from instance, with tag level inheritance and overrides.
`get_test_params`([parameter_set])	Return testing parameter settings for the evaluator.
`is_composite`()	Check if the object is composed of other BaseObjects.
`load_from_path`(serial)	Load object from file location.
`load_from_serial`(serial)	Load object from serialized memory container.
`reset`()	Reset the object to a clean post-init state.
`save`([path, serialization_format])	Save serialized self to bytes-like object or to (.zip) file.
`set_config`(**config_dict)	Set config flags to given values.
`set_params`(**params)	Set the parameters of this object.
`set_random_state`([random_state, deep, ...])	Set random_state pseudo-random seed parameters for self.
`set_tags`(**tag_dict)	Set instance level tag overrides to given values.