BaseBenchmark

class BaseBenchmark(id_format: str | None = None)

Base class for benchmarks.

A benchmark consists of a set of tasks and a set of estimators.

Parameters:

id_format: str, optional (default=None)

A regex used to enforce task/estimator ID to match a certain format if None, no format is enforced on task/estimator ID

Methods

add_estimator(estimator[, estimator_id])	Register an estimator to the benchmark.
run(output_file[, force_rerun])	Run the benchmarking for all tasks and estimators.

add_estimator(estimator: BaseEstimator, estimator_id: str | None = None)

Register an estimator to the benchmark.

Parameters:

estimatorDict, List or BaseEstimator object

Estimator to add to the benchmark. If Dict, keys are estimator_ids used to customise identifier ID and values are estimators. If List, each element is an estimator. estimator_ids are generated automatically using the estimator’s class name.

estimator_idstr, optional (default=None)

Identifier for estimator. If none given then uses estimator’s class name.

run(output_file: str, force_rerun: str | list[str] = 'none')

Run the benchmarking for all tasks and estimators.

Parameters:

output_filestr

Path to save the results to.

force_rerunUnion[str, list[str]], optional (default=”none”)

If “none”, will skip validation if results already exist. If “all”, will run validation for all tasks and models. If list of str, will run validation for tasks and models in list.