ClassificationBenchmark

ClassificationBenchmark

class ClassificationBenchmark(id_format: str | None = None, backend=None, backend_params=None, return_data=False)[source]

Classification benchmark.

Run a series of classifiers against a series of tasks defined via dataset loaders, cross validation splitting strategies and performance metrics, and return results as a df (as well as saving to file).

Parameters:

id_format: str, optional (default=None)

A regex used to enforce task/estimator ID to match a certain format

backendstring, by default “None”.

Parallelization backend to use for runs.

• “None”: executes loop sequentially, simple list comprehension

• “loky”, “multiprocessing” and “threading”: uses joblib.Parallel loops

• “joblib”: custom and 3rd party joblib backends, e.g., spark

• “dask”: uses dask, requires dask package in environment

• “dask_lazy”: same as “dask”, but changes the return to (lazy)

  dask.dataframe.DataFrame.

• “ray”: uses ray, requires ray package in environment

Recommendation: Use “dask” or “loky” for parallel evaluate. “threading” is unlikely to see speed ups due to the GIL and the serialization backend (cloudpickle) for “dask” and “loky” is generally more robust than the standard pickle library used in “multiprocessing”.

backend_paramsdict, optional

additional parameters passed to the backend as config. Directly passed to utils.parallel.parallelize. Valid keys depend on the value of backend:

• “None”: no additional parameters, backend_params is ignored

• “loky”, “multiprocessing” and “threading”: default joblib backends

any valid keys for joblib.Parallel can be passed here, e.g., n_jobs, with the exception of backend which is directly controlled by backend. If n_jobs is not passed, it will default to -1, other parameters will default to joblib defaults. - “joblib”: custom and 3rd party joblib backends, e.g., spark. any valid keys for joblib.Parallel can be passed here, e.g., n_jobs, backend must be passed as a key of backend_params in this case. If n_jobs is not passed, it will default to -1, other parameters will default to joblib defaults. - “dask”: any valid keys for dask.compute can be passed, e.g., scheduler

• “ray”: The following keys can be passed:

  • “ray_remote_args”: dictionary of valid keys for ray.init

  • “shutdown_ray”: bool, default=True; False prevents ray from shutting

    down after parallelization.

  • “logger_name”: str, default=”ray”; name of the logger to use.

  • “mute_warnings”: bool, default=False; if True, suppresses warnings

return_databool, optional (default=False)

Whether to return the prediction and the ground truth data in the results.

Attributes:

failed_experiments

Failed task-estimator pairs from the most recent benchmark run.

Methods

add(*args)	Add estimators, task components, full task tuples, or catalogues.
add_estimator(estimator[, estimator_id])	Register an estimator to the benchmark.
add_task(dataset_loader, cv_splitter, scorers)	Register a classification task to the benchmark.
register_stored_tasks()	Register stored tasks from datasets, metrics, and CV splitters.
run([output_file, force_rerun])	Run the benchmarking for all tasks and estimators.