benchopt.run_benchmark

benchopt.run_benchmark(benchmark_path, solver_names=None, forced_solvers=(), dataset_names=None, objective_filters=None, max_runs=10, n_repetitions=1, timeout=None, n_jobs=None, parallel_config=None, slurm=None, plot_result=True, display=True, html=True, collect=False, show_progress=True, pdb=False, no_cache=False, output_file='None')

Run full benchmark.

Parameters:

benchmarkbenchopt.Benchmark object

Object to represent the benchmark.

solver_nameslist | None

List of solver names to include in the benchmark. If None all solvers available are run.

forced_solverslist | None

List of solvers to include in the benchmark and for which one forces recomputation.

dataset_nameslist | None

List of dataset names to include. If None all available datasets are used.

objective_filterslist | None

Filters to select specific objective parameters. If None, all objective parameters are tested

max_runsint

The maximum number of solver runs to perform to estimate the convergence curve.

n_repetitionsint

The number of repetitions to run. Defaults to 1.

timeoutfloat

The maximum duration in seconds of the solver run.

n_jobsint

Maximal number of workers to use to run the benchmark in parallel.

parallel_configdict | None

If not None, launch the job in parallel. The provided config serves to set up parallelism using joblib.parallel_backend or submitit. See Distributed run with Benchopt for detailed description.

slurmPath | None, (_Deprecated_)

If not None, launch the job on a slurm cluster using the file to get the cluster config parameters.

plot_resultbool

If set to True (default), generate the result plot and save them in the benchmark directory.

displaybool

If set to True (default), open the result plots at the end of the run, otherwise, simply save them.

htmlbool

If set to True (default), display the result plot in HTML, otherwise in matplotlib figures, default is True.

collectbool

If set to True, only collect the results that have been put in cache, and ignore the results that are not computed yet, default is False.

show_progressbool

If show_progress is set to True, display the progress of the benchmark.

pdbbool

If pdb is set to True, open a debugger on error.

no_cachebool

If set to True, this deactivates the caching mechanism integrated in benchopt. Note that this makes the run less tolerant to errors, use it with caution.

output_filestr

Filename for the parquet output. If given, the results will be stored at <BENCHMARK>/outputs/<filename>.parquet.

Returns:

exit_codeint

Exit code of the benchmark run. 0 if everything went fine, 1 otherwise.

output_filePath

Path to the output file where the results have been saved.