Run a benchmark#
Let’s use the Lasso benchmark to illustrate ways of running a benchmark.
Head to Get started to first install benchopt and setup the Lasso benchmark.
With the Command line interface (CLI) references, there are two ways to run a benchmark: passing options with flags in the CLI, or with a configuration file.
Specifying options with CLI flags#
It is possible to specify the solvers as well as the datasets
to include in the benchmark run by using flags after
benchopt run ..
For instance, the following command runs the benchmark with solvers
celer, on datasets
benchopt run . -s skglm -s celer -d leukemia -d simulated
-s flag is to specify a solver whereas
-d specifies a dataset.
To include multiple datasets/solvers, use multiple
-s flags, as in the above snippet.
run command accepts other flags such as
-j to run the benchmark in parallel with a given number of processes.
The list of flags is available through
benchopt run --help or in the Command line interface (CLI) references page.
In addition, it is possible to specify the parameters of solvers and datasets by wrapping them in square brackets in comma separated format.
The following snippet runs the
Python-PGD solver with its acceleration parameter set to
True, on the
This dataset has parameters
n_features that we set to
benchopt run . -s Python-PGD[use_acceleration=True] -d simulated[n_samples=100,n_features=20]
benchopt run . -s "Python-PGD[use_acceleration=True]" -d "simulated[n_samples=100,n_features=20]"
If a parameter of a solver/dataset is not explicitly set via CLI, benchopt uses all its values specified in the code.
Using a configuration file#
When using a complex configuration, it is more handy to specify it through a configuration file.
Using a YAML file and the
--config flag, it is possible to describe all details of the benchmark run and execute instead
benchopt run . --config ./example_config.yml
Here is the content of configuration file
example_config.yml if we were to run the two previous example into a single one.
solver: - skglm - celer - python-pgd[use_acceleration=True] dataset: - leukemia - simulated[n_samples=100,n_features=20]
A third, less frequent, option to run a benchmark is using a Python script. Check it out on advanced usage.