Run a benchmark

Run a benchmark#

Let’s use the Lasso benchmark to illustrate ways of running a benchmark.

Hint

Head to Get started to first install benchopt and setup the Lasso benchmark.

With the Command line interface (CLI) references, there are two ways to run a benchmark: passing options with flags in the CLI, or with a configuration file.

Specifying options with CLI flags#

It is possible to specify the solvers as well as the datasets to include in the benchmark run by using flags after benchopt run ..

For instance, the following command runs the benchmark with solvers skglm and celer, on datasets leukemia and simulated.

benchopt run . -s skglm -s celer -d leukemia -d simulated

The -s flag is to specify a solver whereas -d specifies a dataset. To include multiple datasets/solvers, use multiple -d/-s flags, as in the above snippet.

Note

The run command accepts other flags such as -j to run the benchmark in parallel with a given number of processes. The list of flags is available through benchopt run --help or in the Command line interface (CLI) references page.

In addition, it is possible to specify the parameters of solvers and datasets by wrapping them in square brackets in comma separated format.

The following snippet runs the Python-PGD solver with its acceleration parameter set to True, on the simulated dataset. This dataset has parameters n_samples and n_features that we set to 100 and 20 respectively.

benchopt run . -s Python-PGD[use_acceleration=True] -d simulated[n_samples=100,n_features=20]

benchopt run . -s "Python-PGD[use_acceleration=True]" -d "simulated[n_samples=100,n_features=20]"

Note

If a parameter of a solver/dataset is not explicitly set via CLI, benchopt uses all its values specified in the code.

Using a configuration file#

When using a complex configuration, it is more handy to specify it through a configuration file. Using a YAML file and the --config flag, it is possible to describe all details of the benchmark run and execute instead

benchopt run . --config ./example_config.yml

Here is the content of configuration file example_config.yml if we were to run the two previous examples into a single one.

solver:
    - skglm
    - celer
    - python-pgd[use_acceleration=True]

dataset:
    - leukemia
    - simulated[n_samples=100,n_features=10]
    - simulated:
        n_samples: 100
        n_features: [20, 30]
    - simulated:
        n_samples, n_features: [[200, 20], [150, 30]]

Note

A third, less frequent, option to run a benchmark is using a Python script. Check it out on advanced usage.

Contents