Advanced usage#

This page intends to list some advanced functionality to make it easier to use the benchmark.

Running the benchmark on a SLURM cluster#

Benchopt also allows easily running the benchmark in parallel on a SLURM cluster. To install the necessary dependencies, please run:

pip install benchopt[slurm]

# Or for dev install
pip install -e .[slurm]

Note that for some clusters with shared python installation, it is necessary to call pip install --user to install the packages in the user space and not in the the system one.

Using the --slurm option for benchopt run, one can pass a config file used to setup the SLURM jobs. This file is a YAML file that can contain any key to be passed to the update_parameters method of submitit.SlurmExecutor. Hereafter is an example of such config file:

slurm_time: 01:00:00        # max runtime 1 hour
slurm_gres: gpu:1           # requires 1 GPU per job
slurm_additional_parameters:
  ntasks: 1                 # Number of tasks per job
  cpus-per-task: 10         # requires 10 CPUs per job
  qos: QOS_NAME             # Queue used for the jobs
  distribution: block:block # Distribution on the node's architectures
  account: ACC@NAME         # Account for the jobs
slurm_setup:  # sbatch script commands added before the main job
  - '#SBATCH -C v100-16g'
  - module purge
  - module load cuda/10.1.2 cudnn/7.6.5.32-cuda-10.1 nccl/2.5.6-2-cuda

Using such option, each configuration of (dataset, objective, solver) with unique parameters are launched as a separated job in a job-array on the SLURM cluster. Note that by default, no limitation is used on the number of simultaneous jobs that are run.

If slurm_time is not set in the config file, benchopt uses by default the value of --timeout multiplied by 1.5 for each job. Note that the logs of each benchmark run can be found in ./benchopt_run/.

As we rely on joblib.Memory for caching the results, the cache should work exactly as if you were running the computation sequentially, as long as you have a shared file-system between the nodes used for the computations.

Skipping a solver for a given problem#

Some solvers might not be able to run with all the datasets present in a benchmark. This is typically the case when some datasets are represented using sparse data or for datasets that are too large.

In this cases, a solver can be skipped at runtime, depending on the characteristic of the objective. In order to define for which cases a solver should be skip, the user needs to implement a method skip() in the solver class that will take the input as the set_objective() method. This method should return a boolean value that evaluate to True if the solver should be skipped, and a string giving the reason of why the solver is skipped, for display purposes. For instance, for a solver where the objective is set with keys X, y, reg, we get

class Solver(BaseSolver):
    ...
    def skip(self, X, y, reg):
        from scipy import sparse

        if sparse.issparse(X):
            return True, "solver does not support sparse data X."

        if reg == 0:
            return True, "solver does not work with reg=0"

        return False, None

Changing the strategy to grow the computational budget (`stop_val`)#

Benchopt varies the computational budget by varying either the number of iterations or the tolerance given to the method. The default policy is to vary these two quantities exponentially between two evaluations of the objective. However, in some cases, this exponential growth might hide some effects, or might not be adapted to a given solver.

The way this value is changed can be specified for each solver by implementing a get_next method in the Solver class. This method takes as input the previous value where the objective function has been logged, and outputs the next one. For instance, if a solver needs to be evaluated every 10 iterations, we would have

class Solver(BaseSolver):
    ...
    def get_next(self, stop_val):
        return stop_val + 10

Reusing some code in a benchmark#

In some situations, multiple solvers need to have access to the same functions. As a benchmark is not structured as proper python packages but imported dynamically to avoid installation issues, we resort to a special way of importing modules and functions defined for a benchmark.

First, all code that need to be imported should be placed under BENCHMARK_DIR/benchmark_utils/, as described here:

my_benchmark/
├── objective.py  # contains the definition of the objective
├── datasets/
├── solvers/
└── benchmark_utils/
    ├── __init__.py
    ├── helper1.py  # some helper
    └─── helper_module  # a submodule
        ├── __init__.py
        └── submodule1.py  # some more helpers

Then, these modules and packages can be imported as a regular package, i.e., .. code-block:

from benchopt import safe_import_context

with safe_import_context() as import_ctx:
    from benchmark_utils import helper1
    from benchmark_utils.helper1 import func1
    from benchmark_utils.helper_module.submodule1 import func2

Caching pre-compilation and warmup effects#

For some solvers, such as solver relying on just-in-time compilation with numba or jax, the first iteration might be longer due to “warmup” effects. To avoid having such effect in the benchmark results, it is usually advised to call the solver once before running the benchmark. This should be implemented in the Solver.warm_up method, which is empty by default and called after the set_objective method. For solvers with sampling_strategy in {'tolerance', 'iteration'}, simply calling the Solver.run with a simple enough value is usually enough. For solvers with sampling_strategy set to 'callback', it is possible to call Solver.run_once, which will call the run method with a simple callback that does not compute the objective value and stops after n_iter calls to callback (default to 1).

class Solver(BaseSolver):
    ...

    def warm_up(self):
        # Cache pre-compilation and other one-time setups that should
        # not be included in the benchmark timing.
        self.run(1)  # For sampling_strategy == 'iteration' | 'tolerance'
        self.run_once()  # For sampling_strategy == 'callback'

Run a benchmark using a Python script#

Another way to run a benchmark is via a Python script. Typical use-cases of that are

Automating the run of several benchmarks
Using vscode debugger where the python script serves as an entry point to benchopt internals

The following script illustrate running the benchmark Lasso. It assume that the python script is located at the same level as the benchmark folder.

from benchopt import run_benchmark
from benchopt.benchmark import Benchmark

# load benchmark
BENCHMARK_PATH = "./"
benchmark = Benchmark(BENCHMARK_PATH)

# run benchmark
run_benchmark(
    benchmark,
    solver_names=[
        "skglm",
        "celer",
        "python-pgd[use_acceleration=True]",
    ],
    dataset_names=[
        "leukemia",
        "simulated[n_samples=100,n_features=20]"
    ],
)

Note

Learn more about the different parameters supported by run_benchmark function on API references.

Advanced usage

Contents

Advanced usage#

Running the benchmark on a SLURM cluster#

Skipping a solver for a given problem#

Changing the strategy to grow the computational budget (stop_val)#

Reusing some code in a benchmark#

Caching pre-compilation and warmup effects#

Run a benchmark using a Python script#

Changing the strategy to grow the computational budget (`stop_val`)#