Advanced usage#

This page intends to list some advanced functionality to make it easier to use the benchmark.

Skipping a configuration#

Some configurations of a benchmark cannot run. A dataset can have incompatible parameters, an objective might not apply to some datasets, and a solver might not support sparse data or scale to large datasets. Each component can skip them by implementing a skip method, which receives the components chosen before it:

  • skip() takes no argument and skips all the runs of the dataset.

  • skip() receives the dataset and skips the runs of all solvers.

  • skip() receives the objective and the dataset.

skip returns the reason to skip, displayed in the output, or None to run. Any other value or a failing skip is reported as an error, and its runs are not created.

The skips are checked once per configuration, before launching the runs. Checks on the parameters are free. Calling dataset.get_data() or objective.get_objective() loads the data of the first repetition on the machine running benchopt run, once per dataset. get_seed can use the component itself and the ones that skip receives. Other components are only set when group_by chooses them first.

For instance, with an objective parameter reg and a dataset giving X:

class Dataset(BaseDataset):
    ...
    def skip(self):
        if self.n_features > self.n_samples:
            return "more features than samples"

class Solver(BaseSolver):
    ...
    def skip(self, objective, dataset):
        from scipy import sparse

        if objective.reg == 0:
            return "solver does not work with reg=0"

        # Loads the data on the node running `benchopt run`.
        if sparse.issparse(dataset.get_data()["X"]):
            return "solver does not support sparse data X."

As skip runs on the machine running benchopt run, a check on the hardware should look at the machine that will run the solver. For a GPU, gpu_available() checks the local machine, or with the submitit backend, that the solver’s jobs request a GPU. For other checks, get_worker_config() returns the config of the workers that run the solver.

from benchopt.helpers.requires_gpu import gpu_available

class Solver(BaseSolver):
    ...
    def skip(self, objective, dataset):
        if not gpu_available(self):
            return "solver needs a GPU"

Note

In older versions, skip received the data as keyword arguments, e.g. Solver.skip(self, X, y, reg) with the output of Objective.get_objective, and Objective.skip(self, X, y) with the output of Dataset.get_data. It returned a (skip, reason) tuple. Both still work, with a FutureWarning, and are checked before launching the runs, but the old signature always loads the data. They will be removed in benchopt 1.12.

Advanced usage of Objective class#

The Objective class is used to evaluate each method’s result, with a call to the evaluate_result method. This method is called with the dictionary returned by the solver’s get_result method and should return a dictionary, whose keys/values are the names and values of the metrics. Each solver’s run constitutes a single row in the benchmark result dataframe. For more flexibility, the Objective class can also be used to produce multiple rows in the benchmark result dataframe at once, or to save the final results of a solver.

Producing multiple evaluations at once#

The evaluate_result method can be used to produce multiple rows in the benchmark result dataframe. This is done by returning a list of dictionaries instead of a single dictionary. Each dictionary in the list should be a valid result dictionary, i.e., it should not contain a name key and should have a key that matches the key_to_monitor for the solver (see When are the solvers stopped?).

This feature typically allows to store metrics for each sample in a test set or for each fold in a cross-validation setting, allowing to compute aggregated statistics at plotting time.

Saving Final Results of a Solver#

Using the save_final_results(**results) method of the objective function to retrieve the results to save. They are saved in outputs/final_results/ directory and reference is added in the benchmark .parquet file.

Reusing some code in a benchmark#

In some situations, multiple solvers need to have access to the same functions. As a benchmark is not structured as proper python packages but imported dynamically to avoid installation issues, we resort to a special way of importing modules and functions defined for a benchmark.

First, all code that needs to be imported should be placed under BENCHMARK_DIR/benchmark_utils/, as described here:

my_benchmark/
├── objective.py  # contains the definition of the objective
├── datasets/
├── solvers/
└── benchmark_utils/
    ├── __init__.py
    ├── helper1.py  # some helper
    └─── helper_module  # a submodule
        ├── __init__.py
        └── submodule1.py  # some more helpers

Then, these modules and packages can be imported as a regular package, i.e.,

from benchmark_utils import helper1
from benchmark_utils.helper1 import func1
from benchmark_utils.helper_module.submodule1 import func2

Caching pre-compilation and warmup effects#

For some solvers, such as solvers relying on just-in-time compilation with numba or jax, the first iteration might be longer due to “warmup” effects. To avoid having such effects in the benchmark results, it is usually advised to call the solver once before running the benchmark. This should be implemented in the Solver.warm_up method, which is empty by default and called after the set_objective method. For solvers with sampling_strategy in {'tolerance',  'iteration'}, simply calling the Solver.run with a simple enough value is usually enough. For solvers with sampling_strategy set to 'callback', it is possible to call Solver.run_once, which will call the run method with a simple callback that does not compute the objective value and stops after n_iter calls to callback (default to 1).

class Solver(BaseSolver):
    ...

    def warm_up(self):
        # Cache pre-compilation and other one-time setups that should
        # not be included in the benchmark timing.
        self.run(1)  # For sampling_strategy == 'iteration' | 'tolerance'
        self.run_once()  # For sampling_strategy == 'callback'

Data loading and memory#

Consecutive runs in a process reuse the same component instances. Benchopt skips the calls whose result is already there: Dataset.get_data is not called again for the same dataset, and Objective.set_data is not called again for the same dataset and objective, unless their seed changes with the repetition. Solver.set_objective and Solver.run are called for every run. The solver instance keeps its attributes between consecutive runs, so a solver can skip costly steps it already did, such as loading model weights. Benchopt does not do this for the solver. Its code must check whether the attribute is already set, for instance in set_objective.

Once a run uses another component, the previous instance is released: it goes back to its state after creation, so its data can be freed. The state of an objective depends on its dataset, and the state of a solver on its objective and dataset. Each process thus keeps at most one state per component, and loads each dataset once per block of consecutive runs using it.

The order of the runs therefore decides what is reused. By default, the runs are ordered by dataset, then objective, then solver, so each dataset is loaded once and set_data is called once per objective. With --group-by (see Grouping runs to reduce overhead), the listed axes come first. For instance, --group-by solver runs each solver on all datasets in a row, keeping the same solver instance, but reloads each dataset for every solver.

Attributes stored on an instance during a run do not survive a release. The warmup described above is kept, so it runs once per solver and process.