benchopt.BaseSolver

class benchopt.BaseSolver(**parameters)

A base class for solver wrappers in Benchopt.

Solvers that derive from this class should implement three methods:

• set_objective(self, **objective_parameters): prepares the solver to be called on a given problem. **objective_parameters is the output of Objective.get_objective from the benchmark objective. In particular, this method should dumps the parameter to compute the objective function in a file for command line solvers to reduce the impact of dumping the data to the disk in the benchmark.

• run(self, n_iter/tolerance): performs the computation for the previously given objective function, after a call to set_objective. This method is the one timed in the benchmark and should not perform any operation unrelated to the optimization procedure.

• get_result(self): returns all parameters of interest, as a dict. The output is passed to Objective.evaluate_result.

Note that four sampling_strategy can be used to construct the benchmark curve:

• 'iteration': call the run method with max_iter number increasing logarithmically to get more an more precise points.

• 'tolerance': call the run method with tolerance decreasing logarithmically to get more and more precise points.

• 'callback': a callable that should be called after each iteration or epoch. This callable periodically calls the objective’s compute and returns False when the solver should stop.

• 'run_once': call the run method once to get a single point. This is typically used for ML benchmarks.

abstract get_result()

Return the parameters computed by the previous run.

The parameters should be returned as a dictionary.

Returns:

parametersdictionary

All quantities of interest to evaluate the objective.

pre_run_hook(stop_val)

Hook to run pre-run operations.

This is mostly necessary to cache stop_val dependent computations, for instance in jax with different number of iterations in a for loop.

Parameters:

stop_valint | float | callable

Value for the stopping criterion of the solver for. It allows to sample the time/accuracy curve in the benchmark. If it is a callable, then it should act as a callback. This callback should be called once for each iteration with argument the current iterate parameters. The callback returns False when the computations should stop.

abstract run(stop_val)

Call the solver with the given stop_val.

This function should not return the parameters which will be retrieved by a subsequent call to get_result.

If sampling_strategy is set to “callback”, then run should call the callback at each iteration. The callback will compute the time, the objective function and store relevant quantities for BenchOpt. Else, the stop_val parameter should be specified.

Parameters:

stop_valint | float | callable

Value for the stopping criterion of the solver for. It allows to sample the time/accuracy curve in the benchmark. If it is a callable, then it should act as a callback. This callback should be called once for each iteration with argument the current iterate parameters. The callback returns False when the computations should stop.

run_once(stop_val=1)

Run the solver once, to cache warmup times (e.g. pre-compilations).

This function is intended to be called in Solver.warm_up method to avoid taking into account a solver’s warmup costs.

Parameters:

stop_valint or float, (default: 1)

If sampling_strategy is ‘iteration’, this should be an integer corresponding to the number of iterations the solver is run for. If it is ‘callback’, it is an integer corresponding to the number of times the callback is called. If it is ‘tolerance’, it is a float which can be passed to call the solver on an easy to solve problem.

abstract set_objective(**objective_dict)

Prepare the objective for the solver.

Parameters:

**objective_parametersdict

Dictionary obtained as the output of the method get_objective from the benchmark Objective.

skip(**objective_dict)

Hook to decide if the Solver is compatible with the objective.

Parameters:

**objective_parametersdict

Dictionary obtained as the output of the method get_objective from the benchmark Objective.

Returns:

skipbool

Whether this solver should be skipped or not for this objective.

reasonstr | None

The reason why it should be skipped for display purposes. If skip is False, the reason should be None.

warm_up()

User specified warm up step, called once before the runs.

The time it takes to run this function is not taken into account. The function Solver.run_once can be used here for solvers that require jit compilation.