benchopt.BaseSolver#
- class benchopt.BaseSolver(**parameters)#
A base class for solver wrappers in Benchopt.
Solvers that derive from this class should implement three methods:
set_objective(self, **objective_parameters): prepares the solver to be called on a given problem.**objective_parametersis the output ofObjective.get_objectivefrom the benchmark objective. In particular, this method should dumps the parameter to compute the objective function in a file for command line solvers to reduce the impact of dumping the data to the disk in the benchmark.run(self, n_iter/tolerance/cb): performs the computation for the previously given objective function, after a call toset_objective. This method is the one timed in the benchmark and should not perform any operation unrelated to the optimization procedure.get_result(self): returns all parameters of interest, as a dict. The output is passed toObjective.evaluate_result.
Optionaly, the
Solvercan implement the following methods to change its behavior:skip(self, objective, dataset): decide if the solver is compatible with the objective and the dataset of the run. It returns the reason to skip, displayed in the CLI output, or None to run.get_next(stop_val): Return the next iteration where the result will be evaluated. This is only necessary when sampling_strategy is set to ‘iteration’ or ‘tolerance’ and the default logarithmic spacing is not desired.warm_up(): User specified warm up step, called once before the runs. The time it takes to run this function is not taken into account. The functionSolver.run_oncecan be used here for solvers that require jit compilation.pre_run_hook(stop_val): Hook to run pre-run operations, that are not timed in the benchmark. This is mostly necessary to cache stop_val dependent computations, for instance injaxwith different number of iterations in a for loop.
The
Solverclass also defines class attributes to specify how the benchmark curve should be sampled:sampling_strategy: defines how the benchmark curve should be sampled. It should be one of the following strings: ‘iteration’, ‘tolerance’, ‘callback’ or ‘run_once’:'iteration': call the run method with max_iter number increasing logarithmically to get more an more precise points.'tolerance': call the run method with tolerance decreasing logarithmically to get more and more precise points.'callback': a callable that should be called after each iteration or epoch. This callable periodically runsObjective.evaluate_resultand returns False when the solver should stop.'run_once': call the run method once to get a single point. This is typically used for ML benchmarks.
stopping_criterion: an instance ofStoppingCriterionthat defines when the solver should stop. If not set, a default stopping criterion is used depending on thesampling_strategy. See When are the solvers stopped? for available options.
Note that default values for these attributes can be set at the
Objectivelevel so that all solvers in a benchmark share the same default behavior. Typically, for ML benchmarks, all solvers can be run only once by settingsampling_strategy = 'run_once'in the benchmark’sObjective. More details on how the curves are sampled can be found in the Evaluating an iterative method user guide.- abstractmethod get_result()#
Return the parameters computed by the previous run.
The parameters should be returned as a dictionary.
- Returns:
- parametersdictionary
All quantities of interest to evaluate the objective.
- pre_run_hook(stop_val)#
Hook to run pre-run operations.
This is mostly necessary to cache stop_val dependent computations, for instance in
jaxwith different number of iterations in a for loop.- Parameters:
- stop_valint | float | callable
Value for the stopping criterion of the solver for. It allows to sample the time/accuracy curve in the benchmark. If it is a callable, then it should act as a callback. This callback should be called once for each iteration with argument the current iterate parameters. The callback returns False when the computations should stop.
- abstractmethod run(stop_val)#
Call the solver with the given stop_val.
This function should not return the parameters which will be retrieved by a subsequent call to get_result.
If sampling_strategy is set to “callback”, then run should call the callback at each iteration. The callback will compute the time, the objective function and store relevant quantities for BenchOpt. Else, the stop_val parameter should be specified.
- Parameters:
- stop_valint | float | callable
Value for the stopping criterion of the solver for. It allows to sample the time/accuracy curve in the benchmark. If it is a callable, then it should act as a callback. This callback should be called once for each iteration with argument the current iterate parameters. The callback returns False when the computations should stop.
- run_once(stop_val=1)#
Run the solver once, to cache warmup times (e.g. pre-compilations).
This function is intended to be called in
Solver.warm_upmethod to avoid taking into account a solver’s warmup costs.- Parameters:
- stop_valint or float, (default: 1)
If
sampling_strategyis ‘iteration’, this should be an integer corresponding to the number of iterations the solver is run for. If it is ‘callback’, it is an integer corresponding to the number of times the callback is called. If it is ‘tolerance’, it is a float which can be passed to call the solver on an easy to solve problem.
- abstractmethod set_objective(**objective_dict)#
Prepare the objective for the solver.
- Parameters:
- **objective_parametersdict
Dictionary obtained as the output of the method
get_objectivefrom the benchmarkObjective.
- skip(objective, dataset)#
Hook to decide if the
Solveris compatible with the objective.It is called before launching the runs. Checks on the parameters (e.g.
objective.reg,dataset.n_samples) are free. Callingobjective.get_objective()ordataset.get_data()loads the data on the machine runningbenchopt run.- Parameters:
- objectivebenchopt.BaseObjective
The objective of the run.
- datasetbenchopt.BaseDataset
The dataset of the run.
- Returns:
- reasonstr | None
The reason to skip this solver for this objective, displayed in the output, or None to run it. Any other value raises an error.
- warm_up()#
User specified warm up step, called once before the runs.
The time it takes to run this function is not taken into account. The function Solver.run_once can be used here for solvers that require jit compilation.