Add a solver to an existing benchmark#
This tutorial shows how to add a new solver to a benchmark.
We illustrate the process on the Ridge regression benchmark by implementing a solver based on the scikit-learn Ridge estimator.
Hint
If not yet done, you can review the get started page to learn how to install benchopt and download an existing benchmark.
Before the implementation#
First, create a mysolver.py file in the solvers/ directory and put inside it the following content:
from benchopt import BaseSolver
class Solver(BaseSolver):
name = 'mysolver'
As you can see, a solver is a Python class, Solver, that inherits from benchopt.BaseSolver and is declared in a standalone Python file in the benchmark’s solvers/ folder.
The attribute name does not have to match the file name, but it makes it easier to locate the solver.
Doing the latter steps, our benchmark folder will resemble
benchmark_ridge/
├── objective.py # existing implementation of the Objective
├── datasets/
│ ├── ... # existing datasets
└── solvers/
├── mysolver.py # our newly added solver
├── ... # other solvers
Implementation#
Once the Python file is created, we can start the implementation by adding methods and attributes to the Solver class.
To do so, we follow the order that benchopt uses to call the methods during the benchmark’s run: first set_objective, then run and finally get_result.
Let’s go over them one by one.
Initializing the setup#
The first method we need to implement is set_objective.
It receives all the information about the dataset and objective parameters, that the solver will need to run.
This information is standardized in the objective.py file of the benchmark, through the Objective.get_objective method.
This method is part of the objective definition and has already been implemented when the benchmark was created; we do not need to write it.
In the Ridge benchmark, the relevant information is the following:
...
class Objective(BaseObjective):
...
def get_objective(self):
return dict(
X=self.X, y=self.y,
lmbd=self.lmbd,
fit_intercept=self.fit_intercept
)
...
We see that get_objective returns a dictionary with four keys: X, y, lmbd, and fit_intercept.
Therefore our set_objective must take them as input arguments.
Note
If you are working with another benchmark, check the definition of Objective.get_objective in objective.py to see which arguments must be passed to Solver.set_objective.
The set_objective method is meant to store references of dataset and objective parameters.
It is also used to initialize unchanging variables across the solver run.
In our case, we store X, y, lmbd, and fit_intercept to use them when we will actually run the solver.
We also use the method to instantiate a Ridge estimator that will be used to perform computation of the solution.
class Solver(BaseSolver):
...
def set_objective(self, X, y, lmbd, fit_intercept):
# store any info needed to run the solver as class attribute
self.X, self.y = X, y
self.lmbd = lmbd
# declare anything that will be used to run your solver
self.model = sklearn.linear_model.Ridge(
alpha=lmbd,
fit_intercept=fit_intercept
)
...
Defining the solver run procedure#
Next, we implement the run method, and declare the sampling_strategy attribute of the solver.
Together, they define how the performance curves of the solver will be constructed.
There are three possible choices for the sampling_strategy attribute: iteration, tolerance, and callback.
We show how to implement the run method for each one of them.
sampling_strategy = "iteration"
This sampling strategy is for solvers that are controlled by the maximum number of iterations they perform. In this case, benchopt treats the solver as a black box and observes its behavior for different number of iterations.
Therefore, the signature of the run method is run(self, n_iter) and its implementation resembles the snippet below.
class Solver(BaseSolver):
...
sampling_strategy = "iteration"
...
def run(self, n_iter):
# configure sklearn to run for n_iter
self.model.max_iter = n_iter
# make sure sklearn goes until n_iter
self.model.tol = 0
self.model.fit(self.X, self.y)
# store reference to the solution
self.beta = self.model.coef_
...
sampling_strategy = "tolerance"
Similar to iteration, this sampling strategy is used for solvers controlled by the tolerance on the optimization process.
In this case, the signature of the run method is run(self, tolerance); it would be implemented as follows.
class Solver(BaseSolver):
...
sampling_strategy = "tolerance"
...
def run(self, tolerance):
# configure sklearn to run for tolerance
self.model.tol = tolerance
# configure sklearn to run until tolerance is reached
self.model.max_iter = int(1e12)
self.model.fit(self.X, self.y)
# store reference to the solution
self.beta = self.model.coef_
...
sampling_strategy = "callback"
One may want to code the solver themselves rather than using a black-box one. In that case, all intermediate iterates are available, and one should use the callback sampling strategy.
Let’s say that we no longer implement the scikit-learn solver, but instead our own implementation of Gradient Descent. The following snippet shows how to use the callback strategy with a user-coded solver.
class Solver(BaseSolver):
...
sampling_strategy = "callback"
...
def run(self, callback):
X, y = self.X, self.y
n_features = self.X.shape[1]
# init vars
self.beta = np.zeros(n_features)
step = 1 / (np.linalg.norm(self.X, ord=2) ** 2 + self.lmbd)
while callback():
# do one iteration of the solver here:
grad = self.X.T @ (self.X @ self.beta - y) + self.lmbd * self.beta
self.beta -= step * grad
...
sampling_strategy = "run_once"
In some cases, we are not interested in convergence curves over time but only in the final result.
In this case, we can use the run_once sampling strategy.
class Solver(BaseSolver):
...
sampling_strategy = "run_once"
...
def run(self, _):
self.model.fit(self.X, self.y)
# store reference to the solution
self.beta = self.model.coef_
...
Note
The Performance curves page provides a complete guide on the way benchopt constructs performance curves, and on the different sampling strategies.
Getting the solver’s results#
Finally, we define a get_result method that is used to pass the solver’s result back to the objective.
It must return a dictionary whose keys are the input arguments of Objective.evaluate_result.
In the Ridge case the input of Objective.evaluate_result is beta, hence we return a dictionary with a single key, "beta".
class Solver(BaseSolver):
...
def get_result(self):
return {'beta': self.beta}
...
Note
If you are working with another benchmark, check the arguments of Objective.evaluate_result in objective.py to see which keys must be returned by Solver.get_result.
With these methods being implemented, your solver is now ready to be run!
Specifying the solver parameters#
If your solver has hyperparameters, you can specify them by adding an attribute parameters.
This attribute is a dictionary whose keys are the solver’s hyperparameters.
For example, if our solver has two hyperparameters, stepsize and momentum, we implement them as follows:
class Solver(BaseSolver):
...
parameters = {
'stepsize': [0.1, 0.5],
'momentum': [0.9, 0.95],
}
...
They are then available in the class methods as self.stepsize and self.momentum.
Note
When running the solver, benchopt will use all possible combinations of hyperparameter values. Hence, unless specified otherwise, our solver will be run 2 x 2 = 4 times.