Note
Go to the end to download the full example code.
Create and run an R solver benchmark#
This example shows how to add an R solver in a simple benchmark using benchopt’s helpers to call R code from Python.
The benchmark objective is a simple minimization task:
\[\min_{\hat{X}} \; \mathrm{MSE}(X, \hat{X})\]
We define:
a Python
Objectivethat evaluates MSE betweenXandX_hat;a Python
Datasetthat generates a random matrixX;two solvers:
Python-GDimplemented in Python;R-PGDimplemented in R and called throughrpy2.
At the end, we run the benchmark and display the comparison.
# Import example helpers to define the benchmark and
# programmatically call the CLI.
from benchopt.helpers.run_examples import ExampleBenchmark
from benchopt.helpers.run_examples import benchopt_cli
from benchopt.helpers.run_examples import EXAMPLES_ROOT
First, we define the initial Python benchmark, based on the benchmark
examples/minimal_benchmark. It contains an objective.py file,
a simulated dataset and a full python solver based on gradient descent.
benchmark = ExampleBenchmark(
base="minimal_benchmark", name="r_solver",
ignore=["custom_plot.py", "example_config.yml"]
)
benchmark
"from benchopt import BaseObjective import numpy as np class Objective(BaseObjective): # Name of the Objective function name = 'Quadratic' # The three methods below define the links between the Dataset, # the Objective and the Solver. def set_data(self, X): """Set the data from a Dataset to compute the objective. The argument are the keys of the dictionary returned by ``Dataset.get_data``. """ self.X = X def get_objective(self): "Returns a dict passed to ``Solver.set_objective`` method." return dict(X=self.X) def evaluate_result(self, X_hat): """Compute the objective value(s) given the output of a solver. The arguments are the keys in the dictionary returned by ``Solver.get_result``. """ return dict(value=np.linalg.norm(self.X - X_hat)) def get_one_result(self): """Return one solution for which the objective can be evaluated. This function is mostly used for testing and debugging purposes. """ return dict(X_hat=1)from benchopt import BaseDataset import numpy as np class Dataset(BaseDataset): # Name of the Dataset, used to select it in the CLI name = 'simulated' # ``get_data()`` is the only method a dataset should implement. def get_data(self): """Load the data for this Dataset. Usually, the data are either loaded from disk as arrays or Tensors, or a dataset/dataloader object is used to allow the models to load the data in more flexible forms (e.g. with mini-batches). The dictionary's keys are the kwargs passed to ``Objective.set_data``. """ return dict(X=np.random.randn(10, 2))from benchopt import BaseSolver import numpy as np class Solver(BaseSolver): # Name of the Solver, used to select it in the CLI name = 'gd' # By default, benchopt will evaluate the result of a method after various # number of iterations. Setting the sampling_strategy controls how this is # done. Here, we use a callback function that is called at each iteration. sampling_strategy = 'callback' # Parameters of the method, that will be tested by the benchmark. # Each parameter ``param_name`` will be accessible as ``self.param_name``. parameters = {'lr': [1e-3, 1e-2]} # The three methods below define the necessary methods for the Solver, to # get the info from the Objective, to run the method and to return a # result that can be evaluated by the Objective. def set_objective(self, X): """Set the info from a Objective, to run the method. This method is also typically used to adapt the solver's parameters to the data (e.g. scaling) or to initialize the algorithm. The kwargs are the keys of the dictionary returned by ``Objective.get_objective``. """ self.X = X self.X_hat = np.zeros_like(X) def run(self, cb): """Run the actual method to benchmark. Here, as we use a "callback", we need to call it at each iteration to evaluate the result as the procedure progresses. The callback implements a stopping mechanism, based on the number of iterations, the time and the evoluation of the performances. """ while cb(): self.X_hat = self.X_hat - self.lr * (self.X_hat - self.X) def get_result(self): """Format the output of the method to be evaluated in the Objective. Returns a dict which is passed to ``Objective.evaluate_result`` method. """ return {'X_hat': self.X_hat}#loaded from minimal_benchmark/config.yml plot_configs: Subopt. (log): plot_kind: objective_curve scale: loglog Runtimes: plot_kind: bar_chart
We can now add a solver in R with the same algorithm.
To do this, we create a new file r_pgd.py that defines a solver calling
an R function via benchopt.helpers.r_lang and rpy2.
The R code is defined in a separate file r_pgd.R, loaded from Python.
R_SOLVER = EXAMPLES_ROOT / "language_solvers" / "r_pgd.py"
R_SOLVER_PY = R_SOLVER.read_text(encoding="utf-8")
R_SOLVER_R = R_SOLVER.with_suffix(".R").read_text(encoding="utf-8")
benchmark.update(
solvers={"r_pgd.py": R_SOLVER_PY, "r_pgd.R": R_SOLVER_R},
)
"We now update the following files:
from pathlib import Path from benchopt import BaseSolver import numpy as np # Import helpers from rpy2 and benchopt.helpers.r_lang from benchopt.helpers.r_lang import import_func_from_r_file, converter_ctx # Import R function defined in r_pgd.R so they can be retrieved as python # functions using `func = robjects.r['FUNC_NAME']` R_FILE = str(Path(__file__).with_suffix('.R')) class Solver(BaseSolver): name = "R-PGD" install_cmd = 'conda' requirements = ['r-base', 'rpy2'] sampling_strategy = 'iteration' parameters = {'lr': [1e-3, 1e-2]} def set_objective(self, X): self.X = X robjects = import_func_from_r_file(R_FILE) self.r_gd = robjects.r['gradient_descent'] def run(self, n_iter): with converter_ctx(): coefs = self.r_gd( self.X, self.lr, n_iter=n_iter ) self.X_hat = np.asarray(coefs) def get_result(self): return {'X_hat': self.X_hat}##' Functions used in GD algorithm ##' ##' @title Functions used in GD algorithm ##' @author Thomas Moreau ##' @export # Main algorithm gradient_descent <- function(X, lr, n_iter) { # --------- Initialize parameter --------- p <- ncol(X) parameters <- X * 0 # --------- Run GD for n_iter iterations --------- for (i in 1:n_iter) { # Compute the gradient grad <- (parameters - X) # # Update the parameters parameters <- parameters - lr * grad } return(parameters) }
To run this benchmark, we need to install solver dependencies.
We use benchopt install with -s to select only this solver.
If R is not available in your environment, this command can install it
through conda using the solver requirements.
benchopt_cli(f"install {benchmark.benchmark_dir} -s r-pgd")
$ benchopt install temp_benchmark_7tddtybe/r_solver -s r-pgd
Installing 'r_solver' requirements
# Install
Collecting packages:
- Quadratic: already available ✓
- R-PGD: collected ✓
done
Installing required packages for:
- R-PGD
...Retrieving notices: - done
Channels:
- conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): | / - \ | / - \ | / - \ done
Solving environment: / done
gcc_impl_linux-64-15 | 77.4 MB | | 0%
sysroot_linux-64-2.3 | 40.1 MB | | 0%
r-base-4.5.3 | 26.1 MB | | 0%
libstdcxx-devel_linu | 19.8 MB | | 0%
gfortran_impl_linux- | 19.1 MB | | 0%
gxx_impl_linux-64-15 | 15.6 MB | | 0%
libsanitizer-15.2.0 | 7.6 MB | | 0%
libopenblas-0.3.33 | 5.7 MB | | 0%
libglib-2.88.1 | 4.5 MB | | 0%
binutils_impl_linux- | 3.5 MB | | 0%
gsl-2.7 | 3.2 MB | | 0%
libgcc-devel_linux-6 | 3.0 MB | | 0%
libgfortran5-15.2.0 | 2.4 MB | | 0%
harfbuzz-14.2.0 | 2.2 MB | | 0%
rpy2-3.6.7 | 1.8 MB | | 0%
font-ttf-ubuntu-0.83 | 1.5 MB | | 0%
kernel-headers_linux | 1.5 MB | | 0%
pcre2-10.47 | 1.2 MB | | 0%
cairo-1.18.4 | 966 KB | | 0%
gcc_impl_linux-64-15 | 77.4 MB | 5 | 5% [A
sysroot_linux-64-2.3 | 40.1 MB | 3 | 4%
r-base-4.5.3 | 26.1 MB | #2 | 13%
libstdcxx-devel_linu | 19.8 MB | 3 | 4%
gcc_impl_linux-64-15 | 77.4 MB | #3 | 14%
sysroot_linux-64-2.3 | 40.1 MB | #9 | 19%
r-base-4.5.3 | 26.1 MB | ###8 | 39%
libstdcxx-devel_linu | 19.8 MB | ###4 | 34%
gcc_impl_linux-64-15 | 77.4 MB | ##2 | 23%
sysroot_linux-64-2.3 | 40.1 MB | ###4 | 35%
r-base-4.5.3 | 26.1 MB | ######5 | 65%
libstdcxx-devel_linu | 19.8 MB | ######6 | 66%
gcc_impl_linux-64-15 | 77.4 MB | ###2 | 33%
sysroot_linux-64-2.3 | 40.1 MB | #####5 | 55%
gfortran_impl_linux- | 19.1 MB | ########## | 100%
sysroot_linux-64-2.3 | 40.1 MB | ########8 | 88%
gcc_impl_linux-64-15 | 77.4 MB | ####1 | 42%
gcc_impl_linux-64-15 | 77.4 MB | ####9 | 50%
libstdcxx-devel_linu | 19.8 MB | ########## | 100%
libstdcxx-devel_linu | 19.8 MB | ########## | 100%
r-base-4.5.3 | 26.1 MB | ########## | 100%
r-base-4.5.3 | 26.1 MB | ########## | 100%
gcc_impl_linux-64-15 | 77.4 MB | #####8 | 58%
libopenblas-0.3.33 | 5.7 MB | | 0%
libsanitizer-15.2.0 | 7.6 MB | ######1 | 61%
gcc_impl_linux-64-15 | 77.4 MB | #######4 | 74%
libopenblas-0.3.33 | 5.7 MB | ########## | 100%
libglib-2.88.1 | 4.5 MB | | 0%
libsanitizer-15.2.0 | 7.6 MB | ########## | 100%
libsanitizer-15.2.0 | 7.6 MB | ########## | 100%
gxx_impl_linux-64-15 | 15.6 MB | ########## | 100%
gxx_impl_linux-64-15 | 15.6 MB | ########## | 100%
gsl-2.7 | 3.2 MB | | 0%
gcc_impl_linux-64-15 | 77.4 MB | ########1 | 82%
libglib-2.88.1 | 4.5 MB | ########## | 100%
libglib-2.88.1 | 4.5 MB | ########## | 100%
gsl-2.7 | 3.2 MB | ########## | 100%
gsl-2.7 | 3.2 MB | ########## | 100%
libgfortran5-15.2.0 | 2.4 MB | | 1%
gcc_impl_linux-64-15 | 77.4 MB | ########8 | 88%
binutils_impl_linux- | 3.5 MB | ########## | 100%
binutils_impl_linux- | 3.5 MB | ########## | 100%
harfbuzz-14.2.0 | 2.2 MB | | 1%
libgfortran5-15.2.0 | 2.4 MB | ########## | 100%
gcc_impl_linux-64-15 | 77.4 MB | #########5 | 96%
libgcc-devel_linux-6 | 3.0 MB | ########## | 100%
libgcc-devel_linux-6 | 3.0 MB | ########## | 100%
font-ttf-ubuntu-0.83 | 1.5 MB | 1 | 1%
sysroot_linux-64-2.3 | 40.1 MB | ########## | 100%
harfbuzz-14.2.0 | 2.2 MB | ########## | 100%
harfbuzz-14.2.0 | 2.2 MB | ########## | 100%
rpy2-3.6.7 | 1.8 MB | ########## | 100%
font-ttf-ubuntu-0.83 | 1.5 MB | ########## | 100%
cairo-1.18.4 | 966 KB | 1 | 2%
kernel-headers_linux | 1.5 MB | 1 | 1%
pcre2-10.47 | 1.2 MB | 1 | 1%
... (more hidden) ...
cairo-1.18.4 | 966 KB | ########## | 100%
... (more hidden) ...
pcre2-10.47 | 1.2 MB | ########## | 100%
kernel-headers_linux | 1.5 MB | ########## | 100%
gfortran_impl_linux- | 19.1 MB | ########## | 100%
libopenblas-0.3.33 | 5.7 MB | ########## | 100%
gcc_impl_linux-64-15 | 77.4 MB | ########## | 100%
libstdcxx-devel_linu | 19.8 MB | ########## | 100%
r-base-4.5.3 | 26.1 MB | ########## | 100%
gxx_impl_linux-64-15 | 15.6 MB | ########## | 100%
libglib-2.88.1 | 4.5 MB | ########## | 100%
binutils_impl_linux- | 3.5 MB | ########## | 100%
libgfortran5-15.2.0 | 2.4 MB | ########## | 100%
libgfortran5-15.2.0 | 2.4 MB | ########## | 100%
libgcc-devel_linux-6 | 3.0 MB | ########## | 100%
harfbuzz-14.2.0 | 2.2 MB | ########## | 100%
gsl-2.7 | 3.2 MB | ########## | 100%
rpy2-3.6.7 | 1.8 MB | ########## | 100%
rpy2-3.6.7 | 1.8 MB | ########## | 100%
font-ttf-ubuntu-0.83 | 1.5 MB | ########## | 100%
font-ttf-ubuntu-0.83 | 1.5 MB | ########## | 100%
cairo-1.18.4 | 966 KB | ########## | 100%
cairo-1.18.4 | 966 KB | ########## | 100%
... (more hidden) ...
... (more hidden) ...
pcre2-10.47 | 1.2 MB | ########## | 100%
pcre2-10.47 | 1.2 MB | ########## | 100%
kernel-headers_linux | 1.5 MB | ########## | 100%
kernel-headers_linux | 1.5 MB | ########## | 100%
gcc_impl_linux-64-15 | 77.4 MB | ########## | 100%
A
done
Preparing transaction: \ | done
Verifying transaction: - \ | / - \ | / - \ | / - \ | / - \ | / - done
Executing transaction: | / - \ | / - \ | / - \ | / - \ | / - \ | / done
#
# To activate this environment, use
#
# $ conda activate benchopt-docs
#
# To deactivate an active environment, use
#
# $ conda deactivate
done
- Checking installed packages...Failed to import Solver from
temp_benchmark_7tddtybe/r_solver/solvers/r_pgd.py. Please fix the following
error to use this file with benchopt:
Traceback (most recent call last):
File "/home/circleci/project/benchopt/utils/dynamic_modules.py", line 101,
in _load_class_from_module
module = _get_module_from_file(module_filename, benchmark_dir)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/circleci/project/benchopt/utils/dynamic_modules.py", line 68, in
_get_module_from_file
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 999, in exec_module
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File
"/home/circleci/project/examples/temp_benchmark_7tddtybe/r_solver/solvers/r_pg
d.py", line 8, in <module>
from benchopt.helpers.r_lang import import_func_from_r_file, converter_ctx
File "/home/circleci/project/benchopt/helpers/r_lang.py", line 11, in
<module>
raise ImportError(
ImportError: rpy2 is not installed. Please make sure the solver requirements
are installed. If the requirements are missing, add `requirements = ['r-base',
'rpy2']` to your solver.
/home/circleci/project/benchopt/benchmark.py:674: UserWarning: Some solvers
were not successfully installed, and will thus be ignored. Use 'export
BENCHOPT_RAISE_INSTALL_ERROR=true' to stop at any installation failure and
print the traceback.
warnings.warn(
done (missing deps: {'R-PGD'})
Then, we can run the benchmark and show the comparison.
benchopt_cli(f"run {benchmark.benchmark_dir} -n 20 -r 4")
$ benchopt run temp_benchmark_7tddtybe/r_solver -n 20 -r 4
Benchopt is running!
Benchopt called using profiling
Loading objective, datasets and solvers... done.
simulated
|--Quadratic
No seed was specified. Selected global seed: 0
|--gd[lr=0.001]: done (not enough run)
|--gd[lr=0.01]: done (not enough run)
|--R-PGD[lr=0.001]: done (not enough run)
|--R-PGD[lr=0.01]: done (not enough run)
Saving result in:
temp_benchmark_7tddtybe/r_solver/outputs/benchopt_run_2026-05-28_00h58m02.parq
uet
Rendering benchmark results...
Processing
temp_benchmark_7tddtybe/r_solver/outputs/benchopt_run_2026-05-28_00h58m02.parq
uet
done
Writing results to
temp_benchmark_7tddtybe/r_solver/outputs/r_solver_benchopt_run_2026-05-28_00h5
8m02.html
Writing r_solver index to
temp_benchmark_7tddtybe/r_solver/outputs/r_solver.html
Here, you should see that the R solver and Python solver obtain similar convergence profiles, with runtime differences depending on your setup.
Total running time of the script: (1 minutes 27.996 seconds)