This example shows how to add a Julia solver in a simple benchmark using
benchopt’s helpers to call Julia code from Python.
The benchmark objective is a simple minimization task:
\[\min_{\hat{X}} \; \mathrm{MSE}(X, \hat{X})\]
We define:
a Python Objective that evaluates MSE between X and X_hat;
a Python Dataset that generates a random matrix X;
two solvers:
Python-GD implemented in Python;
Julia-GD implemented in Julia and called through JuliaSolver.
At the end, we run the benchmark and display the comparison.
# Import example helpers to define the benchmark and# programmatically call the CLI.frombenchopt.helpers.run_examplesimportExampleBenchmarkfrombenchopt.helpers.run_examplesimportbenchopt_clifrombenchopt.helpers.run_examplesimportEXAMPLES_ROOT
First, we define the initial Python benchmark, based on the benchmark
examples/minimal_benchmark. It contains an objective.py file,
a simulated dataset and a full python solver based on gradient descent.
frombenchoptimportBaseObjectiveimportnumpyasnpclassObjective(BaseObjective):# Name of the Objective functionname='Quadratic'# The three methods below define the links between the Dataset,# the Objective and the Solver.defset_data(self,X):"""Set the data from a Dataset to compute the objective. The argument are the keys of the dictionary returned by ``Dataset.get_data``. """self.X=Xdefget_objective(self):"Returns a dict passed to ``Solver.set_objective`` method."returndict(X=self.X)defevaluate_result(self,X_hat):"""Compute the objective value(s) given the output of a solver. The arguments are the keys in the dictionary returned by ``Solver.get_result``. """returndict(value=np.linalg.norm(self.X-X_hat))defget_one_result(self):"""Return one solution for which the objective can be evaluated. This function is mostly used for testing and debugging purposes. """returndict(X_hat=1)
frombenchoptimportBaseDatasetimportnumpyasnpclassDataset(BaseDataset):# Name of the Dataset, used to select it in the CLIname='simulated'# ``get_data()`` is the only method a dataset should implement.defget_data(self):"""Load the data for this Dataset. Usually, the data are either loaded from disk as arrays or Tensors, or a dataset/dataloader object is used to allow the models to load the data in more flexible forms (e.g. with mini-batches). The dictionary's keys are the kwargs passed to ``Objective.set_data``. """returndict(X=np.random.randn(10,2))
frombenchoptimportBaseSolverimportnumpyasnpclassSolver(BaseSolver):# Name of the Solver, used to select it in the CLIname='gd'# By default, benchopt will evaluate the result of a method after various# number of iterations. Setting the sampling_strategy controls how this is# done. Here, we use a callback function that is called at each iteration.sampling_strategy='callback'# Parameters of the method, that will be tested by the benchmark.# Each parameter ``param_name`` will be accessible as ``self.param_name``.parameters={'lr':[1e-3,1e-2]}# The three methods below define the necessary methods for the Solver, to# get the info from the Objective, to run the method and to return a# result that can be evaluated by the Objective.defset_objective(self,X):"""Set the info from a Objective, to run the method. This method is also typically used to adapt the solver's parameters to the data (e.g. scaling) or to initialize the algorithm. The kwargs are the keys of the dictionary returned by ``Objective.get_objective``. """self.X=Xself.X_hat=np.zeros_like(X)defrun(self,cb):"""Run the actual method to benchmark. Here, as we use a "callback", we need to call it at each iteration to evaluate the result as the procedure progresses. The callback implements a stopping mechanism, based on the number of iterations, the time and the evoluation of the performances. """whilecb():self.X_hat=self.X_hat-self.lr*(self.X_hat-self.X)defget_result(self):"""Format the output of the method to be evaluated in the Objective. Returns a dict which is passed to ``Objective.evaluate_result`` method. """return{'X_hat':self.X_hat}
#loaded from minimal_benchmark/config.ymlplot_configs:Subopt. (log):plot_kind:objective_curvescale:loglogRuntimes:plot_kind:bar_chart
"
We can now add solver in Julia with the same algorithm.
To do this, we create a new file julia_gd.py that defines a solver based
on the JuliaSolver class. This class provides helpers to call Julia code
from Python, and to define the dependencies of the solver.
The Julia code is defined in a separate file julia_gd.jl, that is loaded
and called from the Python solver.
frompathlibimportPathfrombenchopt.helpers.juliaimportJuliaSolverfrombenchopt.helpers.juliaimportget_jl_interpreterJULIA_SOLVER_FILE=str(Path(__file__).with_suffix('.jl'))classSolver(JuliaSolver):name="Julia-GD"sampling_strategy="iteration"parameters={"lr":[1e-3,1e-2]}requirements=["https://repo.prefix.dev/julia-forge::julia","pip::julia",]defset_objective(self,X):self.X=Xjl=get_jl_interpreter()self.julia_gd=jl.include(JULIA_SOLVER_FILE)defwarm_up(self):# Make sure we don't account for the Julia loading time in the# first iteration of the benchmark.self.julia_gd(self.X,self.lr,20)defrun(self,n_iter):# Here we cannot call a python callback, so we call iteratively# the solver with a growing number of iterations.self.X_hat=self.julia_gd(self.X,self.lr,n_iter)defget_result(self):returndict(X_hat=self.X_hat)
In order to load the Julia interpreter, we use get_jl_interpreter. This
function returns a Julia object from PyJulia, that can be used to
interact with Julia. In particular, we can use the include method to load
a Julia file and retrieve the functions defined in it as attributes of the
returned object.
Note that the Julia solver cannot call a Python callback to report
intermediate results, so we call iteratively the Julia solver with a growing
number of iterations to be able to report the curve of the convergence.
To be able to run this benchmark, we need to install its dependencies. We can
do this with benchoptinstall, using with the -s option which allow
to select only this solver if multiple solvers are present. If Julia is not
available in your environment, this command will use conda to install it.
Benchopt is running!
Benchopt called using profiling
Loading objective, datasets and solvers... done.
simulated
|--Quadratic
No seed was specified. Selected global seed: 0
|--gd[lr=0.001]:done (not enough run) |--gd[lr=0.01]:done (not enough run)
/home/circleci/miniconda/envs/benchopt-docs/lib/python3.12/site-packages/julia
/core.py:715: FutureWarning: Accessing `Julia().<name>` to obtain Julia
objects is deprecated. Use `from julia import Main; Main.<name>` or `jl =
Julia(); jl.eval('<name>')`.
warnings.warn(
|--Julia-GD[lr=0.001]:done (not enough run) |--Julia-GD[lr=0.01]:done (not enough run)Saving result in: temp_benchmark__sdtdi20/julia_solver/outputs/benchopt_run_2026-05-28_01h02m08.parquet
Rendering benchmark results...
Processing
temp_benchmark__sdtdi20/julia_solver/outputs/benchopt_run_2026-05-28_01h02m08.
parquet
done
Writing results to
temp_benchmark__sdtdi20/julia_solver/outputs/julia_solver_benchopt_run_2026-05
-28_01h02m08.html
Writing julia_solver index to
temp_benchmark__sdtdi20/julia_solver/outputs/julia_solver.html
Here, you see that the Julia solver is faster than the Python one.
You also notice that the first iteration seems to take much longer than the
other, hinting to a loading time for the solver.
Total running time of the script: (4 minutes 6.549 seconds)
