Benchopt#
—Making your ML and optimization benchmarks simple and open—
Benchopt is a benchmarking suite tailored for machine learning and optimization. It makes it simple to run, publish and replicate benchmarks. The command line interface allows running various solvers on benchmark problems, providing convergence curves for iterative methods or performance comparisons for ML estimators.
Reproducing an existing benchmark should be as easy as running the following commands:
A benchmark comparing various optimizers on training NanoGPT models.
git clone https://github.com/benchopt/benchmark_nanogpt.git
benchopt run benchmark_nanogpt
which will produce an interactive HTML report to visualize the results.
A benchmark comparing various methods for unsupervised domain adaptation.
git clone https://github.com/scikit-adaptation/skada-bench.git
benchopt run skada-bench --config configs/Simulated.yml --no-plot
which will produce a parquet file with the results that can be visualized
using instruction on the README.md of the scikit-adaptation/skada-bench.
A minimal benchmark comparing various solvers on a toy problem.
benchopt run examples/minimal_benchmark
which will produce an interactive HTML report to visualize the results.
These different tabs illustrate the diversity of benchmarks that can be built with benchopt, from deep learning optimization to more classical machine learning tasks.
Tip
Want to create a new benchmark with benchopt? Use our templates to get started: ML benchmarks | Optimization benchmarks
There are already many Available benchmarks that have been created using benchopt. Learn how to run them and how to construct your own with the following pages!
Get started
Install benchopt and run your first benchmark
Benchmark workflow
Write an ML or optimization benchmark from scratch, run it, visualize it, and publish it
Tutorials
Gallery of use-cases crafted by the benchopt community
User guide
Full documentation of benchopt API and CLI
Frequently asked questions (FAQ)#
Can I use benchopt for ML benchmarks (not just optimization)?
Yes! Despite its name, Benchopt supports both machine learning and optimization benchmarks. Most features in benchopt are common to both types of benchmarks. The main difference lies in how performance is evaluated:
Optimization benchmarks: Track iterative solvers’ convergence over time and iterations.
ML benchmarks: Compare estimators/models on prediction tasks (accuracy, F1, etc.).
Setting sampling_strategy = "run_once" for a solver or for the full
benchmark allows to only evaluate once to completion.
See Setting up an ML benchmark for a complete guide, or use our
ML template to get started quickly.
How to add my solver to an existing benchmark?
Visit the Add a solver to an existing benchmark tutorial for a step-by-step procedure to add a solver to an existing benchmark.
How can I write a benchmark?
Learn how to Write a benchmark, including creating an objective, a solver, and a dataset.
Also take a look at our template repository for Optimization and ML to easily start a new benchmark.
How are performance curves constructed and the solvers stopped?
One of benchopt’s goals is to evaluate the method’s performance with respect to its computational budget. Benchopt allows several strategies to vary the computational budget, that can be set on a per solver basis. It is also possible to set various stopping criterions to decide when to stop growing the computational budget, to avoid wasting resources. Visit the performance_curves page for more details.
How can I reuse code in a benchmark?
For some solvers and datasets, it is handy to share some operations or pre-processing steps. Benchopt allows to factorize this code by Reusing some code in a benchmark.
Can I run a benchmark in parallel?
Benchopt allows to run different benchmarked methods in parallel, either with joblib using -j 4 to run on multiple CPUs of a single machine or with more advanced distributed backend, detailed in Distributed run with Benchopt.
Join the community#
Join benchopt discord server and get in touch with the community!
Feel free to drop a message to get help with running/constructing benchmarks or (why not) discuss new features to be added and future development directions that benchopt should take.
Citing Benchopt#
Benchopt is a continuous effort to make reproducible and transparent ML and optimization benchmarks. Join this endeavor! If you use benchopt in a scientific publication, please cite
@inproceedings{benchopt,
author = {Moreau, Thomas and Massias, Mathurin and Gramfort, Alexandre
and Ablin, Pierre and Bannier, Pierre-Antoine
and Charlier, Benjamin and Dagréou, Mathieu and Dupré la Tour, Tom
and Durif, Ghislain and F. Dantas, Cassio and Klopfenstein, Quentin
and Larsson, Johan and Lai, En and Lefort, Tanguy
and Malézieux, Benoit and Moufad, Badr and T. Nguyen, Binh and Rakotomamonjy,
Alain and Ramzi, Zaccharie and Salmon, Joseph and Vaiter, Samuel},
title = {Benchopt: Reproducible, efficient and collaborative optimization benchmarks},
year = {2022},
booktitle = {NeurIPS},
url = {https://arxiv.org/abs/2206.13424}
}