What’s new

What’s new#

Version 1.7 - in development#

Major change#

CLI#

  • Add --no-cache option to benchopt run, to disable caching. By Thomas Moreau (#800)

  • Add --gpu flag to benchopt install, to handle different requirements for GPU and CPU. By Mathurin Massias (#793)

  • Make it possible to run benchopt as python -m benchopt, to ease running in various environment and debugging. By Rémi Flamary (#685)

API#

  • Add slurm_params attribute to Solver to allow overriding the default SLURM config. By `Pierre-Louis Barbarant`_ (#805)

  • Support requirements being a dictionary with keys "gpu" and "cpu", for classes whose install differ on GPU and CPU. By Mathurin Massias (#793)

  • Change channel specification in requirements, replacing the split format with :: instead of :. This allow specifying URL channels. By Thomas Moreau (#758)

  • Add Objective, Solver and Dataset parameters as columns in the result DataFrame. The parameters’ names are respectively prefixed with p_obj_|p_solver_|p_dataset_ to avoid collapse between the different components. By Melvine Nargeot and Thomas Moreau (#703).

  • Objective can now return multiple evaluation at once, to store non-aggregated metrics. See Producing multiple evaluations at once. By Thomas Moreau (#778).

FIX#

Version 1.6 - 15/07/2024#

API#

  • Add a save_final_results method to Objective. If implemented it is run after the last solver iteration, to get desired outputs to be saved to file system. By Pierre-Antoine Comby (#722)

  • Add native way to do cross-validation in a benchmark with Objective.cv attribute that change split for each repetition. By Christopher Marouani and Thomas Moreau (#623).

  • Run-config files now support having parameters as nested dict, with potentially non-trivial structures (like dictionaries). By Thomas Moreau (#706).

  • Raise error when an invalid install_cmd is provided. By Jad Yehya (#714).

  • Add boxplot option to plot the benchmark results. By Melvine Nargeot (#714).

CLI#

  • Add --collect option to allow gathering results which are already in cache in a single parquet file. By Thomas Moreau (#710)

  • Add --download option in benchopt install to allow downloading the data when installing the benchmark. By Thomas Moreau (#718)

  • Add --no-timeout option in benchopt run to allow solvers to bypass timeout. By Célestin Eve (#725)

  • Remove support for deprecated .ini config files. All config files should now use the yaml format. By Thomas Moreau (#699)

FIX#

DOC#

Version 1.5.1 - 22/09/2023#

Bugfix release.

FIX#

  • Fix benchopt dependency specification to install benchopt in child env with extra [test]. By Thomas Moreau (#662).

Version 1.5 - 18/09/2023#

API#

  • Add a Objective.url attribute to specify the original repo of the benchmark. By Thomas Moreau (#621).

  • Deprecate passing in arguments to callback of when sampling_strategy='callback'. Now on, the results from the Solver are collected using get_result. By Thomas Moreau (#631).

  • Deprecate Objective.get_one_solution in favor of Objective.get_one_result for consistency with Objective.evaluate_result. By Thomas Moreau (#631).

  • Deprecate Objective.compute in favor of Objective.evaluate_result, for consistency with Solver.get_result. Like Dataset.get_data, Solver.get_result must now return a dictionary, which is unpacked as arguments to Objective.evaluate_result. By Mathurin Massias (#576).

  • Solver.support_sparse attribute is deprecated in favor of the use of Solver.skip, by Mathurin Massias (#614).

  • stopping_strategy attribute is replaced by sampling_strategy to clarify the concept, by Mathurin Massias (#585).

  • Add Solver.warm_up function for explicit warmup instructions, such as empty run for jitting. This function is called only once per solver. By Pierre Ablin (#602).

PLOT#

DOC#

Internals#

  • Add helper to store and retrieve metadata in parquet files. This will allow storing per-run plotting information. By Thomas Moreau (#637).

Version 1.4 - 03/07/2023#

CLI#

API#

  • The get_next method of BaseSolver is no longer static. By Badr Moufad (#566)

  • Add SingleRunCriterion to run a solver only once. This can be used for benchmarking methods where we are interested in objective value at convergence. By Thomas Moreau (#511)

  • Add run_once() helper to easily warmup solvers with callback. By Thomas Moreau (#511)

  • Add pre_run_hook() hook to ignore cost that cannot be cached globally for a solver. By Thomas Moreau (#525)

  • Remove deprecated Objective.to_dict, safe_import_context.import_from. Force implementation of get_one_solution(). By Thomas Moreau (#569)

PLOT#

  • Add a tooltip beside to show description of objective. Description is provided as docstring of the BaseObjective class. By Badr Moufad (#556)

  • Show solver description when hovering over solvers. Description is provided as docstring of the BaseSolver class. By Badr Moufad (#543)

  • Enable visualizing the objective as function of stopping_criterion: time, iteration, or tolerance. By Badr Moufad (#479)

  • Add button to share and set specific views on the plot. For now, the view needs to be defined manually in the benchmark config file but an export button will be added in follow up PRs. By Tanguy Lefort, Amélie Vernay and Thomas Moreau (#552).

FIX#

  • Do not fail and raise a warning when safe_import_context is not named import_ctx. By Mathurin Massias (#524)

Version 1.3.1 - 01/12/2022#

Bug fix release

FIX#

  • Typo in README and doc

  • Barchart solver color to be the same as the other plots.

  • Warning for deprecation not using the right class

  • Plot quantile incorrect display

Version 1.3 - 21/11/2022#

CLI#

  • Add support for custom parameters in CLI for objectives, datasets, and solvers, through the syntax -s solver_name[parameter=value]. See the CLI documentation for more details on the syntax. By Tom Dupré la Tour (#362).

  • Add --slurm option in benchopt run to allow running the benchmark on a SLURM cluster. See the Running the benchmark on a SLURM cluster for more details on the config. By Thomas Moreau (#407)

  • Add benchopt archive to create a tar.gz archive with the benchmark’s files for sharing with others or as supplementary materials for papers. By Thomas Moreau (#408).

  • Now the result data are saved in the Parquet format. The use of CSV files is deprecated. By Melvine Nargeot (#433).

  • Change the default number of repetitions to 1. By Benoît Malézieux (#457).

API#

Version 1.2 - 06/05/2022#

Changelog#

  • New benchopt info command to display information about solvers and datasets of a benchmark, by Ghislain Durif (#140).

  • New --profile option to the run command in order to profile with the line-profiler package all functions decorated with benchopt.utils.profile(), by Alexandre Gramfort (#186).

  • Replace SufficientDescentCriterion by SufficientProgressCriterion, which measures progress relative to the best attained value instead of the previous one, by Thomas Moreau (#176)

  • Now all values returned by Objective.compute are included in reports, by Thomas Moreau and Alexandre Gramfort (#200).

  • New --n-jobs, -j option to run the benchmark in parallel with joblib, by Thomas Moreau (#265).

API#

  • When returning a dictionary, Objective.compute should at least include value key instead of objective_value, by Thomas Moreau and Alexandre Gramfort (#200).

  • stop_strategy attribute is replaced by stopping_strategy to harmonize with stopping_criterion, by Benoît Malézieux (#274).

  • Add import_from method in safe_import_context to allow importing common files and packages without installation from BENCHMARK_DIR/utils, by Thomas Moreau (#286).

  • Add X_density argument to datasets.make_correlated_data to simulate sparse design matrices, by Mathurin Massias (#289).

  • Dataset.get_data should now return a dictionary and not a tuple. A point for testing should be returned by a dedicated method Objective.get_one_solution, by Thomas Moreau (#345).

CLI#

Version 1.1 - 22-04-2021#

Changelog#

API#

  • Objective.compute can now return a dictionary with multiple outputs to monitor several metrics at once, by Thomas Moreau (#84).

  • Solver.skip can now be used to skip objectives that are incompatible for the Solver, by Thomas Moreau (#113).

  • Solver can now use stop_strategy='callback' to allow for single call curve construction, by Tanguy Lefort and Thomas Moreau (#137).

  • Add StoppingCriterion to reliably and flexibly assess a solver convergence (cvg). For now, only SufficientDescentCriterion is implemented but better API to set criterion per benchmark should be implemented in a future release, by Thomas Moreau (#151)

CLI#

  • Add --version option for benchopt, by Thomas Moreau (#83).

  • Add --pdb option for benchopt run to open debugger on error and help benchmark debugging, by Thomas Moreau (#86).

  • Change default run to local mode. Can call a run in a dedicated env with option --env or --env-name ENV_NAME to specify the env, by Thomas Moreau (#94).

  • Add benchopt publish command to push benchmark results to GitHub, by Thomas Moreau (#110).

  • Add benchopt clean command to remove cached files and output files locally, by Thomas Moreau (#128).

  • Add benchopt config command to allow easy configuration of benchopt using the CLI, by Thomas Moreau (#128).

  • Add benchopt install command to install benchmark requirements (not done in benchopt run anymore) by Ghislain Durif (#135).

  • Add benchopt info command to print information about a benchmark (including solvers, datasets, dependencies, etc.) by Ghislain Durif (#140).

BUG#

  • Throw a warning when benchopt version in conda env does not match the one of calling benchopt, by Thomas Moreau (#83).

  • Fix Lapack issue with R code, by Tanguy Lefort (#97).

DOC#

The committer list for this release is the following:

Version 1.0 - 2020-09-25#

Release highlights#

  • Provide a command line interface for benchmarking optimisation algorithm implementations:

    • benchopt run to run the benchmarks

    • benchopt plot to display the results

    • benchopt test to test that a benchmark folder is correctly structured.

The committer list for this release is the following:

Contents