What’s new#
Version 1.5.1 - 22/09/2023#
Bugfix release.
FIX#
Fix benchopt dependency specification to install benchopt in child env with extra
[test]
. By Thomas Moreau (#662).
Version 1.5 - 18/09/2023#
API#
Add a
Objective.url
attribute to specify the original repo of the benchmark. By Thomas Moreau (#621).Deprecate passing in arguments to callback of when
sampling_strategy='callback'
. Now on, the results from theSolver
are collected usingget_result
. By Thomas Moreau (#631).Deprecate
Objective.get_one_solution
in favor ofObjective.get_one_result
for consistency withObjective.evaluate_result
. By Thomas Moreau (#631).Deprecate
Objective.compute
in favor ofObjective.evaluate_result
, for consistency withSolver.get_result
. LikeDataset.get_data
,Solver.get_result
must now return a dictionary, which is unpacked as arguments toObjective.evaluate_result
. By Mathurin Massias (#576).Solver.support_sparse
attribute is deprecated in favor of the use ofSolver.skip
, by Mathurin Massias (#614).stopping_strategy
attribute is replaced bysampling_strategy
to clarify the concept, by Mathurin Massias (#585).Add
Solver.warm_up
function for explicit warmup instructions, such as empty run for jitting. This function is called only once per solver. By Pierre Ablin (#602).
PLOT#
Add the possibility to save views of the plot in the HTML. These views can be created in the HTML interface and saved in config files, linked to output parquet files, by `Amelie Vernay`_, Tanguy Lefort, Melvine Nargeot and Thomas Moreau (#552).
DOC#
Reformatting and enriching the documentation for easy onboarding. By Badr Moufad and Mathurin Massias (#619, #629).
Tutorial on adding a new solver to a benchmark. By Badr MOUFAD and Mathurin Massias (#635).
Internals#
Add helper to store and retrieve metadata in parquet files. This will allow storing per-run plotting information. By Thomas Moreau (#637).
Version 1.4 - 03/07/2023#
CLI#
Add support for minute and hour unit suffix in timeout limit through the syntax
--timeout 10m
or--timeout 1h
. By Mathurin Massias (#535).Remove deprecated
-o/--objective-filter
option inbenchopt run
. By Thomas Moreau (#569)Deprecate
.ini
config file and use.yml
files instead. A conversion should be performed automatically. By Tanguy Lefort, `Amelie Vernay`_ and Thomas Moreau (#552).
API#
The
get_next
method ofBaseSolver
is no longer static. By Badr Moufad (#566)Add
SingleRunCriterion
to run a solver only once. This can be used for benchmarking methods where we are interested in objective value at convergence. By Thomas Moreau (#511)Add
run_once()
helper to easily warmup solvers with callback. By Thomas Moreau (#511)Add
pre_run_hook()
hook to ignore cost that cannot be cached globally for a solver. By Thomas Moreau (#525)Remove deprecated
Objective.to_dict
,safe_import_context.import_from
. Force implementation ofget_one_solution()
. By Thomas Moreau (#569)
PLOT#
Add a tooltip beside to show description of objective. Description is provided as docstring of the
BaseObjective
class. By Badr Moufad (#556)Show solver description when hovering over solvers. Description is provided as docstring of the
BaseSolver
class. By Badr Moufad (#543)Enable visualizing the objective as function of
stopping_criterion
: time, iteration, or tolerance. By Badr Moufad (#479)Add button to share and set specific views on the plot. For now, the view need to be defined manually in the benchmark config file but an export button will be added in follow up PRs. By Tanguy Lefort, `Amelie Vernay`_ and Thomas Moreau (#552).
FIX#
Do not fail and raise a warning when
safe_import_context
is not namedimport_ctx
. By Mathurin Massias (#524)
Version 1.3.1 - 01/12/2022#
Bug fix release
FIX#
Typo in README and doc
Barchart solver color to be the same as the other plots.
Warning for deprecation not using the right class
Plot quantile incorrect display
Version 1.3 - 21/11/2022#
CLI#
Add support for custom parameters in CLI for objectives, datasets, and solvers, through the syntax
-s solver_name[parameter=value]
. See the CLI documentation for more details on the syntax. By Tom Dupré la Tour (#362).Add
--slurm
option inbenchopt run
to allow running the benchmark on a SLURM cluster. See the Running the benchmark on a SLURM cluster for more details on the config. By Thomas Moreau (#407)Add
benchopt archive
to create atar.gz
archive with the benchmark’s files for sharing with others or as supplementary materials for papers. By Thomas Moreau (#408).Now the result data are saved in the Parquet format. The use of CSV files is deprecated. By Melvine Nargeot (#433).
Change the default number of repetitions to
1
. By Benoît Malézieux (#457).
API#
Allow changing tracked metric in
StoppingCriterion
. By Amélie Vernay and Thomas Moreau (#461).Add latest git tag via
benchmark-git-tag
key inbenchopt.utils.sys_info.get_sys_info()
. By Mathurin Massias (#421).Deprecate
Objective.to_dict
in favor ofget_objective()
. By Mathurin Massias (#489).Deprecate
import_from
in favor of abenchmark_utils
module dynamically installed when running a benchmark. By Mathurin Massias and Thomas Moreau (#472).Allow specifying channels for conda requirements with syntax
chan:deps
. By Thomas Moreau (#483).Allow for template
Solver
andDataset
indatasets
/solvers
directory. By Thomas Moreau (#473).
Version 1.2 - 06/05/2022#
Changelog#
New
benchopt info
command to display information about solvers and datasets of a benchmark, by Ghislain Durif (#140).New
--profile
option to therun
command in order to profile with the line-profiler package all functions decorated withbenchopt.utils.profile()
, by Alexandre Gramfort (#186).Replace
SufficientDescentCriterion
bySufficientProgressCriterion
, which measures progress relative to the best attained value instead of the previous one, by Thomas Moreau (#176)Now all values returned by
Objective.compute
are included in reports, by Thomas Moreau and Alexandre Gramfort (#200).New
--n-jobs, -j
option to run the benchmark in parallel withjoblib
, by Thomas Moreau (#265).
API#
When returning a dictionary,
Objective.compute
should at least includevalue
key instead ofobjective_value
, by Thomas Moreau and Alexandre Gramfort (#200).stop_strategy
attribute is replaced bystopping_strategy
to harmonize withstopping_criterion
, by Benoît Malézieux (#274).Add
import_from
method insafe_import_context
to allow importing common files and packages without installation from BENCHMARK_DIR/utils, by Thomas Moreau (#286).Add
X_density
argument todatasets.make_correlated_data
to simulate sparse design matrices, by Mathurin Massias (#289).Dataset.get_data
should now return a dictionary and not a tuple. A point for testing should be returned by a dedicated methodObjective.get_one_solution
, by Thomas Moreau (#345).
CLI#
Replace
-p
flag by-o
for Objective, by Mathurin Massias (#281).Add
--config
option to support passing argument with ayaml
config file, by Mathurin Massias (#325).
Version 1.1 - 22-04-2021#
Changelog#
New plotting functions with different optimality criteria, by Nidham Gazagnadou (#96).
Support Plotly for plotting functions, by Thomas Moreau, Tanguy Lefort and Joseph Salmon (#110, #111, #112).
Change envrionment variable for config from
BENCHO_*
toBENCHOPT_*
, by Thomas Moreau (#128).Add autocompletion support in the
benchopt
command, by Alexandre Gramfort, Tanguy Lefort and Thomas Moreau (#133, #135).Move most CI to GitHub action, with auto-release on PyPi, by Thomas Moreau (#150, #154).
Remove
BENCHOPT_ALLOW_INSTALL
and always install to requested env as the user now must request explicitly the install, by Thomas Moreau (#155).
API#
Objective.compute
can now return a dictionary with multiple outputs to monitor several metrics at once, by Thomas Moreau (#84).Solver.skip
can now be used to skip objectives that are incompatible for the Solver, by Thomas Moreau (#113).Solver
can now usestop_strategy='callback'
to allow for single call curve construction, by Tanguy Lefort and Thomas Moreau (#137).Add
StoppingCriterion
to reliably and flexibly assess a solver convergence (cvg). For now, onlySufficientDescentCriterion
is implemented but better API to set criterion per benchmark should be implemented in a future release, by Thomas Moreau (#151)
CLI#
Add
--version
option forbenchopt
, by Thomas Moreau (#83).Add
--pdb
option forbenchopt run
to open debugger on error and help benchmark debugging, by Thomas Moreau (#86).Change default run to local mode. Can call a run in a dedicated env with option
--env
or--env-name ENV_NAME
to specify the env, by Thomas Moreau (#94).Add
benchopt publish
command to push benchmark results to GitHub, by Thomas Moreau (#110).Add
benchopt clean
command to remove cached files and output files locally, by Thomas Moreau (#128).Add
benchopt config
command to allow easy configuration ofbenchopt
using the CLI, by Thomas Moreau (#128).Add
benchopt install
command to install benchmark requirements (not done inbenchopt run
anymore) by Ghislain Durif (#135).Add
benchopt info
command to print information about a benchmark (including solvers, datasets, dependencies, etc.) by Ghislain Durif (#140).
BUG#
Throw a warning when benchopt version in conda env does not match the one of calling
benchopt
, by Thomas Moreau (#83).Fix Lapack issue with R code, by Tanguy Lefort (#97).
DOC#
Improve how-to narrative documentation, by Alexandre Gramfort (#93).
Add what’s new page, by Alexandre Gramfort (#114).
Add documentation on how to publish results, by Alexandre Gramfort (#118).
The committer list for this release is the following:
Version 1.0 - 2020-09-25#
Release highlights#
Provide a command line interface for benchmarking optimisation algorithm implementations:
benchopt run
to run the benchmarksbenchopt plot
to display the resultsbenchopt test
to test that a benchmark folder is correctly structured.
The committer list for this release is the following: