benchopt.BaseObjective#
- class benchopt.BaseObjective(**parameters)#
Base class to define an objective function
Objectives that derive from this class needs to implement four methods:
set_data(**data): stores the info from a given dataset to be able to compute the objective value on these data.
get_objective(): exports the data from the dataset and the parameters from the objective function as a dictionary that will be passed as parameters of the solver’s set_objective method in order to specify the objective function of the benchmark.
evaluate_result(**result): evaluate the metrics on the results of a solver. Its arguments should correspond to the key of the dictionary returned by Solver.get_result and it can return a scalar value or a dictionary. If it returns a dictionary, it should at least contain a key value associated to a scalar value which will be used to detect convergence. With a dictionary, multiple metric values can be stored at once instead of running each separately.
get_one_result(): return one result for which the objective can be evaluated. This should be a dictionary where the keys correspond to the keyword arguments of evaluate_result.
This class is also used to specify information about the benchmark. In particular, it should have the following class attributes:
name: a name for the benchmark, that will be used to display results.
url: the url of the original benchmark repository.
requirements: the minimal requirements to be able to run the benchmark.
min_benchopt_version: the minimal version of benchopt required to run this benchmark.
- abstract evaluate_result(**solver_result)#
Compute the objective value given the output of a solver.
The arguments are the keys in the result dictionary returned by
Solver.get_result
.- Parameters
- solver_resultdict
All values needed to compute the objective metrics. This dictionary is retrieved by calling
solver_result = Solver.get_result()
.
- Returns
- objective_valuefloat or dict {‘name’: float}
The value(s) of the objective function. If a dictionary is returned, it should at least contain a key value associated to a scalar value which will be used to detect convergence. With a dictionary, multiple metric values can be stored at once instead of running each separately.
- abstract get_objective()#
Return the objective parameters for the solver.
- Returns
- objective_dict: dict
Parameters of the objective that will be given to the solver when calling
Solver.set_objective(**objective_dict)
.
- abstract get_one_result()#
Return one result for which the objective can be evaluated.
This method is mainly for testing purposes, to check that the method Objective.compute can be called and that it returns a compatible type for benchopt. The returned object will be passed to
Objective.compute
.
- abstract set_data(**data)#
Store the info on a dataset to be able to compute the objective.
- Parameters
- **data: dict
Extra parameters of the objective. This dictionary is retrieved by calling
data = Dataset.get_data()
.
- skip(**data)#
Used to decide if the
Objective
is compatible with the data.- Parameters
- **data: dict
Extra parameters of the objective. This dictionary is retrieved by calling
data = Dataset.get_data()
.
- Returns
- skipbool
Whether this objective should be skipped or not for this data (accessible in the objective attributes).
- reasonstr | None
The reason why it should be skipped for display purposes. If skip is False, the reason should be None.