benchopt.BaseObjective

class benchopt.BaseObjective(**parameters)

Base class to define an objective function

Objectives that derive from this class needs to implement four methods:

• set_data(**data): stores the info from a given dataset to be able to compute the objective value on these data.

• get_objective(): exports the data from the dataset and the parameters from the objective function as a dictionary that will be passed as parameters of the solver’s set_objective method in order to specify the objective function of the benchmark.

• evaluate_result(**result): evaluate the metrics on the results of a solver. Its arguments should correspond to the key of the dictionary returned by Solver.get_result and it can return a scalar value or a dictionary. If it returns a dictionary, it should at least contain a key value associated to a scalar value which will be used to detect convergence. With a dictionary, multiple metric values can be stored at once instead of running each separately.

• get_one_result(): return one result for which the objective can be evaluated. This should be a dictionary where the keys correspond to the keyword arguments of evaluate_result.

Optionally, the Solver can implement the following methods to change its behavior:

• save_final_results(**result): Return the data to be saved from the

  results of the solver. It will be saved as a .pkl file in the output/results folder, and link to the benchmark results.

• get_next(stop_val): Return the next iteration where the result will be evaluated.

This class is also used to specify information about the benchmark. In particular, it should have the following class attributes:

• name: a name for the benchmark, that will be used to display results.

• url: the url of the original benchmark repository.

• requirements: the minimal requirements to be able to run the benchmark.

• min_benchopt_version: the minimal version of benchopt required to run this benchmark.

abstract evaluate_result(**solver_result)

Compute the objective value given the output of a solver.

The arguments are the keys in the result dictionary returned by Solver.get_result.

Parameters:

solver_resultdict

All values needed to compute the objective metrics. This dictionary is retrieved by calling solver_result = Solver.get_result().

Returns:

objective_valuefloat or dict {‘name’: float}

The value(s) of the objective function. If a dictionary is returned, it should at least contain a key value associated to a scalar value which will be used to detect convergence. With a dictionary, multiple metric values can be stored at once instead of running each separately.

abstract get_objective()

Return the objective parameters for the solver.

Returns:

objective_dict: dict

Parameters of the objective that will be given to the solver when calling Solver.set_objective(**objective_dict).

abstract get_one_result()

Return one result for which the objective can be evaluated.

This method is mainly for testing purposes, to check that the method Objective.compute can be called and that it returns a compatible type for benchopt. The returned object will be passed to Objective.compute.

get_split(*arrays)

Return the split of the data according to the cv attribute.

Parameters:

arrays: list of array-like

The data to split. It should be indexable with the output of the cv.split iterator, or compatible with Objective.split.

save_final_results(**solver_result)

Save the final results of the solver.

Parameters:

solver_resultdict

All values needed to compute the objective metrics. This dictionary is retrieved by calling solver_result = Solver.get_result().

Returns

——-

dict of values to save

abstract set_data(**data)

Store the info on a dataset to be able to compute the objective.

Parameters:

**data: dict

Extra parameters of the objective. This dictionary is retrieved by calling data = Dataset.get_data().

skip(**data)

Used to decide if the Objective is compatible with the data.

Parameters:

**data: dict

Extra parameters of the objective. This dictionary is retrieved by calling data = Dataset.get_data().

Returns:

skipbool

Whether this objective should be skipped or not for this data (accessible in the objective attributes).

reasonstr | None

The reason why it should be skipped for display purposes. If skip is False, the reason should be None.