benchopt.BaseObjective

class benchopt.BaseObjective(**parameters)

Base class to define an objective function

Objectives that derive from this class needs to implement three methods:

• set_data(**data): stores the info from a given dataset to be able to compute the objective value on these data.

• get_objective(): exports the data from the dataset and the parameters from the objective function as a dictionary that will be passed as parameters of the solver’s set_objective method in order to specify the objective function of the benchmark.

• evaluate_result(**result): evaluate the metrics on the results of a solver. Its arguments should correspond to the key of the dictionary returned by Solver.get_result and it can return a scalar value, a dictionary or a list of dictionaries. If it returns a dictionary, it should at least contain a key value associated to a scalar value which will be used to detect convergence. With a dictionary, multiple metric values can be stored at once instead of running each separately. With a list of dictionaries, these metrics can be computed on different objects.

Optionally, the Objective can implement the following methods to change its behavior:

• get_one_result(): return one dummy result compatible with evaluate_result. Used by benchopt test to validate metric computation. If not implemented, benchopt run works normally but the test-time metric validation step is silently skipped.

• save_final_results(**result): persist artefacts (trained models, arrays, …) from the final solver run as a .pkl file alongside the parquet results.

This class is also used to specify information about the benchmark. In particular, it should have the following class attributes:

• name: a name for the benchmark, that will be used to display results.

• url: the url of the original benchmark repository.

• requirements: the minimal requirements to be able to run the

  benchmark.

• min_benchopt_version: the minimal version of benchopt required to run this benchmark.

Some extra configurations can be set through optional class attributes:

• test_dataset_name: the name of the dataset to use for testing.

• stopping_criterion: the default stopping criterion to use for this benchmark.

• sampling_strategy: the default sampling strategy to use for this benchmark.

abstractmethod evaluate_result(**solver_result)

Compute the objective value given the output of a solver.

The arguments are the keys in the result dictionary returned by Solver.get_result.

Parameters:

solver_resultdict

All values needed to compute the objective metrics. This dictionary is retrieved by calling solver_result = Solver.get_result().

Returns:

objective_valuefloat or dict {str: float} or list of dict

The value(s) of the objective function. If a dictionary is returned, it should at least contain a key value associated to a scalar value which will be used to detect convergence. With a dictionary, multiple metric values can be stored at once instead of running each separately.

abstractmethod get_objective()

Return the objective parameters for the solver.

Returns:

objective_dict: dict

Parameters of the objective that will be given to the solver when calling Solver.set_objective(**objective_dict).

get_one_result()

Return one result for which the objective can be evaluated.

This method is used by benchopt test to check that the benchmark components are compatible without running the full benchmark. It should return a dictionary whose keys match the keyword arguments of Objective.evaluate_result().

This method is optional: benchopt run works normally without it, but benchopt test will skip the objective validation step when it is not implemented.

Returns:

resultdict

A dummy result compatible with Objective.evaluate_result().

get_split(*arrays)

Return the split of the data according to the cv attribute.

Parameters:

arrays: list of array-like

The data to split. It should be indexable with the output of the cv.split iterator, or compatible with Objective.split.

save_final_results(**solver_result)

Optionally save artefacts from the final solver run.

Called once after the last benchmark run for each solver configuration. Use this to persist heavy objects (trained models, arrays, …) that are too large to include in the main results file.

Parameters:

**solver_resultdict

The dictionary returned by Solver.get_result() for the final run. Keys match the keyword arguments of evaluate_result.

Returns:

dict

Objects to save. Serialised with pickle and stored alongside the parquet results file; the path is recorded in the results.

abstractmethod set_data(**data)

Store the info on a dataset to be able to compute the objective.

Parameters:

**data: dict

Extra parameters of the objective. This dictionary is retrieved by calling data = Dataset.get_data().

skip(**data)

Used to decide if the Objective is compatible with the data.

Parameters:

**data: dict

Extra parameters of the objective. This dictionary is retrieved by calling data = Dataset.get_data().

Returns:

skipbool

Whether this objective should be skipped or not for this data (accessible in the objective attributes).

reasonstr | None

The reason why it should be skipped for display purposes. If skip is False, the reason should be None.