benchopt.BaseDataset#

class benchopt.BaseDataset(**parameters)#

Base class to define a dataset in a benchmark.

Datasets that derive from this class should implement one method:

  • get_data(): retrieves/simulates the data contained in this data set and returns a dictionary containing the data. This dictionary is passed as arguments of the objective’s method set_data.

Optionally, datasets can implement:

  • skip(): decide if the dataset parameters are compatible. It returns the reason to skip, displayed in the CLI output, or None to run.

  • prepare(): performs expensive one-time preparation (downloads, extraction, preprocessing) that is cached and separated from loading.

abstractmethod get_data()#

Return the data to feed to the objective .

The result is cached on the instance, so calling it again (e.g. in a skip) does not reload the data.

Returns:
data: dict

Extra parameters of the objective. The objective will be instanciated by calling Objective.set_data(**data).

prepare()#

Prepare the dataset for use (optional).

Called before benchmark runs to perform expensive one-time operations such as downloading data, extracting archives, or pre-processing. Benchopt caches the result with joblib so that repeated calls with the same parameters are no-ops. Triggered via benchopt prepare.

Notes

  • Defaults to a no-op; datasets without a custom prepare() fall back to calling get_data() for backward compatibility.

  • Should be idempotent: calling it multiple times must be safe.

  • Parameters listed in prepare_cache_ignore are excluded from the cache key. Use prepare_cache_ignore = "all" to cache at most once per dataset class regardless of parameterization, or list specific names to ignore (e.g. a random seed):

    class Dataset(BaseDataset):
        parameters = {'n_samples': [100, 1000], 'seed': [0, 1, 2]}
        prepare_cache_ignore = ('seed',)  # 2 calls instead of 6
    
  • The benchmark --seed is part of the prepare cache key, so preparing with a different seed re-runs preparation. Pass the same --seed to benchopt prepare and benchopt run so the prepared data matches the run. If preparation does not depend on the seed, it can be ignored adding base_seed in prepare_cache_ignore (included in "all").

  • At preparation time only the dataset is known, so get_seed (in prepare or in the get_data it triggers) may only be called with use_dataset=True. Requesting the objective, solver or repetition seed raises a ValueError, as they are not yet defined. See Reproducibility helpers: seeding and run artifacts.

skip()#

Hook to decide if the Dataset should be skipped.

It is called before launching the runs, so skipping here removes the runs of all objectives and solvers, e.g. for incompatible parameters. Calling self.get_data() loads the data of the first repetition, on the machine running benchopt run.

Returns:
reasonstr | None

The reason to skip this dataset, displayed in the output, or None to run it. Any other value raises an error.