benchopt.BaseDataset#
- class benchopt.BaseDataset(**parameters)#
Base class to define a dataset in a benchmark.
Datasets that derive from this class should implement one method:
get_data(): retrieves/simulates the data contained in this data set and returns a dictionary containing the data. This dictionary is passed as arguments of the objective’s methodset_data.
Optionally, datasets can implement:
skip(): decide if the dataset parameters are compatible. It returns the reason to skip, displayed in the CLI output, or None to run.prepare(): performs expensive one-time preparation (downloads, extraction, preprocessing) that is cached and separated from loading.
- abstractmethod get_data()#
Return the data to feed to the objective .
The result is cached on the instance, so calling it again (e.g. in a
skip) does not reload the data.- Returns:
- data: dict
Extra parameters of the objective. The objective will be instanciated by calling
Objective.set_data(**data).
- prepare()#
Prepare the dataset for use (optional).
Called before benchmark runs to perform expensive one-time operations such as downloading data, extracting archives, or pre-processing. Benchopt caches the result with joblib so that repeated calls with the same parameters are no-ops. Triggered via
benchopt prepare.Notes
Defaults to a no-op; datasets without a custom
prepare()fall back to callingget_data()for backward compatibility.Should be idempotent: calling it multiple times must be safe.
Parameters listed in
prepare_cache_ignoreare excluded from the cache key. Useprepare_cache_ignore = "all"to cache at most once per dataset class regardless of parameterization, or list specific names to ignore (e.g. a random seed):class Dataset(BaseDataset): parameters = {'n_samples': [100, 1000], 'seed': [0, 1, 2]} prepare_cache_ignore = ('seed',) # 2 calls instead of 6
The benchmark
--seedis part of the prepare cache key, so preparing with a different seed re-runs preparation. Pass the same--seedtobenchopt prepareandbenchopt runso the prepared data matches the run. If preparation does not depend on the seed, it can be ignored addingbase_seedinprepare_cache_ignore(included in"all").At preparation time only the dataset is known, so
get_seed(inprepareor in theget_datait triggers) may only be called withuse_dataset=True. Requesting the objective, solver or repetition seed raises aValueError, as they are not yet defined. See Reproducibility helpers: seeding and run artifacts.
- skip()#
Hook to decide if the
Datasetshould be skipped.It is called before launching the runs, so skipping here removes the runs of all objectives and solvers, e.g. for incompatible parameters. Calling
self.get_data()loads the data of the first repetition, on the machine runningbenchopt run.- Returns:
- reasonstr | None
The reason to skip this dataset, displayed in the output, or None to run it. Any other value raises an error.