Add a custom plot to a benchmark#
Benchopt provides a set of default plots to visualize the results of a benchmark.
These plots can be complemented with custom plots, defined in the benchmark,
to visualize the results in a different way. These plots are defined in the
plots directory, by adding python files with classes inheriting from
benchopt.BasePlot. This page details the API to generate custom
visualizations for your benchmark.
Structure of a custom plot#
A custom plot is defined by a class inheriting from benchopt.BasePlot and implementing:
name: The name of the plot title. This will be the name that appears in the plot selection menu of the HTML interface, or the name you can use to select this plot in config files for your benchmark.type: The type of the plot, which defines how the output of plot will be rendered. Supported types are"scatter","bar_chart","boxplot","table"and"image".options: A dictionary defining the different options available for the plot. Typically, this can be used to have different plots depending on dataset’s or objective’s parameters, or to display customization options. The keys in the dictionary are the names of the options, associated to a list of their possible values. If a keyobjective/dataset/solver/objective_columnis associated with the value..., the options are automatically inferred from the results DataFrame, as all unique values associated with this key. A value can also be a callable taking the results DataFrame as input and returning the list of possible values for the option.plot(self, df, **kwargs): give the data to produce one plot, that is rendered with the plotly or matplotlib backend. The method takes the results DataFrame and the options values as arguments, and returns the plot data. The output depends on the plot’s type, and are detailed below for each of them.get_metadata(self, df, **kwargs): Gives global information about the plot, such as the title and axis labels. The method takes the results DataFrame and the options values as arguments, and returns the metadata of the plot, which is specific to each plot type.
The get_metadata method allow to change global properties of
the resulting visualization, and the plot method outputs the data
necessary to render it.
The visualization is rendered using either the plotly or matplotly backend.
from benchopt import BasePlot
class Plot(BasePlot):
name = "My Custom Plot"
type = "scatter" # or "bar_chart", "boxplot", "table" or "image"
options = {
"dataset": ..., # Automatic options from DataFrame columns
"objective": ...,
"my_parameter": [1, 2], # custom options
# options computed from the results DataFrame
"solver": lambda df: df["solver_name"].unique().tolist(),
}
# The inputs args of this method correspond to `df` and
# the keys in the `options` dictionary.
def plot(self, df, dataset, objective, my_parameter, solver):
# ... process df ...
return plot_data
def get_metadata(self, df, dataset, objective, my_parameter, solver):
return {
"title": f"Plot for {dataset}",
"xlabel": "X Label",
"ylabel": "Y Label",
}
Plot Options#
The options dictionary keys define the arguments passed to
plot and get_metadata. Special keys like
dataset, objective, solver will automatically
try to match columns in the dataframe. Using ... as a value
will populate the options with all unique values from the dataframe
column {key}_name (e.g. dataset_name). An option value can
also be a {"value": ..., "description": ...} dict, to show a hover
icon with the description HTML next to the selector. plot
and get_metadata receive the plain value.
Scatter Plot#
For a scatter plot, the plot method should return a list of dictionaries, where
each dictionary represents a trace in the plot. Each dictionary must contain:
x: A list of x values.y: A list of y values.label: The label of the trace.description(optional): HTML shown by a hover icon next to the label in the legend. A trace labeled with a solver, dataset or objective name shows its description and fixed parameters by default. Set it to override this, or toNoneto hide the icon.color(optional): The color of the trace.marker(optional): The marker style of the trace.y_low,y_high(optional): Lists of values to display uncertainty in the plot. They will be used to display shaded area around the plot.x_low,x_high(optional): Lists of values to display uncertainty in the plot. They will be used to display shaded area around the plot. You can use either y_low/y_high or x_low/x_high, but not both.
The metadata dictionary returned by get_metadata should contain:
title: The title of the plot.xlabel: The label of the x-axis.ylabel: The label of the y-axis.grid(optional, default=True): Whether to show grid lines in the plot. This only affects the matplotlib backend, not the html page.scale(optional, default=”loglog”): The scale of the axes in the matplotlib backend, can be either “linear”, “semilog-x”, “semilog-y” or “loglog”.
def plot(self, df, dataset, objective, my_parameter, solver):
# Filter the dataframe
df = df.query(
"dataset_name == @dataset and objective_name == @objective"
)
plot_traces = []
for solver, df_solver in df.groupby('solver_name'):
# Compute the median over the repetitions
curve = (
df_solver.groupby("stop_val")[["time", "'objective_value"]]
.median()
)
plot_traces.append({
"x": curve['time'].tolist(),
"y": curve['objective_value'].tolist(),
"label": solver,
**self.get_style(solver)
})
return plot_traces
def get_metadata(self, df, dataset, objective, my_parameter, solver):
return {
"title": f"Convergence for {dataset}",
"xlabel": "Time [sec]",
"ylabel": "Objective value",
}
Note
To help with consistent style accross figures, you can use
the helper get_style, as described in Plotting Utilities.
Bar Chart#
For a bar chart, the plot method should return a list of dictionaries,
where each dictionary represents a bar. For each bar, the median value will be
used to determine its height, while the individual values will be displayed as
scatter points. The dictionary should contain:
y: The list of values for the bar (the median will be the height of the bar).label: The label of the bar.color(optional): The color of the bar.text(optional): The text to display on the bar.
The metadata dictionary returned by get_metadata should contain:
title: The title of the plot.ylabel: The label of the y-axis.grid(optional, default=True): Whether to show grid lines on the y-axis in the plot. This only affects the matplotlib backend, not the html page.
def plot(self, df, dataset, objective, **kwargs):
df = df.query(
"dataset_name == @dataset and objective_name == @objective"
)
bars = []
for solver, df_solver in df.groupby('solver_name'):
# Select the total runtime for each repetition
runtimes = df_solver.groupby("idx_rep")["runtime"].last()
bars.append({
"y": runtimes.tolist(),
"label": solver,
"text": "",
"color": self.get_style(solver)['color']
})
return bars
def get_metadata(self, df, dataset, objective, **kwargs):
return {
"title": f"Average times for {objective} on {dataset}",
"ylabel": "Time [sec]",
}
Box Plot#
For a box plot, the plot method should return a list of dictionaries,
where each dictionary represents a box. Each dictionary should contain:
x: The x coordinate.y: The values of the box for the corresponding x coordinate.label: The label of the box.color(optional): The color of the box.
The metadata dictionary returned by get_metadata should contain:
title: The title of the plot.xlabel: The label of the x-axis.ylabel: The label of the y-axis.grid(optional, default=True): Whether to show grid lines on the y-axis in the plot. This only affects the matplotlib backend, not the html page.box_width(optional, default=0.6): The width of the boxes, only affects the matplotlib backend, not the html page.showfliers(optional, default=False): Whether to show fliers in the boxplot. Fliers are points that are outside the whiskers of the boxplot, which represent outliers in the data. This only affects the matplotlib backend, not the html page.
def plot(self, df, dataset, objective, **kwargs):
df = df.query(
"dataset_name == @dataset and objective_name == @objective"
)
plot_data = []
for solver, df_solver in df.groupby('solver_name'):
# Example: boxplot for the final objective values
# for each solver
final_objective_value = (
df_solver.groupby("idx_rep")['objective_value'].last()
)
plot_data.append({
"x": [solver],
"y": [final_objective_value.tolist()],
"label": solver,
"color": self.get_style(solver)['color']
})
return plot_data
def get_metadata(self, df, dataset, objective, **kwargs):
return {
"title": f"Boxplot for {objective} on {dataset}",
"xlabel": "Solver",
"ylabel": "Objective value",
}
Table Plot#
For a table plot, the plot method should return a list of lists,
where each inner list represents a row in the table.
The metadata dictionary returned by get_metadata should contain:
title: The title of the plot.columns: A list of column names.default_order_column(optional, default=0): The column to sort the rows on by default in the html report, given as a column name or a 0-based column index.default_order_ascending(optional, default=True): Whether the default ordering is in increasing order.
String cells can carry simple formatting with **bold**,
*italic* and __underlined__ markers. A literal marker is
escaped with a backslash, as in r"a * b". A cell holding a solver,
dataset or objective name, with or without markers, gets a hover icon with
its description. A cell can also be a
{"value": ..., "description": ...} dict, to set its hover HTML
explicitly, or hide it with None.
In the html report, each column header can be clicked to sort on that column, and the arrow next to it can be toggled to switch between increasing and decreasing order. A search bar allows filtering the rows, and each column can be shown or hidden with the checkboxes below the table.
Solver, dataset and objective names are already shortened (see Shortened labels and hover descriptions):
def plot(self, df, dataset, objective, **kwargs):
df = df.query(
"dataset_name == @dataset and objective_name == @objective"
)
rows = []
for solver, df_solver in df.groupby('solver_name'):
# Example: table with solver name and mean time
# when using `sampling_strategy = 'run_once'`
rows.append([f"**{solver}**", df_solver['time'].mean()])
return rows
def get_metadata(self, df, dataset, objective, **kwargs):
return {
"title": f"Summary for {dataset}",
"columns": ["Solver", "Mean Time [sec]"],
"default_order_column": "Mean Time [sec]",
"default_order_ascending": True,
}
Image Plot#
For an image plot, the plot method should return a list of dictionaries,
where each dictionary represents one image card displayed in a grid.
Each dictionary must contain:
image: Either an image-compatible array (rendered as a PNG) or a list of image-compatible arrays (rendered as an animated GIF showing per-iteration progress). A pre-computed base64 data URI or URL are also accepted. If set toNone, this will create an empty image, which can be used for alignment purposes.
Optional keys:
label: Text displayed below the image card.
Arrays are expected to have values in [0, 1] and are converted automatically
using Pillow, so no manual encoding is needed.
The metadata dictionary returned by get_metadata should contain:
title: The title displayed above the grid.ncols: Number of columns in the grid (default: min(n_images, 3)).
Note
In the HTML result page, animated GIFs are rendered when a list of arrays is provided. In the matplotlib backend, each image card is shown as a static subplot using the last frame for animated sequences.
Plotting Utilities#
To ensure consistency across plots (e.g., using the same color and marker for a
given solver), benchopt.BasePlot provides the helper method
get_style(label). This method returns a dictionary with color
and marker keys, which can be directly unpacked into the trace dictionary
for scatter plots or used to select the color for other plot types. It
automatically assigns a color from the default color palette based on the hash
of the label, ensuring that the same solver always gets the same color.
# Usage in plot()
style = self.get_style(solver_name)
trace = {
# ...
"color": style["color"],
"marker": style["marker"]
}
# Or simply:
trace = {
# ...
**self.get_style(solver_name)
}
Shortened labels and hover descriptions#
Parametrized class names (e.g. Solver[alpha=0.1,n_iter=100]) can be
verbose. The HTML report automatically shortens them, keeping only the
parameters that vary across the results, and a hover icon next to each entry
reveals its description: the class docstring and the full parameter list.
Traces, table cells and option values that are a solver, dataset or objective
name get it implicitly, unless they set an explicit description. If
names of different kinds share a short label, the description is taken from
the solver, then the dataset, then the objective.
This happens before plot and get_metadata are called: the
solver_name, dataset_name and objective_name columns of the df
you receive already hold the short labels, so a custom plot that groups on
those columns gets short labels for free. The full name of each row is kept in
the *_full_name columns (solver_full_name, dataset_full_name,
objective_full_name), and the *_description columns
(solver_description, dataset_description, obj_description) hold
the hover HTML. A trace labeled with a short name gets its hover icon
automatically:
def plot(self, df, dataset, objective, **kwargs):
traces = []
for solver, df_solver in df.groupby('solver_name'):
# `solver` is already the short label here.
traces.append({
"x": ..., "y": ..., "label": solver,
**self.get_style(solver),
})
return traces
With another label, pass the matching *_description as the trace’s
description:
"label": f"{solver} ({dataset})",
"description": df_solver["solver_description"].iloc[0],
The short labels are display-only and computed relative to the current
results. Never write them back into persisted results, and use the
*_full_name columns as the canonical identity.