Benchmarking
This document gives an overview of the benchmarking infrastructure of Concrete.
Concrete Python
Concrete Python uses progress-tracker-python to do benchmarks. Please refer to its README to learn how it works.
How to run all benchmarks?
Use the makefile target:
make benchmark
Note that this command removes the previous benchmark results before doing the benchmark.
How to run a single benchmark?
Since the full benchmark suite takes a long time to run, it's not recommended for development. Instead, use the following command to run just a single benchmark.
TARGET=foo make benchmark-target
This command would only run the benchmarks defined in benchmarks/foo.py
. It also retains the previous runs, so it can be run back to back to collect data from multiple benchmarks.
How to add new benchmarks?
Simply add a new Python script in benchmarks
directory and write your logic.
The recommended file structure is as follows:
# import progress tracker
import py_progress_tracker as progress
# import any other dependencies
from concrete import fhe
# create a list of targets to benchmark
targets = [
{
"id": (
f"name-of-the-benchmark :: "
f"parameter1 = {foo} | parameter2 = {bar}"
),
"name": (
f"Name of the benchmark with parameter1 of {foo} and parameter2 of {bar}"
),
"parameters": {
"parameter1": foo,
"parameter2": bar,
},
}
]
# write the benchmark logic
@progress.track(targets)
def main(parameter1, parameter2):
...
# to track timings
with progress.measure(id="some-metric-ms", label="Some metric (ms)"):
# execution time of this block will be measured
...
...
# to track values
progress.measure(id="another-metric", label="Another metric", value=some_metric)
...
Feel free to check benchmarks/primitive.py
to see this structure in action.
Last updated
Was this helpful?