Concrete
WebsiteLibrariesProducts & ServicesDevelopersSupport
2.10
2.10
  • Welcome
  • Get Started
    • What is Concrete?
    • Installation
    • Quick start
    • Quick overview
    • Terminology
  • Operations
    • Table Lookups basics
    • Non-linear operations
    • Other operations
      • Bit extraction
      • Common tips
      • Extensions
  • Compilation
    • Combining compiled functions
      • With composition
      • With modules
    • Key-related options for faster execution
      • Multi precision
      • Multi parameters
    • Compression
    • Reusing arguments
    • Parameter compatibility with restrictions
    • Common errors
  • Execution / Analysis
    • Simulation
    • Debugging and artifact
    • Performance
    • GPU acceleration
    • Other
      • Statistics
      • Progressbar
      • Formatting and drawing
  • Guides
    • Configure
    • Manage keys
    • Deploy
    • TFHE-rs Interoperability
      • Shared key
      • Serialization
    • Optimization
      • Improve parallelism
        • Dataflow parallelism
        • Tensorizing operations
      • Optimize table lookups
        • Reducing TLU
        • Implementation strategies
        • Round/truncating
        • Approximate mode
        • Bit extraction
      • Optimize cryptographic parameters
        • Error probability
        • Composition
  • Tutorials
    • See all tutorials
    • Part I: Concrete - FHE compiler
    • Part II: The Architecture of Concrete
  • References
    • API
    • Supported operations
  • Explanations
    • Compiler workflow
    • Advanced features
      • Table Lookups advanced
      • Rounding
      • Truncating
      • Floating points
      • Comparisons
      • Min/Max operations
      • Bitwise operations
      • Direct circuits
      • Tagging
    • Cryptography basics
    • Security
    • Frontend fusing
  • Developers
    • Contributing
      • Project layout
      • Compiler backend
        • Adding a new backend
      • Optimizer
      • MLIR FHE dialects
        • FHELinalg dialect
        • FHE dialect
        • TFHE dialect
        • Concrete dialect
        • Tracing dialect
        • Runtime dialect
        • SDFG dialect
      • Call FHE circuits from other languages
      • Benchmarking
      • Examples
      • Making a release
    • Release note
    • Feature request
    • Bug report
Powered by GitBook
On this page
  • Concrete Python
  • How to run all benchmarks?
  • How to run a single benchmark?
  • How to add new benchmarks?

Was this helpful?

Export as PDF
  1. Developers
  2. Contributing

Benchmarking

PreviousCall FHE circuits from other languagesNextExamples

Last updated 1 month ago

Was this helpful?

Libraries

  • TFHE-rs
  • Concrete
  • Concrete ML
  • fhEVM

Developers

  • Blog
  • Documentation
  • Github
  • FHE resources

Company

  • About
  • Introduction to FHE
  • Media
  • Careers

This document gives an overview of the benchmarking infrastructure of Concrete.

Concrete Python

Concrete Python uses to do benchmarks. Please refer to its to learn how it works.

How to run all benchmarks?

Use the makefile target:

make benchmark

Note that this command removes the previous benchmark results before doing the benchmark.

How to run a single benchmark?

Since the full benchmark suite takes a long time to run, it's not recommended for development. Instead, use the following command to run just a single benchmark.

TARGET=foo make benchmark-target

This command would only run the benchmarks defined in benchmarks/foo.py. It also retains the previous runs, so it can be run back to back to collect data from multiple benchmarks.

How to add new benchmarks?

Simply add a new Python script in benchmarks directory and write your logic.

The recommended file structure is as follows:

# import progress tracker
import py_progress_tracker as progress

# import any other dependencies
from concrete import fhe

# create a list of targets to benchmark
targets = [
    {
        "id": (
            f"name-of-the-benchmark :: "
            f"parameter1 = {foo} | parameter2 = {bar}"
        ),
        "name": (
            f"Name of the benchmark with parameter1 of {foo} and parameter2 of {bar}"
        ),
        "parameters": {
            "parameter1": foo,
            "parameter2": bar,
        },
    }
]

# write the benchmark logic
@progress.track(targets)
def main(parameter1, parameter2):
    ...

    # to track timings
    with progress.measure(id="some-metric-ms", label="Some metric (ms)"):
        # execution time of this block will be measured
        ...

    ...

    # to track values
    progress.measure(id="another-metric", label="Another metric", value=some_metric)

    ...

Feel free to check benchmarks/primitive.py to see this structure in action.

progress-tracker-python
README