This document provides guides on how to install Concrete ML using PyPi or Docker.

Prerequisite

Before you start, determine your environment:

• Hardware platform

• Operating System (OS) version

• Python version

OS/HW support

Depending on your OS/HW, Concrete ML may be installed with Docker or with pip:

Python support

• Version: In the current release, Concrete ML supports only 3.8, 3.9 and 3.10 versions of python.

• Linux requirement: The Concrete ML Python package requires glibc >= 2.28. On Linux, you can check your glibc version by running ldd --version.

• Kaggle installation: Concrete ML can be installed on Kaggle (see question on community for more details) and on Google Colab.

Most of these limits are shared with the rest of the Concrete stack (namely Concrete Python). Support for more platforms will be added in the future.

Installation using PyPi

Requirements

Installing Concrete ML using PyPi requires a Linux-based OS or macOS (both x86 and Apple Silicon CPUs are supported).

Installation

To install Concrete ML from PyPi, run the following:

pip install -U pip wheel setuptools
pip install concrete-ml

This will automatically install all dependencies, notably Concrete.

If you encounter any issue during installation on Apple Silicon mac, please visit this troubleshooting guide on community.

Installation using Docker

You can install Concrete ML using Docker by either pulling the latest image or a specific version:

docker pull zamafhe/concrete-ml:latest
# or
docker pull zamafhe/concrete-ml:v0.4.0

You can use the image with Docker volumes, see the Docker documentation here. Use the following command:

# Without local volume:
docker run --rm -it -p 8888:8888 zamafhe/concrete-ml

# With local volume to save notebooks on host:
docker run --rm -it -p 8888:8888 -v /host/path:/data zamafhe/concrete-ml

This will launch a Concrete ML enabled Jupyter server in Docker that can be accessed directly from a browser.

Alternatively, you can launch a shell in Docker, with or without volumes:

docker run --rm -it zamafhe/concrete-ml /bin/bash

Concrete ML is an open source, privacy-preserving, machine learning framework based on Fully Homomorphic Encryption (FHE). It enables data scientists without any prior knowledge of cryptography to perform:

• Automatic model conversion: Use familiar APIs from scikit-learn and PyTorch to convert machine learning models to their FHE equivalent. This is applicable for linear models, tree-based models, and neural networks).

• Encrypted data training: Train models directly on encrypted data to maintain privacy.

• Encrypted data pre-processing: Pre-process encrypted data using a DataFrame paradigm.

Key features

• Training on encrypted data: FHE is an encryption technique that allows computing directly on encrypted data, without needing to decrypt it. With FHE, you can build private-by-design applications without compromising on features. Learn more about FHE in this introduction or join the FHE.org community.

• Federated learning: Training on encrypted data provides the highest level of privacy but is slower than training on clear data. Federated learning is an alternative approach, where data privacy can be ensured by using a trusted gradient aggregator, coupled with optional differential privacy instead of encryption. Concrete ML can import all types of models: linear, tree-based and neural networks, that are trained using federated learning using the from_sklearn_model function and the compile_torch_model function.

Example usage

Here is a simple example of classification on encrypted data using logistic regression. You can find more examples here.

This example shows the typical flow of a Concrete ML model:

1. Training the model: Train the model on unencrypted (plaintext) data using scikit-learn. Since Fully Homomorphic Encryption (FHE) operates over integers, Concrete ML quantizes the model to use only integers during inference.

2. Compiling the model: Compile the quantized model to an FHE equivalent. Under the hood, the model is first converted to a Concrete Python program and then compiled.

3. Performing inference: Perform inference on encrypted data. The example above shows encrypted inference in the model-development phase. Alternatively, during deployment in a client/server setting, the client encrypts the data, the server processes it securely, and then the client decrypts the results.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from concrete.ml.sklearn import LogisticRegression

# Lets create a synthetic data-set
x, y = make_classification(n_samples=100, class_sep=2, n_features=30, random_state=42)

# Split the data-set into a train and test set
X_train, X_test, y_train, y_test = train_test_split(
    x, y, test_size=0.2, random_state=42
)

# Now we train in the clear and quantize the weights
model = LogisticRegression(n_bits=8)
model.fit(X_train, y_train)

# We can simulate the predictions in the clear
y_pred_clear = model.predict(X_test)

# We then compile on a representative set
model.compile(X_train)

# Finally we run the inference on encrypted inputs
y_pred_fhe = model.predict(X_test, fhe="execute")

print(f"In clear  : {y_pred_clear}")
print(f"In FHE    : {y_pred_fhe}")
print(f"Similarity: {(y_pred_fhe == y_pred_clear).mean():.1%}")

# Output:
    # In clear  : [0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 1 0 0]
    # In FHE    : [0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 1 0 0]
    # Similarity: 100.0%

It is also possible to call encryption, model prediction, and decryption functions separately as follows. Executing these steps separately is equivalent to calling predict_proba on the model instance.

# Predict probability for a single example
y_proba_fhe = model.predict_proba(X_test[[0]], fhe="execute")

# Quantize an original float input
q_input = model.quantize_input(X_test[[0]])

# Encrypt the input
q_input_enc = model.fhe_circuit.encrypt(q_input)

# Execute the linear product in FHE 
q_y_enc = model.fhe_circuit.run(q_input_enc)

# Decrypt the result (integer)
q_y = model.fhe_circuit.decrypt(q_y_enc)

# De-quantize and post-process the result
y0 = model.post_processing(model.dequantize_output(q_y))

print("Probability with `predict_proba`: ", y_proba_fhe)
print("Probability with encrypt/run/decrypt calls: ", y0)

Current limitations

• Precision and accuracy: In order to run models in FHE, Concrete ML requires models to be within the precision limit, currently 16-bit integers. Thus, machine learning models must be quantized and it sometimes leads to a loss of accuracy versus the original model that operates on plaintext.

• Models availability: Concrete ML currently only supports training on encrypted data for some models, while it supports inference for a large variety of models.

• Processing: Concrete currently doesn't support pre-processing model inputs and post-processing model outputs. These processing stages may involve:

  • Text-to-numerical feature transformation

  • Dimensionality reduction

  • KNN or clustering

  • Featurization

  • Normalization

  • The mixing of ensemble models' results.

These issues are currently being addressed, and significant improvements are expected to be released in the near future.

Concrete stack

Concrete ML is built on top of Zama's Concrete.

Online demos and tutorials

Various tutorials are available for built-in models and deep learning. Several stand-alone demos for use cases can be found in the Demos and Tutorials section.

If you have built awesome projects using Concrete ML, feel free to let us know and we'll link to your work!

Additional resources

• Zama's blog

Support

• Community channels (we answer in less than 24 hours).

Get started

Learn the basics of Concrete ML, set it up, and make it run with ease.

Build with Concrete ML

Start building with Concrete ML by exploring its core features, discovering essential guides, and learning more with user-friendly tutorials.

Explore more

Access to additional resources and join the Zama community.

References & Explanations

Refer to the API, review product architecture, and access additional resources for in-depth explanations while working with Concrete ML.

• Security and correctness

• API

• Quantization

• Pruning

• Compilation

• Advanced features

• Project architecture

Support channels

Ask technical questions and discuss with the community. Our team of experts usually answers within 24 hours in working days.

• Community channels

Developers

Collaborate with us to advance the FHE spaces and drive innovation together.

• Contribute to Concrete ML

• Check the latest release note

• Request a feature

• Report a bug

This page explains Concrete ML linear models for both classification and regression. These models are based on linear models.

Supported models for encrypted inference

The following models are supported for training on clear data and predicting on encrypted data. Their API is similar the one of . These models are also compatible with some of scikit-learn's main workflows, such as Pipeline() and GridSearch().

Supported models for encrypted training

In addition to predicting on encrypted data, the following models support training on encrypted data.

| | |

Quantization parameters

The n_bits parameter controls the bit-width of the inputs and weights of the linear models. Linear models do not use table lookups and thus alllows weight and inputs to be high precision integers.

For models with input dimensions up to 300, the parameter n_bits can be set to 8 or more. When the input dimensions are larger, n_bits must be reduced to 6-7. In many cases, quantized models can preserve all performance metrics compared to the non-quantized float models from scikit-learn when n_bits is down to 6. You should validate accuracy on held-out test sets and adjust n_bits accordingly.

Pre-trained models

Example

Model accuracy

Loading a pre-trained model

An alternative to the example above is to train a scikit-learn model in a separate step and then to convert it to Concrete ML.

This document introduces several 's linear models for classification and regression tree models that Concrete ML provides.

Supported models

Concrete ML also supports 's XGBClassifier and XGBRegressor:

Pre-trained models

You can convert an already trained scikit-learn tree-based model to a Concrete ML one by using the method.

Example

Here's an example of how to use this model in FHE on a popular data-set using some of scikit-learn's pre-processing tools. You can find a more complete example in the .

Quantization parameters

When using a sufficiently high bit-width, quantization has little impact on the decision boundaries of the Concrete ML FHE decision tree model, as quantization is done individually on each input feature. It means FHE models can achieve similar accuracy levels as floating point models. Using 6 bits for quantization is effective in reaching or even exceeding floating point accuracy.

To adjust the number of bits for quantization, use the n_bits parameter. Setting n_bits to a low value may introduce artifacts, potentially reducing accuracy. However, the execution speed in FHE could improve. This adjustment allows you to manage the accuracy/speed trade-off. Additionally, you can recover some accuracy by increasing the n_estimators parameter.

The following graph shows that using 5-6 bits of quantization is usually sufficient to reach the performance of a non-quantized XGBoost model on floating point data. The metrics plotted are accuracy and F1-score on the spambase data-set.

FHE Inference time considerations

The inference time in FHE is strongly dependant on the maximum circuit bit-width. For trees, in most cases, the quantization bit-width will be the same as the circuit bit-width. Therefore, reducing the quantization bit-width to 4 or less will result in fast inference times. Adding more bits will increase FHE inference time exponentially.

In some rare cases, the bit-width of the circuit can be higher than the quantization bit-width. This could happen when the quantization bit-width is low but the tree-depth is high. In such cases, the circuit bit-width is upper bounded by ceil(log2(max_depth + 1) + 1).

This document explains the essential cryptographic terms and the important concepts of Concrete ML model lifecycle with Fully Homomorphic Encryption (FHE).

Concrete ML is built on top of Concrete, which enables the conversion from NumPy programs into FHE circuits.

Lifecycle of a Concrete ML model

With Concrete ML, you can train a model on clear or encrypted data, then deploy it to predict on encrypted inputs. During deployment, data can be pre-processed while being encrypted. Therefore, data stay encrypted during the entire lifecycle of the machine learning model, with some limitations.

I. Model development

1. Training: A model is trained either using plaintext (non-encrypted) training data, or encrypted training data.

2. Quantization: Quantization converts inputs, model weights, and all intermediate values of the inference computation to integer equivalents. More information is available . Concrete ML performs this step in two ways depending on model type:

   • During training (Quantization Aware Training)

   • After training (Post-training Quantization)

3. Simulation: Simulation allows you to execute a model that was quantized, to measure its accuracy in FHE, and to determine the modifications required to make it FHE compatible. Simulation is described in more detail .

4. Compilation: After quantizing the model and confirming that it has good FHE accuracy through simulation, the model then needs to be compiled using Concrete's FHE Compiler to produce an equivalent FHE circuit. This circuit is represented as an MLIR program consisting of low level cryptographic operations. You can read more about FHE compilation , MLIR , and about the low-level Concrete library .

5. Inference: The compiled model can then be executed on encrypted data, once the proper keys have been generated. The model can also be deployed to a server and used to run private inference on encrypted inputs.

You can find examples of the model development workflow .

II. Model deployment

1. Pre-processing: Data owners(client) can generate keys to encrypt/decrypt data and store it in a for further processing on a server. The server can pre-process such data with pre-compiled circuits, to prepare it for encrypted training or inference.

2. Client/server model deployment: In a client/server setting, Concrete ML models can be exported to:

   • Allow the client to generate keys, encrypt, and decrypt.

   • Provide a compiled model that can run on the server to perform inference on encrypted data.

3. Key generation: The data owner (client) needs to generate a set of keys:

   • A private encryption key to encrypt/decrypt their data and results

   • A public evaluation key for the model's FHE evaluation on the server.

Cryptography concepts

Concrete ML and Concrete abstract the details of the underlying cryptography scheme, TFHE. However, understanding some cryptography concepts is still useful:

• Encryption and decryption: Encryption converts human-readable information (plaintext) into data (ciphertext) that is unreadable by a human or computer unless with the proper key. Encryption takes plaintext and an encryption key and produces ciphertext, while decryption is the reverse operation.

• Encrypted inference: FHE allows third parties to execute a machine learning model on encrypted data. The inference result is also encrypted and can only be decrypted by the key holder.

• Key generation: Cryptographic keys are generated using random number generators. Key generation can be time-consuming and produce large keys, but each model used by a client only requires key generation once.

  • Private encryption key: A private encryption key is a series of bits used within an encryption algorithm for encrypting data so that the corresponding ciphertext appears random.

  • Public evaluation key: A public evaluation key is used to perform homomorphic operations on encrypted data, typically by a server.

• Guaranteed correctness of encrypted computations: To ensure security, TFHE adds random noise to ciphertexts. Depending on the noise parameters, it can cause errors during encrypted data processing. By default, Concrete ML uses parameters that guarantee the correctness of encrypted computations, so the results on encrypted data equals to those of simulations on clear data.

Model accuracy considerations under FHE constraints

FHE requires all inputs, constants, and intermediate values to be integers of maximum 16 bits. To make machine learning models compatible with FHE, Concrete ML implements some techniques with accuracy considerations:

This document introduces the simple built-in neural networks models that Concrete ML provides with a scikit-learn interface through the NeuralNetClassifier and NeuralNetRegressor classes.

Supported models

The neural network models are implemented with , which provides a scikit-learn-like interface to Torch models (more ).

Concrete ML models are multi-layer, fully-connected, networks with customizable activation functions and have a number of neurons in each layer. This approach is similar to what is available in scikit-learn when using the MLPClassifier/MLPRegressor classes. The built-in models train easily with a single call to .fit(), which will automatically quantize weights and activations. These models use Quantization Aware Training, allowing good performance for low precision (down to 2-3 bits) weights and activations.

While NeuralNetClassifier and NeuralNetClassifier provide scikit-learn-like models, their architecture is somewhat restricted to make training easy and robust. If you need more advanced models, you can convert custom neural networks as described in the .

Example

To create an instance of a Fully Connected Neural Network (FCNN), you need to instantiate one of the NeuralNetClassifier and NeuralNetRegressor classes and configure a number of parameters that are passed to their constructor.

Note that some parameters need to be prefixed by module__, while others don't. The parameters related to the model must have the prefix, such as the underlying nn.Module. The parameters related to training options do not require the prefix.

The folowing figure shows the Concrete ML neural network trained with Quantization Aware Training in an FHE-compatible configuration and compares it to the floating-point equivalent trained with scikit-learn.

Architecture parameters

• module__n_layers: number of layers in the FCNN.

  • This parameter must be at least 1. Note that this is the total number of layers. For a single, hidden layer NN model, set module__n_layers=2

• module__activation_function: can be one of the Torch activations (such as nn.ReLU)

  • Neural networks with nn.ReLU activation benefit from specific optimizations that make them around 10x faster than networks with other activation functions.

Quantization parameters

• n_w_bits (default 3): number of bits for weights

• n_a_bits (default 3): number of bits for activations and inputs

• n_accum_bits: maximum accumulator bit-width that is desired
• power_of_two_scaling (default True): forces quantization scales to be powers-of-two

  • When coupled with the ReLU activation, this optimize strongly the FHE inference time.

Training parameters (from skorch)

• max_epochs (default 10): The number of epochs to train the network

• verbose (default: False): Whether to log loss/metrics during training

• lr (default 0.001): Learning rate

Advanced parameters

• module__n_hidden_neurons_multiplier (default 4): The number of hidden neurons.

  • This parameter will be automatically set proportional to the dimensionality of the input. It controls the proportionality factor. This value gives good accuracy while avoiding accumulator overflow.

Class weights

You can give weights to each class to use in training. Note that this must be supported by the underlying PyTorch loss function.

Overflow errors

The n_accum_bits parameter influences training accuracy by controlling the number of non-zero neurons allowed in each layer. You can increase n_accum_bits to improve accuracy, but must consider the precision limitations to avoid an overflow in the accumulator. The default value is a balanced choice that generally avoids overflow, but you may need to adjust it to reduce the network breadth if you encounter overflow errors.

The number of neurons in intermediate layers is controlled by the n_hidden_neurons_multiplier parameter. A value of 1 makes intermediate layers have the same number of neurons as the number as the input data dimensions.

This document illustrate how Concrete ML model and DataFrames are deployed in client/server setting when creating privacy-preserving services in the cloud.

Once compiled to FHE, a Concrete ML model or DataFrame generates machine code that execute prediction, training or pre-processing on encrypted data. During this process, Concrete ML generates and .

Communication protocols

The overall communications protocol to enable cloud deployment of machine learning services can be summarized in the following diagram:

The steps detailed above are:

1. Model Deployment: The model developer deploys the compiled machine learning model to the server. This model includes the cryptographic parameters. The server is now ready to provide private inference. Cryptographic parameters and compiled programs for DataFrames are included directly in Concrete ML.

2. Client request: The client requests the cryptographic parameters (client specs). Once the client receives them from the server, the secret and evaluation keys are generated.

3. Key exchanges: The client sends the evaluation key to the server. The server is now ready to accept requests from this client. The client sends their encrypted data. Serialized DataFrames include client evaluation keys.

4. Private inference: The server uses the evaluation key to securely run prediction, training and pre-processing on the user's data and sends back the encrypted result.

5. Decryption: The client now decrypts the result and can send back new requests.

This document introduces the nearest neighbors non-parametric classification models that Concrete ML provides with a scikit-learn interface through the KNeighborsClassifier class.

Example

Quantization parameters

The KNeighborsClassifier class quantizes the training data-set provided to .fit using the specified number of bits (n_bits). To comply with , you must keep this value low. The model's accuracy will depend significantly on a well-chosen n_bits value and the dimensionality of the data.

The predict method of the KNeighborsClassifier performs the following steps:

1. Quantize the test vectors on clear data

2. Compute the top-k class indices of the closest training set vector on encrypted data

3. Vote for the top-k class labels to find the class for each test vector, performed on clear data

Inference time considerations

The FHE inference latency of this model is heavily influenced by the n_bits and the dimensionality of the data. Additionally, the data-set size has a linear impact on the data complexity. The number of nearest neighbors (n_neighbors) also affects performance.

The KNN computation executes in FHE in steps, where is the training data-set size and is n_neighbors. Each step requires several , with their runtime affected by the factors listed above. These factors determine the precision needed to represent the distances between test vectors and training data-set vectors. The PBS input precision required by the circuit is related to the precision of the distance values.

This guide provides a complete example of converting a PyTorch neural network into its FHE-friendly, quantized counterpart. It focuses on Quantization Aware Training a simple network on a synthetic data-set.

In general, quantization can be carried out in two different ways: either during Quantization Aware Training (QAT) or after the training phase with Post-Training Quantization (PTQ).

Regarding FHE-friendly neural networks, QAT is the best way to reach optimal accuracy under FHE constraints. This technique allows weights and activations to be reduced to very low bit-widths (e.g., 2-3 bits), which, combined with pruning, can keep accumulator bit-widths low.

Concrete ML uses the third-party library Brevitas to perform QAT for PyTorch NNs, but options exist for other frameworks such as Keras/Tensorflow.

Several demos and tutorials that use Brevitas are available in the Concrete ML library, such as the CIFAR classification tutorial.

This guide is based on a notebook tutorial, from which some code blocks are documented.

For a more formal description of the usage of Brevitas to build FHE-compatible neural networks, please see the Brevitas usage reference.

Baseline PyTorch model

In PyTorch, using standard layers, a fully connected neural network (FCNN) would look like this:

import torch
from torch import nn

IN_FEAT = 2
OUT_FEAT = 2

class SimpleNet(nn.Module):
    """Simple MLP with PyTorch"""

    def __init__(self, n_hidden = 30):
        super().__init__()
        self.fc1 = nn.Linear(in_features=IN_FEAT, out_features=n_hidden)
        self.fc2 = nn.Linear(in_features=n_hidden, out_features=n_hidden)
        self.fc3 = nn.Linear(in_features=n_hidden, out_features=OUT_FEAT)


    def forward(self, x):
        """Forward pass."""
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = self.fc3(x)
        return x

The notebook tutorial, example shows how to train a FCNN, similarly to the one above, on a synthetic 2D data-set with a checkerboard grid pattern of 100 x 100 points. The data is split into 9500 training and 500 test samples.

Once trained, this PyTorch network can be imported using the compile_torch_model function. This function uses simple PTQ.

The network was trained using different numbers of neurons in the hidden layers, and quantized using 3-bits weights and activations. The mean accumulator size shown below is measured as the mean over 10 runs of the experiment. An accumulator of 6.6 means that 4 times out of 10 the accumulator measured was 6 bits while 6 times it was 7 bits.

This shows that the fp32 accuracy and accumulator size increases with the number of hidden neurons, while the 3-bits accuracy remains low irrespective of the number of neurons. While all the configurations tried here were FHE-compatible (accumulator < 16 bits), it is often preferable to have a lower accumulator size in order to speed up inference time.

Quantization Aware Training:

Quantization Aware Training using Brevitas is the best way to guarantee a good accuracy for Concrete ML compatible neural networks.

Brevitas provides a quantized version of almost all PyTorch layers (Linear layer becomes QuantLinear, ReLU layer becomes QuantReLU and so on), plus some extra quantization parameters, such as :

• bit_width: precision quantization bits for activations

• act_quant: quantization protocol for the activations

• weight_bit_width: precision quantization bits for weights

• weight_quant: quantization protocol for the weights

In order to use FHE, the network must be quantized from end to end, and thanks to the Brevitas's QuantIdentity layer, it is possible to quantize the input by placing it at the entry point of the network. Moreover, it is also possible to combine PyTorch and Brevitas layers, provided that a QuantIdentity is placed after this PyTorch layer. The following table gives the replacements to be made to convert a PyTorch NN for Concrete ML compatibility.

Some PyTorch operators (from the PyTorch functional API), require a brevitas.quant.QuantIdentity to be applied on their inputs.

With Brevitas, the network above becomes:

from brevitas import nn as qnn
from brevitas.core.quant import QuantType
from brevitas.quant import Int8ActPerTensorFloat, Int8WeightPerTensorFloat

N_BITS = 3
IN_FEAT = 2
OUT_FEAT = 2

class QuantSimpleNet(nn.Module):
    def __init__(
        self,
        n_hidden,
        qlinear_args={
            "weight_bit_width": N_BITS,
            "weight_quant": Int8WeightPerTensorFloat,
            "bias": True,
            "bias_quant": None,
            "narrow_range": True
        },
        qidentity_args={"bit_width": N_BITS, "act_quant": Int8ActPerTensorFloat},
    ):
        super().__init__()

        self.quant_inp = qnn.QuantIdentity(**qidentity_args)
        self.fc1 = qnn.QuantLinear(IN_FEAT, n_hidden, **qlinear_args)
        self.relu1 = qnn.QuantReLU(bit_width=qidentity_args["bit_width"])
        self.fc2 = qnn.QuantLinear(n_hidden, n_hidden, **qlinear_args)
        self.relu2 = qnn.QuantReLU(bit_width=qidentity_args["bit_width"])
        self.fc3 = qnn.QuantLinear(n_hidden, OUT_FEAT, **qlinear_args)

        for m in self.modules():
            if isinstance(m, qnn.QuantLinear):
                torch.nn.init.uniform_(m.weight.data, -1, 1)

    def forward(self, x):
        x = self.quant_inp(x)
        x = self.relu1(self.fc1(x))
        x = self.relu2(self.fc2(x))
        x = self.fc3(x)
        return x       

Training this network with pruning (see below) with 30 out of 100 total non-zero neurons gives good accuracy while keeping the accumulator size low.

Pruning using Torch

Considering that FHE only works with limited integer precision, there is a risk of overflowing in the accumulator, which will make Concrete ML raise an error.

To understand how to overcome this limitation, consider a scenario where 2 bits are used for weights and layer inputs/outputs. The Linear layer computes a dot product between weights and inputs $y = \sum_i w_i x_i$. With 2 bits, no overflow can occur during the computation of the Linear layer as long the number of neurons does not exceed 14, as in the sum of 14 products of 2-bits numbers does not exceed 7 bits.

By default, Concrete ML uses symmetric quantization for model weights, with values in the interval $\left[-2^{n_{bits}-1}, 2^{n_{bits}-1}-1\right]$. For example, for $n_{bits}=2$ the possible values are $[-2, -1, 0, 1]$; for $n_{bits}=3$, the values can be $[-4,-3,-2,-1,0,1,2,3]$.

In a typical setting, the weights will not all have the maximum or minimum values (e.g., $-2^{n_{bits}-1}$). Weights typically have a normal distribution around 0, which is one of the motivating factors for their symmetric quantization. A symmetric distribution and many zero-valued weights are desirable because opposite sign weights can cancel each other out and zero weights do not increase the accumulator size.

This fact can be leveraged to train a network with more neurons, while not overflowing the accumulator, using a technique called pruning where the developer can impose a number of zero-valued weights. Torch provides support for pruning out of the box.

The following code shows how to use pruning in the previous example:

import torch.nn.utils.prune as prune

class PrunedQuantNet(SimpleNet):
    """Simple MLP with PyTorch"""

    pruned_layers = set()

    def prune(self, max_non_zero):
        # Linear layer weight has dimensions NumOutputs x NumInputs
        for name, layer in self.named_modules():
            if isinstance(layer, nn.Linear):
                print(name, layer)
                num_zero_weights = (layer.weight.shape[1] - max_non_zero) * layer.weight.shape[0]
                if num_zero_weights <= 0:
                    continue
                print(f"Pruning layer {name} factor {num_zero_weights}")
                prune.l1_unstructured(layer, "weight", amount=num_zero_weights)
                self.pruned_layers.add(name)

    def unprune(self):
        for name, layer in self.named_modules():
            if name in self.pruned_layers:
                prune.remove(layer, "weight")
                self.pruned_layers.remove(name)

Results with PrunedQuantNet, a pruned version of the QuantSimpleNet with 100 neurons on the hidden layers, are given below, showing a mean accumulator size measured over 10 runs of the experiment:

This shows that the fp32 accuracy has been improved while maintaining constant mean accumulator size.

When pruning a larger neural network during training, it is easier to obtain a low bit-width accumulator while maintaining better final accuracy. Thus, pruning is more robust than training a similar, smaller network.

This section provides a set of tools and guidelines to help users build optimized FHE-compatible models. It discusses FHE simulation, the key-cache functionality that helps speed-up FHE result debugging, and gives a guide to evaluate circuit complexity.

Simulation

The simulation functionality of Concrete ML provides a way to evaluate, using clear data, the results that ML models produce on encrypted data. The simulation includes any probabilistic behavior FHE may induce. The simulation is implemented with Concrete's simulation.

The simulation mode can be useful when developing and iterating on an ML model implementation. As FHE non-linear models work with integers up to 16 bits, with a trade-off between the number of bits and the FHE execution speed, the simulation can help to find the optimal model design.

Simulation is much faster than FHE execution. This allows for faster debugging and model optimization. For example, this was used for the red/blue contours in the Classifier Comparison notebook, as computing in FHE for the whole grid and all the classifiers would take significant time.

The following example shows how to use the simulation mode in Concrete ML.

from sklearn.datasets import fetch_openml, make_circles
from concrete.ml.sklearn import RandomForestClassifier

n_bits = 2
X, y = make_circles(n_samples=1000, noise=0.1, factor=0.6, random_state=0)
concrete_clf = RandomForestClassifier(
    n_bits=n_bits, n_estimators=10, max_depth=5
)
concrete_clf.fit(X, y)

concrete_clf.compile(X)

# Running the model using FHE-simulation
y_preds_clear = concrete_clf.predict(X, fhe="simulate")

Caching keys during debugging

It is possible to avoid re-generating the keys of the models you are debugging. This feature is unsafe and should not be used in production. Here is an example that shows how to enable key-caching:

from sklearn.datasets import fetch_openml, make_circles
from concrete.ml.sklearn import RandomForestClassifier
from concrete.fhe import Configuration
debug_config = Configuration(
    enable_unsafe_features=True,
    use_insecure_key_cache=True,
    insecure_key_cache_location="~/.cml_keycache",
)

n_bits = 2
X, y = make_circles(n_samples=1000, noise=0.1, factor=0.6, random_state=0)
concrete_clf = RandomForestClassifier(
    n_bits=n_bits, n_estimators=10, max_depth=5
)
concrete_clf.fit(X, y)

concrete_clf.compile(X, debug_config)

Common compilation errors

1. TLU input maximum bit-width is exceeded

Error message: this [N]-bit value is used as an input to a table lookup

Cause: This error can occur when rounding_threshold_bits is not used and accumulated intermediate values in the computation exceed 16 bits.

Possible solutions:

• Reduce quantization n_bits. However, this may reduce accuracy. When quantization n_bits must be below 6, it is best to use Quantization Aware Training.

• Use rounding_threshold_bits. This feature is described here. It is recommended to use the fhe.Exactness.APPROXIMATE setting, and set the rounding bits to 1 or 2 bits higher than the quantization n_bits

• Use pruning

2. No crypto-parameters can be found

Error message: RuntimeError: NoParametersFound

Cause: This error occurs when using rounding_threshold_bits in the compile_torch_model function.

Possible solutions: The solutions in this case are similar to the ones for the previous error.

3. Quantization import failed

Error message: Error occurred during quantization aware training (QAT) import [...] Could not determine a unique scale for the quantization!.

Cause: This error occurs when the model imported as a quantized-aware training model lacks quantization operators. See this guide on how to use Brevitas layers. This error message indicates that some layers do not take inputs quantized through QuantIdentity layers.

A common example is related to the concatenation operator. Suppose two tensors x and y are produced by two layers and need to be concatenated:

x = self.dense1(x)
y = self.dense2(y)
z = torch.cat([x, y])

In the example above, the x and y layers need quantization before being concatenated.

Possible solutions:

1. If the error occurs for the first layer of the model: Add a QuantIdentity layer in your model and apply it on the input of the forward function, before the first layer is computed.

2. If the error occurs for a concatenation or addition layer: Add a new QuantIdentity layer in your model. Suppose it is called quant_concat. In the forward function, before concatenation of x and y, apply it to both tensors that are concatenated. The usage of a common Quantidentity layer to quantize both tensors that are concatenated ensures that they have the same scale:

z = torch.cat([self.quant_concat(x), self.quant_concat(y)])

Debugging compilation errors

Compilation errors due to FHE incompatible models, such as maximum bit-width exceeded or NoParametersFound can be debugged by examining the bit-widths associated with various intermediate values of the FHE computation.

The following produces a neural network that is not FHE-compatible:

import numpy
import torch

from torch import nn
from concrete.ml.torch.compile import compile_torch_model

N_FEAT = 2
class SimpleNet(nn.Module):
    """Simple MLP with PyTorch"""

    def __init__(self, n_hidden=30):
        super().__init__()
        self.fc1 = nn.Linear(in_features=N_FEAT, out_features=n_hidden)
        self.fc2 = nn.Linear(in_features=n_hidden, out_features=n_hidden)
        self.fc3 = nn.Linear(in_features=n_hidden, out_features=2)


    def forward(self, x):
        """Forward pass."""
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = self.fc3(x)
        return x


torch_input = torch.randn(100, N_FEAT)
torch_model = SimpleNet(120)
try:
    quantized_numpy_module = compile_torch_model(
        torch_model,
        torch_input,
        n_bits=7,
    )
except RuntimeError as err:
    print(err)

Upon execution, the Compiler will raise the following error within the graph representation:

Function you are trying to compile cannot be compiled:

%0 = _x                               # EncryptedTensor<int7, shape=(1, 2)>           ∈ [-64, 63]
%1 = [[ -9  18  ...   30  34]]        # ClearTensor<int7, shape=(2, 120)>             ∈ [-62, 63]              @ /fc1/Gemm.matmul
%2 = matmul(%0, %1)                   # EncryptedTensor<int14, shape=(1, 120)>        ∈ [-5834, 5770]          @ /fc1/Gemm.matmul
%3 = subgraph(%2)                     # EncryptedTensor<uint7, shape=(1, 120)>        ∈ [0, 127]
%4 = [[-36   6  ...   27 -11]]        # ClearTensor<int7, shape=(120, 120)>           ∈ [-63, 63]              @ /fc2/Gemm.matmul
%5 = matmul(%3, %4)                   # EncryptedTensor<int17, shape=(1, 120)>        ∈ [-34666, 37702]        @ /fc2/Gemm.matmul
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this 17-bit value is used as an input to a table lookup

The error this 17-bit value is used as an input to a table lookup indicates that the 16-bit limit on the input of the Table Lookup (TLU) operation has been exceeded. To pinpoint the model layer that causes the error, Concrete ML provides the bitwidth_and_range_report helper function. First, the model must be compiled so that it can be simulated.

On the other hand, NoParametersFound is encountered when using rounding_threshold_bits. When using this setting, the 16-bit accumulator limit is relaxed. However, reducing bit-width, or reducing the rounding_threshold_bits, or using using the fhe.Exactness.APPROXIMATE rounding method can help.

Fixing compilation errors

To make this network FHE-compatible one can apply several techniques:

1. use rounded accumulators by specifying the rounding_threshold_bits parameter. Please evaluate the accuracy of the model using simulation if you use this feature, as it may impact accuracy. Setting a value 2-bit higher than the quantization n_bits should be a good start.

torch_model = SimpleNet(20)

quantized_numpy_module = compile_torch_model(
    torch_model,
    torch_input,
    n_bits=6,
    rounding_threshold_bits=7,
)

1. reduce the accumulator bit-width of the second layer named fc2. To do this, a simple solution is to reduce the number of neurons, as it is proportional to the bit-width.

torch_model = SimpleNet(10)

quantized_numpy_module = compile_torch_model(
    torch_model,
    torch_input,
    n_bits=7,
)

1. adjust the tolerance for one-off errors using the p_error parameter. See this section for more explanation on this tolerance.

torch_model = SimpleNet(10)

quantized_numpy_module = compile_torch_model(
    torch_model,
    torch_input,
    n_bits=7,
    p_error=0.01
)

Complexity analysis

In FHE, univariate functions are encoded as table lookups, which are then implemented using Programmable Bootstrapping (PBS). PBS is a powerful technique but will require significantly more computing resources, and thus time, compared to simpler encrypted operations such as matrix multiplications, convolution, or additions.

Furthermore, the cost of PBS will depend on the bit-width of the compiled circuit. Every additional bit in the maximum bit-width raises the complexity of the PBS by a significant factor. It may be of interest to the model developer, then, to determine the bit-width of the circuit and the amount of PBS it performs.

This can be done by inspecting the MLIR code produced by the Compiler:

print(quantized_numpy_module.fhe_circuit.mlir)

MLIR
--------------------------------------------------------------------------------
module {
  func.func @main(%arg0: tensor<1x2x!FHE.eint<15>>) -> tensor<1x2x!FHE.eint<15>> {
    %cst = arith.constant dense<16384> : tensor<1xi16>
    %0 = "FHELinalg.sub_eint_int"(%arg0, %cst) : (tensor<1x2x!FHE.eint<15>>, tensor<1xi16>) -> tensor<1x2x!FHE.eint<15>>
    %cst_0 = arith.constant dense<[[-13, 43], [-31, 63], [1, -44], [-61, 20], [31, 2]]> : tensor<5x2xi16>
    %cst_1 = arith.constant dense<[[-45, 57, 19, 50, -63], [32, 37, 2, 52, -60], [-41, 25, -1, 31, -26], [-51, -40, -53, 0, 4], [20, -25, 56, 54, -23]]> : tensor<5x5xi16>
    %cst_2 = arith.constant dense<[[-56, -50, 57, 37, -22], [14, -1, 57, -63, 3]]> : tensor<2x5xi16>
    %c16384_i16 = arith.constant 16384 : i16
    %1 = "FHELinalg.matmul_eint_int"(%0, %cst_2) : (tensor<1x2x!FHE.eint<15>>, tensor<2x5xi16>) -> tensor<1x5x!FHE.eint<15>>
    %cst_3 = tensor.from_elements %c16384_i16 : tensor<1xi16>
    %cst_4 = tensor.from_elements %c16384_i16 : tensor<1xi16>
    %2 = "FHELinalg.add_eint_int"(%1, %cst_4) : (tensor<1x5x!FHE.eint<15>>, tensor<1xi16>) -> tensor<1x5x!FHE.eint<15>>
    %cst_5 = arith.constant

: tensor<5x32768xi64>
    %cst_6 = arith.constant dense<[[0, 1, 2, 3, 4]]> : tensor<1x5xindex>
    %3 = "FHELinalg.apply_mapped_lookup_table"(%2, %cst_5, %cst_6) : (tensor<1x5x!FHE.eint<15>>, tensor<5x32768xi64>, tensor<1x5xindex>) -> tensor<1x5x!FHE.eint<15>>
    %4 = "FHELinalg.matmul_eint_int"(%3, %cst_1) : (tensor<1x5x!FHE.eint<15>>, tensor<5x5xi16>) -> tensor<1x5x!FHE.eint<15>>
    %5 = "FHELinalg.add_eint_int"(%4, %cst_3) : (tensor<1x5x!FHE.eint<15>>, tensor<1xi16>) -> tensor<1x5x!FHE.eint<15>>
    %cst_7 = arith.constant

: tensor<5x32768xi64>
    %6 = "FHELinalg.apply_mapped_lookup_table"(%5, %cst_7, %cst_6) : (tensor<1x5x!FHE.eint<15>>, tensor<5x32768xi64>, tensor<1x5xindex>) -> tensor<1x5x!FHE.eint<15>>
    %7 = "FHELinalg.matmul_eint_int"(%6, %cst_0) : (tensor<1x5x!FHE.eint<15>>, tensor<5x2xi16>) -> tensor<1x2x!FHE.eint<15>>
    return %7 : tensor<1x2x!FHE.eint<15>>

  }
}
--------------------------------------------------------------------------------

There are several calls to FHELinalg.apply_mapped_lookup_table and FHELinalg.apply_lookup_table. These calls apply PBS to the cells of their input tensors. Their inputs in the listing above are: tensor<1x2x!FHE.eint<8>> for the first and last call and tensor<1x50x!FHE.eint<8>> for the two calls in the middle. Thus, PBS is applied 104 times.

Retrieving the bit-width of the circuit is then simply:

print(quantized_numpy_module.fhe_circuit.graph.maximum_integer_bit_width())

Decreasing the number of bits and the number of PBS applications induces large reductions in the computation time of the compiled circuit.

This document explains how to train SGD Logistic Regression on encrypted data.

Training on encrypted data is done through an FHE program that is generated by Concrete ML, based on the characteristics of the data that are given to the fit function. Once the FHE program associated with the SGDClassifier object has fit the encrypted data, it performs specifically to that data's distribution and dimensionality.

When deploying encrypted training services, you need to consider the type of data that future users of your services will train on:

• The distribution of the data should match to achieve good accuracy

• The dimensionality of the data needs to match since the deployed FHE programs are compiled for a fixed number of dimensions.

See the deployment section for more details.

Example

The logistic regression training example shows logistic regression training on encrypted data in action.

The following snippet shows how to instantiate a logistic regression model that trains on encrypted data:

from concrete.ml.sklearn import SGDClassifier
parameters_range = (-1.0, 1.0)

model = SGDClassifier(
    random_state=42,
    max_iter=50,
    fit_encrypted=True,
    parameters_range=parameters_range,
)

To activate encrypted training, simply set fit_encrypted=True in the constructor. When the value is set, Concrete ML generates an FHE program which, when called through the fit function, processes encrypted training data, labels and initial weights and outputs trained model weights. If this value is not set, training is performed on clear data using scikit-learn gradient descent.

Next, to perform the training on encrypted data, call the fit function with the fhe="execute" argument:

model.fit(X_binary, y_binary, fhe="execute")

Training configuration

The max_iter parameter controls the number of batches that are processed by the training algorithm.

The parameters_range parameter determines the initialization of the coefficients and the bias of the logistic regression. It is recommended to give values that are close to the min/max of the training data. It is also possible to normalize the training data so that it lies in the range $[-1, 1]$.

Capabilities and Limitations

The trainable logistic model uses Stochastic Gradient Descent (SGD) and quantizes the data, weights, gradients and the error measure. It currently supports training 6-bit models, including g both the coefficients and the bias.

The SGDClassifier does not currently support training models with other bit-width values. The execution time to train a model is proportional to the number of features and the number of training examples in the batch. The SGDClassifier training does not currently support client/server deployment for training.

Deployment

Once you have tested an SGDClassifier that trains on encrypted data, you can build an FHE training service by deploying the FHE training program of the SGDClassifier. See the Production Deloyment page for more details on how to the Concrete ML deployment utility classes. To deploy an FHE training program, you must pass the mode='training' parameter to the FHEModelDev class.

Concrete ML has APIs that make it easy, during model development and testing, to perform encryption, execution in FHE, and decryption in a single step. For more control, these individual steps can be executed separately. The APIs used to accomplish this are different for:

• Built-in models

• Custom models

Built-in models

The following example shows how to create a synthetic data-set and how to use it to train a LogisticRegression model from Concrete ML. Next, we will discuss the dedicated functions for encryption, inference, and decryption.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from concrete.ml.sklearn import LogisticRegression
import numpy

# Create a synthetic data-set for a classification problem
x, y = make_classification(n_samples=100, class_sep=2, n_features=3, n_informative=3, n_redundant=0, random_state=42)

# Split the data-set into a train and test set
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=42)

# Instantiate and train the model
model = LogisticRegression()
model.fit(x_train,y_train)

# Simulate the predictions in the clear (optional)
y_pred_clear = model.predict(x_test)

# Compile the model on a representative set
fhe_circuit = model.compile(x_train)

All Concrete ML built-in models have a monolithic predict method that performs the encryption, FHE execution, and decryption with a single function call. Concrete ML models follow the same API as scikit-learn models, transparently performing the steps related to encryption for convenience.

# Predict in FHE
y_pred_fhe = model.predict(x_test, fhe="execute")

Regarding this LogisticRegression model, as with scikit-learn, it is possible to predict the logits as well as the class probabilities by respectively using the decision_function or predict_proba methods instead.

Alternatively, it is possible to execute all main steps (key generation, quantization, encryption, FHE execution, decryption) separately.

# Generate the keys (set force to True in order to generate new keys at each execution)
fhe_circuit.keygen(force=True)

y_pred_fhe_step = []

for f_input in x_test:
    # Quantize an input (float)
    q_input = model.quantize_input([f_input])
    
    # Encrypt the input
    q_input_enc = fhe_circuit.encrypt(q_input)

    # Execute the linear product in FHE 
    q_y_enc = fhe_circuit.run(q_input_enc)

    # Decrypt the result (integer)
    q_y = fhe_circuit.decrypt(q_y_enc)

    # De-quantize the result
    y = model.dequantize_output(q_y)

    # Apply either the sigmoid if it is a binary classification task, which is the case in this 
    # example, or a softmax function in order to get the probabilities (in the clear)
    y_proba = model.post_processing(y)

    # Since this model does classification, apply the argmax to get the class predictions (in the clear)
    # Note that regression models won't need the following line
    y_class = numpy.argmax(y_proba, axis=1)

    y_pred_fhe_step += list(y_class)

y_pred_fhe_step = numpy.array(y_pred_fhe_step)

print("Predictions in clear:", y_pred_clear)
print("Predictions in FHE  :", y_pred_fhe_step)
print(f"Similarity: {int((y_pred_fhe_step == y_pred_clear).mean()*100)}%")

Custom models

For custom models, the API to execute inference in FHE or simulation is illustrated as:

from torch import nn
from brevitas import nn as qnn
from concrete.ml.torch.compile import compile_brevitas_qat_model

class FCSmall(nn.Module):
    """A small QAT NN."""

    def __init__(self, input_output):
        super().__init__()
        self.quant_input = qnn.QuantIdentity(bit_width=3)
        self.fc1 = qnn.QuantLinear(in_features=input_output, out_features=input_output, weight_bit_width=3, bias=True)
        self.act_f = nn.ReLU()
        self.fc2 = qnn.QuantLinear(in_features=input_output, out_features=input_output, weight_bit_width=3, bias=True)

    def forward(self, x):
        return self.fc2(self.act_f(self.fc1(self.quant_input(x))))

torch_model = FCSmall(3)

quantized_module = compile_brevitas_qat_model(
    torch_model,
    x_train,
)

x_test_q = quantized_module.quantize_input(x_test)
y_pred = quantized_module.quantized_forward(x_test_q, fhe="simulate")
y_pred = quantized_module.dequantize_output(y_pred)

y_pred = numpy.argmax(y_pred, axis=1)

Neural networks pose unique challenges with regards to encrypted inference. Each neuron in a network applies an activation function that requires a PBS operation. The latency of a single PBS depends on the bit-width of the input of the PBS.

Several approaches can be used to reduce the overall latency of a neural network.

Circuit bit-width optimization

Quantization Aware Training and pruning introduce specific hyper-parameters that influence the accumulator sizes. It is possible to chose quantization and pruning configurations that reduce the accumulator size. A trade-off between latency and accuracy can be obtained by varying these hyper-parameters as described in the deep learning design guide.

Structured pruning

While un-structured pruning is used to ensure the accumulator bit-width stays low, structured pruning can eliminate entire neurons from the network. Many neural networks are over-parametrized (since this enables easier training) and some neurons can be removed. Structured pruning, applied to a trained network as a fine-tuning step, can be applied to built-in neural networks using the prune helper function as shown in this example. To apply structured pruning to custom models, it is recommended to use the torch-pruning package.

Rounded activations and quantizers

Reducing the bit-width of the inputs to the Table Lookup (TLU) operations is a major source of improvements in the latency. Post-training, it is possible to leverage some properties of the fused activation and quantization functions expressed in the TLUs to further reduce the accumulator. This is achieved through the rounded PBS feature as described in the rounded activations and quantizers reference. Adjusting the rounding amount, relative to the initial accumulator size, can bring large improvements in latency while maintaining accuracy.

TLU error tolerance adjustment

Finally, the TFHE scheme exposes a TLU error tolerance parameter that has an impact on crypto-system parameters that influence latency. A higher tolerance of TLU off-by-one errors results in faster computations but may reduce accuracy. One can think of the error of obtaining $T[x]$ as a Gaussian distribution centered on $x$: $TLU[x]$ is obtained with probability of 1 - p_error, while $T[x-1]$, $T[x+1]$ are obtained with much lower probability, etc. In Deep NNs, these type of errors can be tolerated up to some point. See the p_error documentation for details and more specifically the usage example of the API for finding the best p_error.

In addition to Concrete ML models and custom models in torch, it is also possible to directly compile ONNX models. This can be particularly appealing, notably to import models trained with Keras.

ONNX models can be compiled by directly importing models that are already quantized with Quantization Aware Training (QAT) or by performing Post-Training Quantization (PTQ) with Concrete ML.

Simple example

The following example shows how to compile an ONNX model using PTQ. The model was initially trained using Keras before being exported to ONNX. The training code is not shown here.

import numpy
import onnx
import tensorflow
import tf2onnx

from concrete.ml.torch.compile import compile_onnx_model
from concrete.fhe.compilation import Configuration


class FC(tensorflow.keras.Model):
    """A fully-connected model."""

    def __init__(self):
        super().__init__()
        hidden_layer_size = 10
        output_size = 5

        self.dense1 = tensorflow.keras.layers.Dense(
            hidden_layer_size,
            activation=tensorflow.nn.relu,
        )
        self.dense2 = tensorflow.keras.layers.Dense(output_size, activation=tensorflow.nn.relu6)
        self.flatten = tensorflow.keras.layers.Flatten()

    def call(self, inputs):
        """Forward function."""
        x = self.flatten(inputs)
        x = self.dense1(x)
        x = self.dense2(x)
        return self.flatten(x)


n_bits = 6
input_output_feature = 2
input_shape = (input_output_feature,)
num_inputs = 1
n_examples = 5000

# Define the Keras model
keras_model = FC()
keras_model.build((None,) + input_shape)
keras_model.compute_output_shape(input_shape=(None, input_output_feature))

# Create random input
input_set = numpy.random.uniform(-100, 100, size=(n_examples, *input_shape))

# Convert to ONNX
tf2onnx.convert.from_keras(keras_model, opset=14, output_path="tmp.model.onnx")

onnx_model = onnx.load("tmp.model.onnx")
onnx.checker.check_model(onnx_model)

# Compile
quantized_module = compile_onnx_model(
    onnx_model, input_set, n_bits=2
)

# Create test data from the same distribution and quantize using
# learned quantization parameters during compilation
x_test = tuple(numpy.random.uniform(-100, 100, size=(1, *input_shape)) for _ in range(num_inputs))

y_clear = quantized_module.forward(*x_test, fhe="disable")
y_fhe = quantized_module.forward(*x_test, fhe="execute")

print("Execution in clear: ", y_clear)
print("Execution in FHE:   ", y_fhe)
print("Equality:           ", numpy.sum(y_clear == y_fhe), "over", numpy.size(y_fhe), "values")

Quantization Aware Training

Models trained using Quantization Aware Training contain quantizers in the ONNX graph. These quantizers ensure that the inputs to the Linear/Dense and Conv layers are quantized. Since these QAT models have quantizers that are configured during training to a specific number of bits, the ONNX graph will need to be imported using the same settings:

# Define the number of bits to use for quantizing weights and activations during training
n_bits_qat = 3  

quantized_numpy_module = compile_onnx_model(
    onnx_model,
    input_set,
    import_qat=True,
    n_bits=n_bits_qat,
)

Supported operators

The following operators are supported for evaluation and conversion to an equivalent FHE circuit. Other operators were not implemented, either due to FHE constraints or because they are rarely used in PyTorch activations or scikit-learn models.

• Abs

• Acos

• Acosh

• Add

• Asin

• Asinh

• Atan

• Atanh

• AveragePool

• BatchNormalization

• Cast

• Celu

• Clip

• Concat

• Constant

• ConstantOfShape

• Conv

• Cos

• Cosh

• Div

• Elu

• Equal

• Erf

• Exp

• Expand

• Flatten

• Floor

• Gather

• Gemm

• Greater

• GreaterOrEqual

• HardSigmoid

• HardSwish

• Identity

• LeakyRelu

• Less

• LessOrEqual

• Log

• MatMul

• Max

• MaxPool

• Min

• Mul

• Neg

• Not

• Or

• PRelu

• Pad

• Pow

• ReduceSum

• Relu

• Reshape

• Round

• Selu

• Shape

• Sigmoid

• Sign

• Sin

• Sinh

• Slice

• Softplus

• Squeeze

• Sub

• Tan

• Tanh

• ThresholdedRelu

• Transpose

• Unfold

• Unsqueeze

• Where

• onnx.brevitas.Quant

This document introduces how to construct and perform operations on encrypted DataFrames using Fully Homomorphic Encryption (FHE).

Introduction

Encrypted DataFrames are a storage format for encrypted tabular data. You can exchange encrypted DataFrames with third parties to collaborate without privacy risks. Potential applications include:

• Encrypt storage of tabular data-sets

• Joint data analysis efforts between multiple parties

• Data preparation steps before machine learning tasks, such as inference or training

• Secure outsourcing of data analysis to untrusted third parties

Encryption and decryption

To encrypt a pandas DataFrame, construct a ClientEngine that manages keys and then call the encrypt_from_pandas function:

from concrete.ml.pandas import ClientEngine
from io import StringIO
import pandas

data_left = """index,total_bill,tip,sex,smoker
1,12.54,2.5,Male,No
2,11.17,1.5,Female,No
3,20.29,2.75,Female,No
"""

# Load your pandas DataFrame
df = pandas.read_csv(StringIO(data_left))

# Obtain client object
client = ClientEngine(keys_path="my_keys")

# Encrypt the DataFrame
df_encrypted = client.encrypt_from_pandas(df)

# Decrypt the DataFrame to produce a pandas DataFrame
df_decrypted = client.decrypt_to_pandas(df_encrypted)

Supported data types and schema definition

• Integer: Integers are supported within a specific range determined by the encryption scheme's quantization parameters. Default range is 1 to 15. 0 being used for the NaN. Values outside this range will cause a ValueError to be raised during the pre-processing stage.

• Quantized Float: Floating-point numbers are quantized to integers within the supported range. This is achieved by computing a scale and zero point for each column, which are used to map the floating-point numbers to the quantized integer space.

• String Enum: String columns are mapped to integers starting from 1. This mapping is stored and later used for de-quantization. If the number of unique strings exceeds 15, a ValueError is raised.

Using a user-defined schema

Before encryption, the data is preprocessed. For example string enums first need to be mapped to integers, and floating point values must be quantized. By default, this mapping is done automatically. However, when two different clients encrypt their data separately, the automatic mappings may differ, possibly due to some missing values in one of the client's DataFrame. Thus the column can not be selected when merging encrypted DataFrames.

The encrypted DataFrame supports user-defined mappings. These schemas are defined as a dictionary where keys represent column names and values contain meta-data about the column. Supported column meta-data are:

• string columns: mapping between string values and integers.

• float columns: the min/max range that the column values lie in.

schema = {
    "string_column": {"abc": 1, "bcd": 2 },
    "float_column": {"min": 0.1, "max": 0.5 }
}

Supported operations

Encrypted DataFrame is designed to support a subset of operations that are available for pandas DataFrames. For now, only the merge operation is supported. More operations will be added in the future releases.

Merge operation

Merge operation allows you to left or right join two DataFrames.

[!NOTE] The merge operation on Encrypted DataFrames can be securely performed on a third-party server, meaning that the server can execute the merge without ever having access to the unencrypted data. The server only requires the encrypted DataFrames.

df_right = """index,day,time,size
2,Thur,Lunch,2
5,Sat,Dinner,3
9,Sun,Dinner,2"""

# Encrypt the DataFrame
df_encrypted2 = client.encrypt_from_pandas(pandas.read_csv(StringIO(df_right)))

df_encrypted_merged = df_encrypted.merge(df_encrypted2, how="left", on="index")

Serialization

You can serialize encrypted DataFrame objects to a file format for storage or transfer. When serialized, they contain the encrypted data and public evaluation keys necessary to perform computations.

[!NOTE] Serialized DataFrames do not contain any private encryption keys . The DataFrames can be exchanged with any third-party without any risk.

To save or load an encrypted DataFrame from a file, use the following commands:

from concrete.ml.pandas import load_encrypted_dataframe

# Save
df_encrypted_merged.save("df_encrypted_merged")

# Load
df_encrypted_merged = load_encrypted_dataframe("df_encrypted_merged")

# Decrypt the DataFrame
df_decrypted = client.decrypt_to_pandas(df_encrypted)

Error handling

During the pre-processing and post-processing stages, the ValueError can happen in the following situations:

• A column contains values outside the allowed range for integers

• Too many unique strings

• Unsupported data type by Concrete ML

• Unsupported data type by the operation attempted

Example workflow

An example workflow where two clients encrypt two DataFrame objects, perform a merge operation on the server side, and then decrypt the results is available in the notebook encrypted_pandas.ipynb.

Current limitations

While this API offers a new secure way to work on remotely stored and encrypted data, it has some strong limitations at the moment:

• Precision of Values: The precision for numerical values is limited to 4 bits.

• Supported Operations: The merge operation is the only one available.

• Index Handling: Index values are not preserved; users should move any relevant data from the index to a dedicated new column before encrypting.

• Integer Range: The range of integers that can be encrypted is between 1 and 15.

• Uniqueness for merge: The merge operation requires that the columns to merge on contain unique values. Currently this means that data-frames are limited to 15 rows.

• Metadata Security: Column names and the mapping of strings to integers are not encrypted and are sent to the server in clear text.

In addition to the built-in models, Concrete ML supports generic machine learning models implemented with Torch, or exported as ONNX graphs.

There are two approaches to build FHE-compatible deep networks:

• Quantization Aware Training (QAT) requires using custom layers, but can quantize weights and activations to low bit-widths. Concrete ML works with Brevitas, a library providing QAT support for PyTorch. To use this mode, compile models using compile_brevitas_qat_model

• Post-training Quantization: This mode allows a vanilla PyTorch model to be compiled. However, when quantizing weights & activations to fewer than 7 bits, the accuracy can decrease strongly. On the other hand, depending on the model size, quantizing with 6-8 bits can be incompatible with FHE constraints. To use this mode, compile models with compile_torch_model.

Both approaches require the rounding_threshold_bits parameter to be set accordingly. The best values for this parameter need to be determined through experimentation. A good initial value to try is 6. See here for more details.

Quantization-aware training

The following example uses a simple QAT PyTorch model that implements a fully connected neural network with two hidden layers. Due to its small size, making this model respect FHE constraints is relatively easy. To use QAT, Brevitas QuantIdentity nodes must be inserted in the PyTorch model, including one that quantizes the input of the forward function.

import brevitas.nn as qnn
import torch.nn as nn
import torch

N_FEAT = 12
n_bits = 3

class QATSimpleNet(nn.Module):
    def __init__(self, n_hidden):
        super().__init__()

        self.quant_inp = qnn.QuantIdentity(bit_width=n_bits, return_quant_tensor=True)
        self.fc1 = qnn.QuantLinear(N_FEAT, n_hidden, True, weight_bit_width=n_bits, bias_quant=None)
        self.quant2 = qnn.QuantIdentity(bit_width=n_bits, return_quant_tensor=True)
        self.fc2 = qnn.QuantLinear(n_hidden, n_hidden, True, weight_bit_width=n_bits, bias_quant=None)
        self.quant3 = qnn.QuantIdentity(bit_width=n_bits, return_quant_tensor=True)
        self.fc3 = qnn.QuantLinear(n_hidden, 2, True, weight_bit_width=n_bits, bias_quant=None)

    def forward(self, x):
        x = self.quant_inp(x)
        x = self.quant2(torch.relu(self.fc1(x)))
        x = self.quant3(torch.relu(self.fc2(x)))
        x = self.fc3(x)
        return x

Once the model is trained, calling the compile_brevitas_qat_model from Concrete ML will automatically perform conversion and compilation of a QAT network. Here, 3-bit quantization is used for both the weights and activations. The compile_brevitas_qat_model function automatically identifies the number of quantization bits used in the Brevitas model.

from concrete.ml.torch.compile import compile_brevitas_qat_model
import numpy

torch_input = torch.randn(100, N_FEAT)
torch_model = QATSimpleNet(30)
quantized_module = compile_brevitas_qat_model(
    torch_model, # our model
    torch_input, # a representative input-set to be used for both quantization and compilation
    rounding_threshold_bits={"n_bits": 6, "method": "approximate"}
)

Post-training quantization

The following example uses a simple PyTorch model that implements a fully connected neural network with two hidden layers. The model is compiled to use FHE using compile_torch_model.

import torch.nn as nn
import torch

N_FEAT = 12
n_bits = 6

class PTQSimpleNet(nn.Module):
    def __init__(self, n_hidden):
        super().__init__()

        self.fc1 = nn.Linear(N_FEAT, n_hidden)
        self.fc2 = nn.Linear(n_hidden, n_hidden)
        self.fc3 = nn.Linear(n_hidden, 2)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = self.fc3(x)
        return x

from concrete.ml.torch.compile import compile_torch_model
import numpy

torch_input = torch.randn(100, N_FEAT)
torch_model = PTQSimpleNet(5)
quantized_module = compile_torch_model(
    torch_model, # our model
    torch_input, # a representative input-set to be used for both quantization and compilation
    n_bits=6,
    rounding_threshold_bits={"n_bits": 6, "method": "approximate"}
)

Configuring quantization parameters

With QAT (the PyTorch/Brevitas models created following the example above), you need to configure quantization parameters such as bit_width (activation bit-width) and weight_bit_width. When using this mode, set n_bits=None in the compile_brevitas_qat_model.

With PTQ, you need to set the n_bits value in the compile_torch_model function and must manually determine the trade-off between accuracy, FHE compatibility, and latency.

The quantization parameters, along with the number of neurons on each layer, will determine the accumulator bit-width of the network. Larger accumulator bit-widths result in higher accuracy but slower FHE inference time.

Running encrypted inference

The model can now perform encrypted inference.

x_test = numpy.array([numpy.random.randn(N_FEAT)])

y_pred = quantized_module.forward(x_test, fhe="execute")

In this example, the input values x_test and the predicted values y_pred are floating points. The quantization (resp. de-quantization) step is done in the clear within the forward method, before (resp. after) any FHE computations.

Simulated FHE Inference in the clear

You can perform the inference on clear data in order to evaluate the impact of quantization and of FHE computation on the accuracy of their model. See this section for more details. Two approaches exist:

• quantized_module.forward(quantized_x, fhe="simulate"): simulates FHE execution taking into account Table Lookup errors. De-quantization must be done in a second step as for actual FHE execution. Simulation takes into account the p_error/global_p_error parameters

• quantized_module.forward(quantized_x, fhe="disable"): computes predictions in the clear on quantized data, and then de-quantize the result. The return value of this function contains the de-quantized (float) output of running the model in the clear. Calling this function on clear data is useful when debugging, but this does not perform actual FHE simulation.