This document explains the essential cryptographic terms and the important concepts of Concrete ML model lifecycle with Fully Homomorphic Encryption (FHE).

Concrete ML is built on top of Concrete, which enables the conversion from NumPy programs into FHE circuits.

Lifecycle of a Concrete ML model

With Concrete ML, you can train a model on clear or encrypted data, then deploy it to predict on encrypted inputs. During deployment, data can be pre-processed while being encrypted. Therefore, data stay encrypted during the entire lifecycle of the machine learning model, with some limitations.

I. Model development

1. Training: A model is trained either using plaintext (non-encrypted) training data, or encrypted training data.

2. Quantization: Quantization converts inputs, model weights, and all intermediate values of the inference computation to integer equivalents. More information is available here. Concrete ML performs this step in two ways depending on model type:

   • During training (Quantization Aware Training)

   • After training (Post-training Quantization)

3. Simulation: Simulation allows you to execute a model that was quantized, to measure its accuracy in FHE, and to determine the modifications required to make it FHE compatible. Simulation is described in more detail here.

4. Compilation: After quantizing the model and confirming that it has good FHE accuracy through simulation, the model then needs to be compiled using Concrete's FHE Compiler to produce an equivalent FHE circuit. This circuit is represented as an MLIR program consisting of low level cryptographic operations. You can read more about FHE compilation here, MLIR here, and about the low-level Concrete library here.

5. Inference: The compiled model can then be executed on encrypted data, once the proper keys have been generated. The model can also be deployed to a server and used to run private inference on encrypted inputs.

You can find examples of the model development workflow here.

II. Model deployment

1. Pre-processing: Data owners(client) can generate keys to encrypt/decrypt data and store it in a DataFrame for further processing on a server. The server can pre-process such data with pre-compiled circuits, to prepare it for encrypted training or inference.

2. Client/server model deployment: In a client/server setting, Concrete ML models can be exported to:

   • Allow the client to generate keys, encrypt, and decrypt.

   • Provide a compiled model that can run on the server to perform inference on encrypted data.

3. Key generation: The data owner (client) needs to generate a set of keys:

   • A private encryption key to encrypt/decrypt their data and results

   • A public evaluation key for the model's FHE evaluation on the server.

You can find an example of the model deployment workflow here.

Cryptography concepts

Concrete ML and Concrete abstract the details of the underlying cryptography scheme, TFHE. However, understanding some cryptography concepts is still useful:

• Encryption and decryption: Encryption converts human-readable information (plaintext) into data (ciphertext) that is unreadable by a human or computer unless with the proper key. Encryption takes plaintext and an encryption key and produces ciphertext, while decryption is the reverse operation.

• Encrypted inference: FHE allows third parties to execute a machine learning model on encrypted data. The inference result is also encrypted and can only be decrypted by the key holder.

• Key generation: Cryptographic keys are generated using random number generators. Key generation can be time-consuming and produce large keys, but each model used by a client only requires key generation once.

  • Private encryption key: A private encryption key is a series of bits used within an encryption algorithm for encrypting data so that the corresponding ciphertext appears random.

  • Public evaluation key: A public evaluation key is used to perform homomorphic operations on encrypted data, typically by a server.

• Guaranteed correctness of encrypted computations: To ensure security, TFHE adds random noise to ciphertexts. Depending on the noise parameters, it can cause errors during encrypted data processing. By default, Concrete ML uses parameters that guarantee the correctness of encrypted computations, so the results on encrypted data equals to those of simulations on clear data.

• Programmable Boostrapping (PBS) : Programmable Bootstrapping enables the homomorphic evaluation of any function of a ciphertext, with a controlled level of noise. Learn more about PBS in this paper.

For a deeper understanding of the cryptography behind the Concrete stack, refer to the whitepaper on TFHE and Programmable Boostrapping or this series of blogs.

Model accuracy considerations under FHE constraints

FHE requires all inputs, constants, and intermediate values to be integers of maximum 16 bits. To make machine learning models compatible with FHE, Concrete ML implements some techniques with accuracy considerations:

• Quantization: Concrete ML quantizes inputs, outputs, weights, and activations to meet FHE limitations. See the quantization documentation for details.

  • Accuracy trade-off: Quantization may reduce accuracy, but careful selection of quantization parameters or of the training approach can mitigate this. Concrete ML offers built-in quantized models; users only configure parameters like bit-width. For more details of quantization configurations, see the advanced quantization guide.

• Additional methods: Dimensionality reduction and pruning are additional ways to make programs compatible for FHE. See Poisson regression example for dimensionality reduction and built-in neural networks for pruning.

This document provides guides on how to install Concrete ML using PyPi or Docker.

Prerequisite

Before you start, determine your environment:

• Hardware platform

• Operating System (OS) version

• Python version

OS/HW support

Depending on your OS/HW, Concrete ML may be installed with Docker or with pip:

Python support

• Version: In the current release, Concrete ML supports only 3.8, 3.9 and 3.10 versions of python.

• Linux requirement: The Concrete ML Python package requires glibc >= 2.28. On Linux, you can check your glibc version by running ldd --version.

• Kaggle installation: Concrete ML can be installed on Kaggle (see question on community for more details) and on Google Colab.

Most of these limits are shared with the rest of the Concrete stack (namely Concrete Python). Support for more platforms will be added in the future.

Installation using PyPi

Requirements

Installing Concrete ML using PyPi requires a Linux-based OS or macOS (both x86 and Apple Silicon CPUs are supported).

Installation

To install Concrete ML from PyPi, run the following:

pip install -U pip wheel setuptools
pip install concrete-ml

This will automatically install all dependencies, notably Concrete.

If you encounter any issue during installation on Apple Silicon mac, please visit this troubleshooting guide on community.

Installation using Docker

You can install Concrete ML using Docker by either pulling the latest image or a specific version:

docker pull zamafhe/concrete-ml:latest
# or
docker pull zamafhe/concrete-ml:v0.4.0

You can use the image with Docker volumes, see the Docker documentation here. Use the following command:

# Without local volume:
docker run --rm -it -p 8888:8888 zamafhe/concrete-ml

# With local volume to save notebooks on host:
docker run --rm -it -p 8888:8888 -v /host/path:/data zamafhe/concrete-ml

This will launch a Concrete ML enabled Jupyter server in Docker that can be accessed directly from a browser.

Alternatively, you can launch a shell in Docker, with or without volumes:

docker run --rm -it zamafhe/concrete-ml /bin/bash

Concrete ML is an open source, privacy-preserving, machine learning framework based on Fully Homomorphic Encryption (FHE). It enables data scientists without any prior knowledge of cryptography to perform:

• Automatic model conversion: Use familiar APIs from scikit-learn and PyTorch to convert machine learning models to their FHE equivalent. This is applicable for linear models, tree-based models, and neural networks).

• Encrypted data training: Train models directly on encrypted data to maintain privacy.

• Encrypted data pre-processing: Pre-process encrypted data using a DataFrame paradigm.

Key features

• Training on encrypted data: FHE is an encryption technique that allows computing directly on encrypted data, without needing to decrypt it. With FHE, you can build private-by-design applications without compromising on features. Learn more about FHE in this introduction or join the FHE.org community.

• Federated learning: Training on encrypted data provides the highest level of privacy but is slower than training on clear data. Federated learning is an alternative approach, where data privacy can be ensured by using a trusted gradient aggregator, coupled with optional differential privacy instead of encryption. Concrete ML can import all types of models: linear, tree-based and neural networks, that are trained using federated learning using the from_sklearn_model function and the compile_torch_model function.

Example usage

Here is a simple example of classification on encrypted data using logistic regression. You can find more examples here.

This example shows the typical flow of a Concrete ML model:

1. Training the model: Train the model on unencrypted (plaintext) data using scikit-learn. Since Fully Homomorphic Encryption (FHE) operates over integers, Concrete ML quantizes the model to use only integers during inference.

2. Compiling the model: Compile the quantized model to an FHE equivalent. Under the hood, the model is first converted to a Concrete Python program and then compiled.

3. Performing inference: Perform inference on encrypted data. The example above shows encrypted inference in the model-development phase. Alternatively, during deployment in a client/server setting, the client encrypts the data, the server processes it securely, and then the client decrypts the results.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from concrete.ml.sklearn import LogisticRegression

# Lets create a synthetic data-set
x, y = make_classification(n_samples=100, class_sep=2, n_features=30, random_state=42)

# Split the data-set into a train and test set
X_train, X_test, y_train, y_test = train_test_split(
    x, y, test_size=0.2, random_state=42
)

# Now we train in the clear and quantize the weights
model = LogisticRegression(n_bits=8)
model.fit(X_train, y_train)

# We can simulate the predictions in the clear
y_pred_clear = model.predict(X_test)

# We then compile on a representative set
model.compile(X_train)

# Finally we run the inference on encrypted inputs
y_pred_fhe = model.predict(X_test, fhe="execute")

print(f"In clear  : {y_pred_clear}")
print(f"In FHE    : {y_pred_fhe}")
print(f"Similarity: {(y_pred_fhe == y_pred_clear).mean():.1%}")

# Output:
    # In clear  : [0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 1 0 0]
    # In FHE    : [0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 1 0 0]
    # Similarity: 100.0%

It is also possible to call encryption, model prediction, and decryption functions separately as follows. Executing these steps separately is equivalent to calling predict_proba on the model instance.

# Predict probability for a single example
y_proba_fhe = model.predict_proba(X_test[[0]], fhe="execute")

# Quantize an original float input
q_input = model.quantize_input(X_test[[0]])

# Encrypt the input
q_input_enc = model.fhe_circuit.encrypt(q_input)

# Execute the linear product in FHE 
q_y_enc = model.fhe_circuit.run(q_input_enc)

# Decrypt the result (integer)
q_y = model.fhe_circuit.decrypt(q_y_enc)

# De-quantize and post-process the result
y0 = model.post_processing(model.dequantize_output(q_y))

print("Probability with `predict_proba`: ", y_proba_fhe)
print("Probability with encrypt/run/decrypt calls: ", y0)

Current limitations

• Precision and accuracy: In order to run models in FHE, Concrete ML requires models to be within the precision limit, currently 16-bit integers. Thus, machine learning models must be quantized and it sometimes leads to a loss of accuracy versus the original model that operates on plaintext.

• Models availability: Concrete ML currently only supports training on encrypted data for some models, while it supports inference for a large variety of models.

• Processing: Concrete currently doesn't support pre-processing model inputs and post-processing model outputs. These processing stages may involve:

  • Text-to-numerical feature transformation

  • Dimensionality reduction

  • KNN or clustering

  • Featurization

  • Normalization

  • The mixing of ensemble models' results.

These issues are currently being addressed, and significant improvements are expected to be released in the near future.

Concrete stack

Concrete ML is built on top of Zama's Concrete.

Online demos and tutorials

Various tutorials are available for built-in models and deep learning. Several stand-alone demos for use cases can be found in the Demos and Tutorials section.

If you have built awesome projects using Concrete ML, feel free to let us know and we'll link to your work!

Additional resources

• Zama's blog

Support

• Community channels (we answer in less than 24 hours).

Get started

Learn the basics of Concrete ML, set it up, and make it run with ease.

Build with Concrete ML

Start building with Concrete ML by exploring its core features, discovering essential guides, and learning more with user-friendly tutorials.

Explore more

Access to additional resources and join the Zama community.

References & Explanations

Refer to the API, review product architecture, and access additional resources for in-depth explanations while working with Concrete ML.

Support channels

Ask technical questions and discuss with the community. Our team of experts usually answers within 24 hours in working days.

Developers

Collaborate with us to advance the FHE spaces and drive innovation together.

This document explains how to train on encrypted data.

Training on encrypted data is done through an FHE program that is generated by Concrete ML, based on the characteristics of the data that are given to the fit function. Once the FHE program associated with the SGDClassifier object has fit the encrypted data, it performs specifically to that data's distribution and dimensionality.

When deploying encrypted training services, you need to consider the type of data that future users of your services will train on:

• The distribution of the data should match to achieve good accuracy

• The dimensionality of the data needs to match since the deployed FHE programs are compiled for a fixed number of dimensions.

See the section for more details.

Example

The example shows logistic regression training on encrypted data in action.

The following snippet shows how to instantiate a logistic regression model that trains on encrypted data:

To activate encrypted training, simply set fit_encrypted=True in the constructor. When the value is set, Concrete ML generates an FHE program which, when called through the fit function, processes encrypted training data, labels and initial weights and outputs trained model weights. If this value is not set, training is performed on clear data using scikit-learn gradient descent.

Next, to perform the training on encrypted data, call the fit function with the fhe="execute" argument:

Training configuration

The max_iter parameter controls the number of batches that are processed by the training algorithm.

Capabilities and Limitations

The trainable logistic model uses Stochastic Gradient Descent (SGD) and quantizes the data, weights, gradients and the error measure. It currently supports training 6-bit models, including g both the coefficients and the bias.

The SGDClassifier does not currently support training models with other bit-width values. The execution time to train a model is proportional to the number of features and the number of training examples in the batch. The SGDClassifier training does not currently support client/server deployment for training.

Deployment

This document introduces the simple built-in neural networks models that Concrete ML provides with a scikit-learn interface through the NeuralNetClassifier and NeuralNetRegressor classes.

Supported models

The neural network models are implemented with , which provides a scikit-learn-like interface to Torch models (more ).

Concrete ML models are multi-layer, fully-connected, networks with customizable activation functions and have a number of neurons in each layer. This approach is similar to what is available in scikit-learn when using the MLPClassifier/MLPRegressor classes. The built-in models train easily with a single call to .fit(), which will automatically quantize weights and activations. These models use Quantization Aware Training, allowing good performance for low precision (down to 2-3 bits) weights and activations.

While NeuralNetClassifier and NeuralNetClassifier provide scikit-learn-like models, their architecture is somewhat restricted to make training easy and robust. If you need more advanced models, you can convert custom neural networks as described in the .

Example

To create an instance of a Fully Connected Neural Network (FCNN), you need to instantiate one of the NeuralNetClassifier and NeuralNetRegressor classes and configure a number of parameters that are passed to their constructor.

Note that some parameters need to be prefixed by module__, while others don't. The parameters related to the model must have the prefix, such as the underlying nn.Module. The parameters related to training options do not require the prefix.

The folowing figure shows the Concrete ML neural network trained with Quantization Aware Training in an FHE-compatible configuration and compares it to the floating-point equivalent trained with scikit-learn.

Architecture parameters

• module__n_layers: number of layers in the FCNN.

  • This parameter must be at least 1. Note that this is the total number of layers. For a single, hidden layer NN model, set module__n_layers=2

• module__activation_function: can be one of the Torch activations (such as nn.ReLU)

  • Neural networks with nn.ReLU activation benefit from specific optimizations that make them around 10x faster than networks with other activation functions.

Quantization parameters

• n_w_bits (default 3): number of bits for weights

• n_a_bits (default 3): number of bits for activations and inputs

• n_accum_bits: maximum accumulator bit-width that is desired
• power_of_two_scaling (default True): forces quantization scales to be powers-of-two

  • When coupled with the ReLU activation, this optimize strongly the FHE inference time.

Training parameters (from skorch)

• max_epochs (default 10): The number of epochs to train the network

• verbose (default: False): Whether to log loss/metrics during training

• lr (default 0.001): Learning rate

Advanced parameters

• module__n_hidden_neurons_multiplier (default 4): The number of hidden neurons.

  • This parameter will be automatically set proportional to the dimensionality of the input. It controls the proportionality factor. This value gives good accuracy while avoiding accumulator overflow.

Class weights

You can give weights to each class to use in training. Note that this must be supported by the underlying PyTorch loss function.

Overflow errors

The n_accum_bits parameter influences training accuracy by controlling the number of non-zero neurons allowed in each layer. You can increase n_accum_bits to improve accuracy, but must consider the precision limitations to avoid an overflow in the accumulator. The default value is a balanced choice that generally avoids overflow, but you may need to adjust it to reduce the network breadth if you encounter overflow errors.

The number of neurons in intermediate layers is controlled by the n_hidden_neurons_multiplier parameter. A value of 1 makes intermediate layers have the same number of neurons as the number as the input data dimensions.

This document illustrate how Concrete ML model and DataFrames are deployed in client/server setting when creating privacy-preserving services in the cloud.

Once compiled to FHE, a Concrete ML model or DataFrame generates machine code that execute prediction, training or pre-processing on encrypted data. During this process, Concrete ML generates and .

Communication protocols

The overall communications protocol to enable cloud deployment of machine learning services can be summarized in the following diagram:

The steps detailed above are:

1. Model Deployment: The model developer deploys the compiled machine learning model to the server. This model includes the cryptographic parameters. The server is now ready to provide private inference. Cryptographic parameters and compiled programs for DataFrames are included directly in Concrete ML.

2. Client request: The client requests the cryptographic parameters (client specs). Once the client receives them from the server, the secret and evaluation keys are generated.

3. Key exchanges: The client sends the evaluation key to the server. The server is now ready to accept requests from this client. The client sends their encrypted data. Serialized DataFrames include client evaluation keys.

4. Private inference: The server uses the evaluation key to securely run prediction, training and pre-processing on the user's data and sends back the encrypted result.

5. Decryption: The client now decrypts the result and can send back new requests.

This document introduces several 's linear models for classification and regression tree models that Concrete ML provides.

Supported models

Concrete ML also supports 's XGBClassifier and XGBRegressor:

Pre-trained models

You can convert an already trained scikit-learn tree-based model to a Concrete ML one by using the method.

Example

Here's an example of how to use this model in FHE on a popular data-set using some of scikit-learn's pre-processing tools. You can find a more complete example in the .

Quantization parameters

When using a sufficiently high bit-width, quantization has little impact on the decision boundaries of the Concrete ML FHE decision tree model, as quantization is done individually on each input feature. It means FHE models can achieve similar accuracy levels as floating point models. Using 6 bits for quantization is effective in reaching or even exceeding floating point accuracy.

To adjust the number of bits for quantization, use the n_bits parameter. Setting n_bits to a low value may introduce artifacts, potentially reducing accuracy. However, the execution speed in FHE could improve. This adjustment allows you to manage the accuracy/speed trade-off. Additionally, you can recover some accuracy by increasing the n_estimators parameter.

The following graph shows that using 5-6 bits of quantization is usually sufficient to reach the performance of a non-quantized XGBoost model on floating point data. The metrics plotted are accuracy and F1-score on the spambase data-set.

FHE Inference time considerations

The inference time in FHE is strongly dependant on the maximum circuit bit-width. For trees, in most cases, the quantization bit-width will be the same as the circuit bit-width. Therefore, reducing the quantization bit-width to 4 or less will result in fast inference times. Adding more bits will increase FHE inference time exponentially.

In some rare cases, the bit-width of the circuit can be higher than the quantization bit-width. This could happen when the quantization bit-width is low but the tree-depth is high. In such cases, the circuit bit-width is upper bounded by ceil(log2(max_depth + 1) + 1).

This document introduces the nearest neighbors non-parametric classification models that Concrete ML provides with a scikit-learn interface through the KNeighborsClassifier class.

Example

Quantization parameters

The KNeighborsClassifier class quantizes the training data-set provided to .fit using the specified number of bits (n_bits). To comply with , you must keep this value low. The model's accuracy will depend significantly on a well-chosen n_bits value and the dimensionality of the data.

The predict method of the KNeighborsClassifier performs the following steps:

1. Quantize the test vectors on clear data

2. Compute the top-k class indices of the closest training set vector on encrypted data

3. Vote for the top-k class labels to find the class for each test vector, performed on clear data

Inference time considerations

The FHE inference latency of this model is heavily influenced by the n_bits and the dimensionality of the data. Additionally, the data-set size has a linear impact on the data complexity. The number of nearest neighbors (n_neighbors) also affects performance.

The KNN computation executes in FHE in steps, where is the training data-set size and is n_neighbors. Each step requires several , with their runtime affected by the factors listed above. These factors determine the precision needed to represent the distances between test vectors and training data-set vectors. The PBS input precision required by the circuit is related to the precision of the distance values.

This page explains Concrete ML linear models for both classification and regression. These models are based on linear models.

Supported models for encrypted inference

The following models are supported for training on clear data and predicting on encrypted data. Their API is similar the one of . These models are also compatible with some of scikit-learn's main workflows, such as Pipeline() and GridSearch().

Supported models for encrypted training

In addition to predicting on encrypted data, the following models support training on encrypted data.

| | |

Quantization parameters

The n_bits parameter controls the bit-width of the inputs and weights of the linear models. Linear models do not use table lookups and thus alllows weight and inputs to be high precision integers.

For models with input dimensions up to 300, the parameter n_bits can be set to 8 or more. When the input dimensions are larger, n_bits must be reduced to 6-7. In many cases, quantized models can preserve all performance metrics compared to the non-quantized float models from scikit-learn when n_bits is down to 6. You should validate accuracy on held-out test sets and adjust n_bits accordingly.

Pre-trained models

Example

Model accuracy

Loading a pre-trained model

An alternative to the example above is to train a scikit-learn model in a separate step and then to convert it to Concrete ML.

This section provides a set of tools and guidelines to help users build optimized FHE-compatible models. It discusses FHE simulation, the key-cache functionality that helps speed-up FHE result debugging, and gives a guide to evaluate circuit complexity.

Simulation

The of Concrete ML provides a way to evaluate, using clear data, the results that ML models produce on encrypted data. The simulation includes any probabilistic behavior FHE may induce. The simulation is implemented with .

The simulation mode can be useful when developing and iterating on an ML model implementation. As FHE non-linear models work with integers up to 16 bits, with a trade-off between the number of bits and the FHE execution speed, the simulation can help to find the optimal model design.

Simulation is much faster than FHE execution. This allows for faster debugging and model optimization. For example, this was used for the red/blue contours in the , as computing in FHE for the whole grid and all the classifiers would take significant time.

The following example shows how to use the simulation mode in Concrete ML.

Caching keys during debugging

It is possible to avoid re-generating the keys of the models you are debugging. This feature is unsafe and should not be used in production. Here is an example that shows how to enable key-caching:

Common compilation errors

1. TLU input maximum bit-width is exceeded

Error message: this [N]-bit value is used as an input to a table lookup

Cause: This error can occur when rounding_threshold_bits is not used and accumulated intermediate values in the computation exceed 16 bits.

Possible solutions:

2. No crypto-parameters can be found

Error message: RuntimeError: NoParametersFound

Cause: This error occurs when using rounding_threshold_bits in the compile_torch_model function.

Possible solutions: The solutions in this case are similar to the ones for the previous error.

3. Quantization import failed

Error message: Error occurred during quantization aware training (QAT) import [...] Could not determine a unique scale for the quantization!.

A common example is related to the concatenation operator. Suppose two tensors x and y are produced by two layers and need to be concatenated:

In the example above, the x and y layers need quantization before being concatenated.

Possible solutions:

1. If the error occurs for the first layer of the model: Add a QuantIdentity layer in your model and apply it on the input of the forward function, before the first layer is computed.

2. If the error occurs for a concatenation or addition layer: Add a new QuantIdentity layer in your model. Suppose it is called quant_concat. In the forward function, before concatenation of x and y, apply it to both tensors that are concatenated. The usage of a common Quantidentity layer to quantize both tensors that are concatenated ensures that they have the same scale:

Debugging compilation errors

Compilation errors due to FHE incompatible models, such as maximum bit-width exceeded or NoParametersFound can be debugged by examining the bit-widths associated with various intermediate values of the FHE computation.

The following produces a neural network that is not FHE-compatible:

Upon execution, the Compiler will raise the following error within the graph representation:

Fixing compilation errors

To make this network FHE-compatible one can apply several techniques:

1. reduce the accumulator bit-width of the second layer named fc2. To do this, a simple solution is to reduce the number of neurons, as it is proportional to the bit-width.

Complexity analysis

In FHE, univariate functions are encoded as table lookups, which are then implemented using Programmable Bootstrapping (PBS). PBS is a powerful technique but will require significantly more computing resources, and thus time, compared to simpler encrypted operations such as matrix multiplications, convolution, or additions.

Furthermore, the cost of PBS will depend on the bit-width of the compiled circuit. Every additional bit in the maximum bit-width raises the complexity of the PBS by a significant factor. It may be of interest to the model developer, then, to determine the bit-width of the circuit and the amount of PBS it performs.

This can be done by inspecting the MLIR code produced by the Compiler:

There are several calls to FHELinalg.apply_mapped_lookup_table and FHELinalg.apply_lookup_table. These calls apply PBS to the cells of their input tensors. Their inputs in the listing above are: tensor<1x2x!FHE.eint<8>> for the first and last call and tensor<1x50x!FHE.eint<8>> for the two calls in the middle. Thus, PBS is applied 104 times.

Retrieving the bit-width of the circuit is then simply:

Decreasing the number of bits and the number of PBS applications induces large reductions in the computation time of the compiled circuit.

Neural networks pose unique challenges with regards to encrypted inference. Each neuron in a network applies an activation function that requires a PBS operation. The latency of a single PBS depends on the bit-width of the input of the PBS.

Several approaches can be used to reduce the overall latency of a neural network.

Circuit bit-width optimization

Quantization Aware Training and pruning introduce specific hyper-parameters that influence the accumulator sizes. It is possible to chose quantization and pruning configurations that reduce the accumulator size. A trade-off between latency and accuracy can be obtained by varying these hyper-parameters as described in the deep learning design guide.

Structured pruning

While un-structured pruning is used to ensure the accumulator bit-width stays low, structured pruning can eliminate entire neurons from the network. Many neural networks are over-parametrized (since this enables easier training) and some neurons can be removed. Structured pruning, applied to a trained network as a fine-tuning step, can be applied to built-in neural networks using the prune helper function as shown in this example. To apply structured pruning to custom models, it is recommended to use the torch-pruning package.

Rounded activations and quantizers

Reducing the bit-width of the inputs to the Table Lookup (TLU) operations is a major source of improvements in the latency. Post-training, it is possible to leverage some properties of the fused activation and quantization functions expressed in the TLUs to further reduce the accumulator. This is achieved through the rounded PBS feature as described in the rounded activations and quantizers reference. Adjusting the rounding amount, relative to the initial accumulator size, can bring large improvements in latency while maintaining accuracy.

TLU error tolerance adjustment

Finally, the TFHE scheme exposes a TLU error tolerance parameter that has an impact on crypto-system parameters that influence latency. A higher tolerance of TLU off-by-one errors results in faster computations but may reduce accuracy. One can think of the error of obtaining $T[x]$ as a Gaussian distribution centered on $x$: $TLU[x]$ is obtained with probability of 1 - p_error, while $T[x-1]$, $T[x+1]$ are obtained with much lower probability, etc. In Deep NNs, these type of errors can be tolerated up to some point. See the p_error documentation for details and more specifically the usage example of the API for finding the best p_error.

Concrete ML has APIs that make it easy, during model development and testing, to perform encryption, execution in FHE, and decryption in a single step. For more control, these individual steps can be executed separately. The APIs used to accomplish this are different for:

• Built-in models

• Custom models

Built-in models

The following example shows how to create a synthetic data-set and how to use it to train a LogisticRegression model from Concrete ML. Next, we will discuss the dedicated functions for encryption, inference, and decryption.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from concrete.ml.sklearn import LogisticRegression
import numpy

# Create a synthetic data-set for a classification problem
x, y = make_classification(n_samples=100, class_sep=2, n_features=3, n_informative=3, n_redundant=0, random_state=42)

# Split the data-set into a train and test set
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=42)

# Instantiate and train the model
model = LogisticRegression()
model.fit(x_train,y_train)

# Simulate the predictions in the clear (optional)
y_pred_clear = model.predict(x_test)

# Compile the model on a representative set
fhe_circuit = model.compile(x_train)

All Concrete ML built-in models have a monolithic predict method that performs the encryption, FHE execution, and decryption with a single function call. Concrete ML models follow the same API as scikit-learn models, transparently performing the steps related to encryption for convenience.

# Predict in FHE
y_pred_fhe = model.predict(x_test, fhe="execute")

Regarding this LogisticRegression model, as with scikit-learn, it is possible to predict the logits as well as the class probabilities by respectively using the decision_function or predict_proba methods instead.

Alternatively, it is possible to execute all main steps (key generation, quantization, encryption, FHE execution, decryption) separately.

# Generate the keys (set force to True in order to generate new keys at each execution)
fhe_circuit.keygen(force=True)

y_pred_fhe_step = []

for f_input in x_test:
    # Quantize an input (float)
    q_input = model.quantize_input([f_input])
    
    # Encrypt the input
    q_input_enc = fhe_circuit.encrypt(q_input)

    # Execute the linear product in FHE 
    q_y_enc = fhe_circuit.run(q_input_enc)

    # Decrypt the result (integer)
    q_y = fhe_circuit.decrypt(q_y_enc)

    # De-quantize the result
    y = model.dequantize_output(q_y)

    # Apply either the sigmoid if it is a binary classification task, which is the case in this 
    # example, or a softmax function in order to get the probabilities (in the clear)
    y_proba = model.post_processing(y)

    # Since this model does classification, apply the argmax to get the class predictions (in the clear)
    # Note that regression models won't need the following line
    y_class = numpy.argmax(y_proba, axis=1)

    y_pred_fhe_step += list(y_class)

y_pred_fhe_step = numpy.array(y_pred_fhe_step)

print("Predictions in clear:", y_pred_clear)
print("Predictions in FHE  :", y_pred_fhe_step)
print(f"Similarity: {int((y_pred_fhe_step == y_pred_clear).mean()*100)}%")

Custom models

For custom models, the API to execute inference in FHE or simulation is illustrated as:

from torch import nn
from brevitas import nn as qnn
from concrete.ml.torch.compile import compile_brevitas_qat_model

class FCSmall(nn.Module):
    """A small QAT NN."""

    def __init__(self, input_output):
        super().__init__()
        self.quant_input = qnn.QuantIdentity(bit_width=3)
        self.fc1 = qnn.QuantLinear(in_features=input_output, out_features=input_output, weight_bit_width=3, bias=True)
        self.act_f = nn.ReLU()
        self.fc2 = qnn.QuantLinear(in_features=input_output, out_features=input_output, weight_bit_width=3, bias=True)

    def forward(self, x):
        return self.fc2(self.act_f(self.fc1(self.quant_input(x))))

torch_model = FCSmall(3)

quantized_module = compile_brevitas_qat_model(
    torch_model,
    x_train,
)

x_test_q = quantized_module.quantize_input(x_test)
y_pred = quantized_module.quantized_forward(x_test_q, fhe="simulate")
y_pred = quantized_module.dequantize_output(y_pred)

y_pred = numpy.argmax(y_pred, axis=1)

In addition to the built-in models, Concrete ML supports generic machine learning models implemented with Torch, or exported as ONNX graphs.

There are two approaches to build FHE-compatible deep networks:

• Quantization Aware Training (QAT) requires using custom layers, but can quantize weights and activations to low bit-widths. Concrete ML works with Brevitas, a library providing QAT support for PyTorch. To use this mode, compile models using compile_brevitas_qat_model

• Post-training Quantization: This mode allows a vanilla PyTorch model to be compiled. However, when quantizing weights & activations to fewer than 7 bits, the accuracy can decrease strongly. On the other hand, depending on the model size, quantizing with 6-8 bits can be incompatible with FHE constraints. To use this mode, compile models with compile_torch_model.

Both approaches require the rounding_threshold_bits parameter to be set accordingly. The best values for this parameter need to be determined through experimentation. A good initial value to try is 6. See here for more details.

Quantization-aware training

The following example uses a simple QAT PyTorch model that implements a fully connected neural network with two hidden layers. Due to its small size, making this model respect FHE constraints is relatively easy. To use QAT, Brevitas QuantIdentity nodes must be inserted in the PyTorch model, including one that quantizes the input of the forward function.

import brevitas.nn as qnn
import torch.nn as nn
import torch

N_FEAT = 12
n_bits = 3

class QATSimpleNet(nn.Module):
    def __init__(self, n_hidden):
        super().__init__()

        self.quant_inp = qnn.QuantIdentity(bit_width=n_bits, return_quant_tensor=True)
        self.fc1 = qnn.QuantLinear(N_FEAT, n_hidden, True, weight_bit_width=n_bits, bias_quant=None)
        self.quant2 = qnn.QuantIdentity(bit_width=n_bits, return_quant_tensor=True)
        self.fc2 = qnn.QuantLinear(n_hidden, n_hidden, True, weight_bit_width=n_bits, bias_quant=None)
        self.quant3 = qnn.QuantIdentity(bit_width=n_bits, return_quant_tensor=True)
        self.fc3 = qnn.QuantLinear(n_hidden, 2, True, weight_bit_width=n_bits, bias_quant=None)

    def forward(self, x):
        x = self.quant_inp(x)
        x = self.quant2(torch.relu(self.fc1(x)))
        x = self.quant3(torch.relu(self.fc2(x)))
        x = self.fc3(x)
        return x

Once the model is trained, calling the compile_brevitas_qat_model from Concrete ML will automatically perform conversion and compilation of a QAT network. Here, 3-bit quantization is used for both the weights and activations. The compile_brevitas_qat_model function automatically identifies the number of quantization bits used in the Brevitas model.

from concrete.ml.torch.compile import compile_brevitas_qat_model
import numpy

torch_input = torch.randn(100, N_FEAT)
torch_model = QATSimpleNet(30)
quantized_module = compile_brevitas_qat_model(
    torch_model, # our model
    torch_input, # a representative input-set to be used for both quantization and compilation
    rounding_threshold_bits={"n_bits": 6, "method": "approximate"}
)

Post-training quantization

The following example uses a simple PyTorch model that implements a fully connected neural network with two hidden layers. The model is compiled to use FHE using compile_torch_model.

import torch.nn as nn
import torch

N_FEAT = 12
n_bits = 6

class PTQSimpleNet(nn.Module):
    def __init__(self, n_hidden):
        super().__init__()

        self.fc1 = nn.Linear(N_FEAT, n_hidden)
        self.fc2 = nn.Linear(n_hidden, n_hidden)
        self.fc3 = nn.Linear(n_hidden, 2)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = self.fc3(x)
        return x

from concrete.ml.torch.compile import compile_torch_model
import numpy

torch_input = torch.randn(100, N_FEAT)
torch_model = PTQSimpleNet(5)
quantized_module = compile_torch_model(
    torch_model, # our model
    torch_input, # a representative input-set to be used for both quantization and compilation
    n_bits=6,
    rounding_threshold_bits={"n_bits": 6, "method": "approximate"}
)

Configuring quantization parameters

With QAT (the PyTorch/Brevitas models created following the example above), you need to configure quantization parameters such as bit_width (activation bit-width) and weight_bit_width. When using this mode, set n_bits=None in the compile_brevitas_qat_model.

With PTQ, you need to set the n_bits value in the compile_torch_model function and must manually determine the trade-off between accuracy, FHE compatibility, and latency.

The quantization parameters, along with the number of neurons on each layer, will determine the accumulator bit-width of the network. Larger accumulator bit-widths result in higher accuracy but slower FHE inference time.

Running encrypted inference

The model can now perform encrypted inference.

x_test = numpy.array([numpy.random.randn(N_FEAT)])

y_pred = quantized_module.forward(x_test, fhe="execute")

In this example, the input values x_test and the predicted values y_pred are floating points. The quantization (resp. de-quantization) step is done in the clear within the forward method, before (resp. after) any FHE computations.

Simulated FHE Inference in the clear

You can perform the inference on clear data in order to evaluate the impact of quantization and of FHE computation on the accuracy of their model. See this section for more details. Two approaches exist:

• quantized_module.forward(quantized_x, fhe="simulate"): simulates FHE execution taking into account Table Lookup errors. De-quantization must be done in a second step as for actual FHE execution. Simulation takes into account the p_error/global_p_error parameters

• quantized_module.forward(quantized_x, fhe="disable"): computes predictions in the clear on quantized data, and then de-quantize the result. The return value of this function contains the de-quantized (float) output of running the model in the clear. Calling this function on clear data is useful when debugging, but this does not perform actual FHE simulation.

Supported operators and activations

Concrete ML supports a variety of PyTorch operators that can be used to build fully connected or convolutional neural networks, with normalization and activation layers. Moreover, many element-wise operators are supported.

Operators

Univariate operators

• torch.nn.identity

• torch.clip

• torch.clamp

• torch.round

• torch.floor

• torch.min

• torch.max

• torch.abs

• torch.neg

• torch.sign

• torch.logical_or, torch.Tensor operator ||

• torch.logical_not

• torch.gt, torch.greater

• torch.ge, torch.greater_equal

• torch.lt, torch.less

• torch.le, torch.less_equal

• torch.eq

• torch.where

• torch.exp

• torch.log

• torch.pow

• torch.sum

• torch.mul, torch.Tensor operator *

• torch.div, torch.Tensor operator /

• torch.nn.BatchNorm2d

• torch.nn.BatchNorm3d

• torch.erf, torch.special.erf

• torch.nn.functional.pad

Shape modifying operators

• torch.reshape

• torch.Tensor.view

• torch.flatten

• torch.unsqueeze

• torch.squeeze

• torch.transpose

• torch.concat, torch.cat

• torch.nn.Unfold

Tensor operators

• torch.Tensor.expand

• torch.Tensor.to -- for casting to dtype

Multi-variate operators: encrypted input and unencrypted constants

• torch.nn.Linear

• torch.conv1d, torch.nn.Conv1D

• torch.conv2d, torch.nn.Conv2D

• torch.nn.AvgPool2d

• torch.nn.MaxPool2d

Concrete ML also supports some of their QAT equivalents from Brevitas.

• brevitas.nn.QuantLinear

• brevitas.nn.QuantConv1d

• brevitas.nn.QuantConv2d

Multi-variate operators: encrypted+unencrypted or encrypted+encrypted inputs

• torch.add, torch.Tensor operator +

• torch.sub, torch.Tensor operator -

• torch.matmul

Quantizers

• brevitas.nn.QuantIdentity

Activation functions

• torch.nn.CELU

• torch.nn.ELU

• torch.nn.GELU

• torch.nn.Hardshrink

• torch.nn.HardSigmoid

• torch.nn.Hardswish

• torch.nn.HardTanh

• torch.nn.LeakyReLU

• torch.nn.LogSigmoid

• torch.nn.Mish

• torch.nn.PReLU

• torch.nn.ReLU6

• torch.nn.ReLU

• torch.nn.SELU

• torch.nn.Sigmoid

• torch.nn.SiLU

• torch.nn.Softplus

• torch.nn.Softshrink

• torch.nn.Softsign

• torch.nn.Tanh

• torch.nn.Tanhshrink

• torch.nn.Threshold -- partial support

In addition to Concrete ML models and custom models in torch, it is also possible to directly compile ONNX models. This can be particularly appealing, notably to import models trained with Keras.

ONNX models can be compiled by directly importing models that are already quantized with Quantization Aware Training (QAT) or by performing Post-Training Quantization (PTQ) with Concrete ML.

Simple example

The following example shows how to compile an ONNX model using PTQ. The model was initially trained using Keras before being exported to ONNX. The training code is not shown here.

import numpy
import onnx
import tensorflow
import tf2onnx

from concrete.ml.torch.compile import compile_onnx_model
from concrete.fhe.compilation import Configuration


class FC(tensorflow.keras.Model):
    """A fully-connected model."""

    def __init__(self):
        super().__init__()
        hidden_layer_size = 10
        output_size = 5

        self.dense1 = tensorflow.keras.layers.Dense(
            hidden_layer_size,
            activation=tensorflow.nn.relu,
        )
        self.dense2 = tensorflow.keras.layers.Dense(output_size, activation=tensorflow.nn.relu6)
        self.flatten = tensorflow.keras.layers.Flatten()

    def call(self, inputs):
        """Forward function."""
        x = self.flatten(inputs)
        x = self.dense1(x)
        x = self.dense2(x)
        return self.flatten(x)


n_bits = 6
input_output_feature = 2
input_shape = (input_output_feature,)
num_inputs = 1
n_examples = 5000

# Define the Keras model
keras_model = FC()
keras_model.build((None,) + input_shape)
keras_model.compute_output_shape(input_shape=(None, input_output_feature))

# Create random input
input_set = numpy.random.uniform(-100, 100, size=(n_examples, *input_shape))

# Convert to ONNX
tf2onnx.convert.from_keras(keras_model, opset=14, output_path="tmp.model.onnx")

onnx_model = onnx.load("tmp.model.onnx")
onnx.checker.check_model(onnx_model)

# Compile
quantized_module = compile_onnx_model(
    onnx_model, input_set, n_bits=2
)

# Create test data from the same distribution and quantize using
# learned quantization parameters during compilation
x_test = tuple(numpy.random.uniform(-100, 100, size=(1, *input_shape)) for _ in range(num_inputs))

y_clear = quantized_module.forward(*x_test, fhe="disable")
y_fhe = quantized_module.forward(*x_test, fhe="execute")

print("Execution in clear: ", y_clear)
print("Execution in FHE:   ", y_fhe)
print("Equality:           ", numpy.sum(y_clear == y_fhe), "over", numpy.size(y_fhe), "values")

Quantization Aware Training

Models trained using Quantization Aware Training contain quantizers in the ONNX graph. These quantizers ensure that the inputs to the Linear/Dense and Conv layers are quantized. Since these QAT models have quantizers that are configured during training to a specific number of bits, the ONNX graph will need to be imported using the same settings:

# Define the number of bits to use for quantizing weights and activations during training
n_bits_qat = 3  

quantized_numpy_module = compile_onnx_model(
    onnx_model,
    input_set,
    import_qat=True,
    n_bits=n_bits_qat,
)

Supported operators

The following operators are supported for evaluation and conversion to an equivalent FHE circuit. Other operators were not implemented, either due to FHE constraints or because they are rarely used in PyTorch activations or scikit-learn models.

• Abs

• Acos

• Acosh

• Add

• Asin

• Asinh

• Atan

• Atanh

• AveragePool

• BatchNormalization

• Cast

• Celu

• Clip

• Concat

• Constant

• ConstantOfShape

• Conv

• Cos

• Cosh

• Div

• Elu

• Equal

• Erf

• Exp

• Expand

• Flatten

• Floor

• Gather

• Gemm

• Greater

• GreaterOrEqual

• HardSigmoid

• HardSwish

• Identity

• LeakyRelu

• Less

• LessOrEqual

• Log

• MatMul

• Max

• MaxPool

• Min

• Mul

• Neg

• Not

• Or

• PRelu

• Pad

• Pow

• ReduceSum

• Relu

• Reshape

• Round

• Selu

• Shape

• Sigmoid

• Sign

• Sin

• Sinh

• Slice

• Softplus

• Squeeze

• Sub

• Tan

• Tanh

• ThresholdedRelu

• Transpose

• Unfold

• Unsqueeze

• Where

• onnx.brevitas.Quant

Start here

• Build-in model examples

• Deep learning examples

Go further

Live demos on Hugging Face:

• Encrypted anonymization: Encrypted anonymization uses Fully Homomorphic Encryption (FHE) to anonymize personally identifiable information (PII) within encrypted documents, enabling computations to be performed on the encrypted data.

  • Check the code

• Credit card approval: Predicting credit scoring card approval application in which sensitive data can be shared and analyzed without exposing the actual information to neither the three parties involved, nor the server processing it.

  • Check the code

• Sentiment analysis with transformers: predicting if an encrypted tweet / short message is positive, negative or neutral, using FHE.

  • Check the code and the blog post

• Health diagnosis: giving a diagnosis using FHE to preserve the privacy of the patient based on a patient's symptoms, history and other health factors.

  • Check the code

• Encrypted image filtering: filtering encrypted images by applying filters such as black-and-white, ridge detection, or your own filter.

  • Check the code

Code examples on Github:

• GPT-2 in FHE: Privacy-preserving text generation based on a user's prompt

• Titanic: Train an XGB classifier that can perform encrypted prediction for the Kaggle Titanic competition

• Federated learning and private inference: Use federated learning to train a Logistic Regression while preserving training data confidentiality. Import the model into Concrete ML and perform encrypted prediction

• Neutral network fine-tuning: Fine-tune a VGG network to classify the CIFAR image data-sets and predict on encrypted data

• Encrypted sentiment analysis:A Hugging Face space that securely analyzes the sentiment expressed in a short text

• Credit scoring: Predict the chance of a given loan applicant defaulting on loan repayment

Blog tutorials:

• Running privacy-preserving inferences on Hugging Face endpoints - April 2024

• Build an end-to-end encrypted Shazam application using Concrete ML - February 2024

• Linear regression over encrypted data with homomorphic encryption - June 2023

• Comparison of Concrete ML regressors - June 2023

• How to deploy a machine learning model with Concrete ML - May 2023

• Encrypted image filtering using homomorphic encryption - February 2023

• Sentiment analysis over encrypted data - November 2022

• Titanic Competition with Privacy Preserving Machine Learning - August 2022

Video tutorials

• Work with encrypted DataFrames using Concrete ML - May 2024

• Train a linear classifier on encrypted data using Concrete ML and Fully Homomorphic Encryption (FHE) - February 2024

• How to convert a scikit-learn model into its homomorphic equivalent - June 2023

Concrete ML has support for serializing all available built-in models. Using this feature, one can dump a fitted and compiled model into a JSON string or file. The estimator can then be loaded back using the JSON object.

Saving Models

All built-in models provide the following methods:

• dumps: dumps the model as a string.

• dump: dumps the model into a file.

For example, a logistic regression model can be dumped in a string as below.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

from concrete.ml.sklearn import LogisticRegression

# Create the data for classification:
X, y = make_classification()

# Retrieve train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4)

# Instantiate, train and compile the model
model = LogisticRegression()
model.fit(X_train, y_train)
model.compile(X_train)

# Run the inference in FHE
y_pred_fhe = model.predict(X_test, fhe="execute")

# Dump the model in a string
dumped_model_str = model.dumps()

Similarly, it can be dumped into a file.

from pathlib import Path

dumped_model_path = Path("logistic_regression_model.json")

# Any kind of file-like object can be used 
with dumped_model_path.open("w") as f:

    # Dump the model in a file
    model.dump(f)

Alternatively, Concrete ML provides two equivalent global functions.

from concrete.ml.common.serialization.dumpers import dump, dumps

# Dump the model in a string
dumped_model_str = dumps(model)

# Any kind of file-like object can be used 
with dumped_model_path.open("w") as f:

    # Dump the model in a file
    dump(model, f)

Loading Models

Loading a built-in model is possible through the following functions:

• loads: loads the model from a string.

• load: loads the model from a file.

The above logistic regression model can therefore be loaded as below.

import numpy
from concrete.ml.common.serialization.loaders import load, loads

# Load the model from a string
loaded_model = loads(dumped_model_str)

# Any kind of file-like object can be used 
with dumped_model_path.open("r") as f:

    # Load the model from a file
    loaded_model = load(f)

# Compile the model
loaded_model.compile(X_train)

# Run the inference in FHE using the loaded model
y_pred_fhe_loaded = loaded_model.predict(X_test, fhe="execute")

print("Predictions are equal:", numpy.array_equal(y_pred_fhe, y_pred_fhe_loaded))

# Output:
#   Predictions are equal: True

These examples illustrate the basic usage of built-in Concrete ML models. For more examples showing how to train high-accuracy models on more complex data-sets, see the Demos and Tutorials section.

FHE constraints

In Concrete ML, built-in linear models are exact equivalents to their scikit-learn counterparts. As they do not apply any non-linearity during inference, these models are very fast (~1ms FHE inference time) and can use high-precision integers (between 20-25 bits).

Tree-based models apply non-linear functions that enable comparisons of inputs and trained thresholds. Thus, they are limited with respect to the number of bits used to represent the inputs. But as these examples show, in practice 5-6 bits are sufficient to exactly reproduce the behavior of their scikit-learn counterpart models.

In the examples below, built-in neural networks can be configured to work with user-specified accumulator sizes, which allow the user to adjust the speed/accuracy trade-off.

List of examples

1. Linear models

• Linear Regression example

• Logistic Regression example

• Linear Support Vector Regression example

• Linear SVM classification

These examples show how to use the built-in linear models on synthetic data, which allows for easy visualization of the decision boundaries or trend lines. Executing these 1D and 2D models in FHE takes around 1 millisecond.

2. Generalized linear models

• Poisson Regression example

• Generalized Linear Models comparison

These two examples show generalized linear models (GLM) on the real-world OpenML insurance data-set. As the non-linear, inverse-link functions are computed, these models do not use PBS, and are, thus, very fast (~1ms execution time).

3. Decision tree

• Decision Tree Classifier

Using the OpenML spams data-set, this example shows how to train a classifier that detects spam, based on features extracted from email messages. A grid-search is performed over decision-tree hyper-parameters to find the best ones.

• Decision Tree Regressor

Using the House Price prediction data-set, this example shows how to train regressor that predicts house prices.

4. XGBoost and Random Forest classifier

• XGBoost/Random Forest example

This example shows how to train tree-ensemble models (either XGBoost or Random Forest), first on a synthetic data-set, and then on the Diabetes data-set. Grid-search is used to find the best number of trees in the ensemble.

5. XGBoost regression

• XGBoost Regression example

Privacy-preserving prediction of house prices is shown in this example, using the House Prices data-set. Using 50 trees in the ensemble, with 5 bits of precision for the input features, the FHE regressor obtains an $R^2$ score of 0.90 and an execution time of 7-8 seconds.

6. Fully connected neural network

• NN Iris example

• NN MNIST example

Two different configurations of the built-in, fully-connected neural networks are shown. First, a small bit-width accumulator network is trained on Iris and compared to a PyTorch floating point network. Second, a larger accumulator (>8 bits) is demonstrated on MNIST.

7. Comparison of models

• Classifier comparison

• Regressor comparison

Based on three different synthetic data-sets, all the built-in classifiers are demonstrated in this notebook, showing accuracies, inference times, accumulator bit-widths, and decision boundaries.

7. Training on encrypted data

• LogisticRegression training

This example shows how to configure a training algorithm that works on encrypted data and how to deploy it in a client/server application.

FHE enables cloud applications to process private user data without running the risk of data leaks. Furthermore, deploying ML models in the cloud is advantageous as it eases model updates, allows to scale to large numbers of users by using large amounts of compute power, and protects model IP by keeping the model on a trusted server instead of the client device.

However, not all applications can be easily converted to FHE computation and the computation cost of FHE may make a full conversion exceed latency requirements.

Hybrid models provide a balance between on-device deployment and cloud-based deployment. This approach entails executing parts of the model directly on the client side, while other parts are securely processed with FHE on the server side. Concrete ML facilitates the hybrid deployment of various neural network models, including MLP (multilayer perceptron), CNN (convolutional neural network), and Large Language Models.

The hybrid model deployment API provides an easy way to integrate the standard deployment procedure into neural network style models that are compiled with compile_brevitas_qat_model or compile_torch_model.

Compilation

To use hybrid model deployment, the first step is to define what part of the PyTorch neural network model must be executed in FHE. The model part must be a nn.Module and is identified by its key in the original model's .named_modules().

import numpy as np
import os
import torch

from pathlib import Path
from torch import nn

from concrete.ml.torch.hybrid_model import HybridFHEModel, tuple_to_underscore_str
from concrete.ml.deployment import FHEModelServer


class FCSmall(nn.Module):
    """Torch model for the tests."""

    def __init__(self, dim):
        super().__init__()
        self.seq = nn.Sequential(nn.Linear(dim, dim), nn.ReLU(), nn.Linear(dim, dim))

    def forward(self, x):
        return self.seq(x)

model = FCSmall(10)
model_name = "FCSmall"
submodule_name = "seq.0"

inputs = torch.Tensor(np.random.uniform(size=(10, 10)))
# Prints ['', 'seq', 'seq.0', 'seq.1', 'seq.2']
print([k for (k, _) in model.named_modules()])

# Create a hybrid model
hybrid_model = HybridFHEModel(model, [submodule_name])
hybrid_model.compile_model(
    inputs,
    n_bits=8,
)


models_dir = Path(os.path.abspath('')) / "compiled_models"
models_dir.mkdir(exist_ok=True)
model_dir = models_dir / model_name

hybrid_model.save_and_clear_private_info(model_dir, via_mlir=True)

Server Side Deployment

The save_and_clear_private_info function serializes the FHE circuits corresponding to the various parts of the model that were chosen to be moved server-side. It also saves the client-side model, removing the weights of the layers that are transferred server-side. Furthermore it saves all necessary information required to serve these sub-models with FHE, using the FHEModelDev class.

The FHEModelServer class should be used to create a server application that creates end-points to serve these sub-models:

input_shape_subdir = tuple_to_underscore_str( (1,) + inputs.shape[1:] )
MODULES = { model_name: { submodule_name: {"path":  model_dir / submodule_name / input_shape_subdir }}}
server =  FHEModelServer(str(MODULES[model_name][submodule_name]["path"]))

For more information about serving FHE models, see the client/server section.

Client Side

A client application that deploys a model with hybrid deployment can be developed in a very similar manner to on-premise deployment: the model is loaded normally with PyTorch, but an extra step is required to specify the remote endpoint and the model parts that are to be executed remotely.

# Modify model to use remote FHE server instead of local weights
hybrid_model = HybridFHEModel(
    model,
    submodule_name,
    server_remote_address="http://0.0.0.0:8000",
    model_name=f"{model_name}",
    verbose=False,
)

Next, the client application must obtain the parameters necessary to encrypt and quantize data, as detailed in the client/server documentation.

path_to_clients = Path(__file__).parent / "clients"
hybrid_model.init_client(path_to_clients=path_to_clients)

When the client application is ready to make inference requests to the server, it must set the operation mode of the HybridFHEModel instance to HybridFHEMode.REMOTE: