This document explains the essential cryptographic terms and the important concepts of Concrete ML model lifecycle with Fully Homomorphic Encryption (FHE).

Concrete ML is built on top of Concrete, which enables the conversion from NumPy programs into FHE circuits.

Lifecycle of a Concrete ML model

With Concrete ML, you can train a model on clear or encrypted data, then deploy it to predict on encrypted inputs. During deployment, data can be pre-processed while being encrypted. Therefore, data stay encrypted during the entire lifecycle of the machine learning model, with some limitations.

I. Model development

1. Training: A model is trained either using plaintext (non-encrypted) training data, or encrypted training data.

2. Quantization: Quantization converts inputs, model weights, and all intermediate values of the inference computation to integer equivalents. More information is available here. Concrete ML performs this step in two ways depending on model type:

   • During training (Quantization Aware Training): by adding quantization layers in the neural network model, weights can be forced to have discrete values and activation quantization parameters are optimized through gradient descent. QAT requires re-training a neural network with these quantization layers.

   • After training (Post Training Quantization): the floating point neural network is kept as-is and a calibration step determines quantization parameters for each layer. No re-training is necessary and thus, no training data or labels are needed to convert a neural network to FHE using PTQ.

3. Simulation: Simulation allows you to execute a model that was quantized, to measure its accuracy in FHE, and to determine the modifications required to make it FHE compatible. Simulation is described in more detail here.

4. Compilation: After quantizing the model and confirming that it has good FHE accuracy through simulation, the model then needs to be compiled using Concrete's FHE Compiler to produce an equivalent FHE circuit. This circuit is represented as an MLIR program consisting of low level cryptographic operations. You can read more about FHE compilation here, MLIR here, and about the low-level Concrete library here.

5. Inference: The compiled model can then be executed on encrypted data, once the proper keys have been generated. The model can also be deployed to a server and used to run private inference on encrypted inputs.

You can find examples of the model development workflow here.

II. Model deployment

1. Pre-processing: Data owners(client) can generate keys to encrypt/decrypt data and store it in a DataFrame for further processing on a server. The server can pre-process such data with pre-compiled circuits, to prepare it for encrypted training or inference.

2. Client/server model deployment: In a client/server setting, Concrete ML models can be exported to:

   • Allow the client to generate keys, encrypt, and decrypt.

   • Provide a compiled model that can run on the server to perform inference on encrypted data.

3. Key generation: The data owner (client) needs to generate a set of keys:

   • A private encryption key to encrypt/decrypt their data and results

   • A public evaluation key for the model's FHE evaluation on the server.

You can find an example of the model deployment workflow here.

Cryptography concepts

Concrete ML and Concrete abstract the details of the underlying cryptography scheme, TFHE. However, understanding some cryptography concepts is still useful:

• Encryption and decryption: Encryption converts human-readable information (plaintext) into data (ciphertext) that is unreadable by a human or computer unless with the proper key. Encryption takes plaintext and an encryption key and produces ciphertext, while decryption is the reverse operation.

• Encrypted inference: FHE allows third parties to execute a machine learning model on encrypted data. The inference result is also encrypted and can only be decrypted by the key holder.

• Key generation: Cryptographic keys are generated using random number generators. Key generation can be time-consuming and produce large keys, but each model used by a client only requires key generation once.

  • Private encryption key: A private encryption key is a series of bits used within an encryption algorithm for encrypting data so that the corresponding ciphertext appears random.

  • Public evaluation key: A public evaluation key is used to perform homomorphic operations on encrypted data, typically by a server.

• Guaranteed correctness of encrypted computations: To ensure security, TFHE adds random noise to ciphertexts. Depending on the noise parameters, it can cause errors during encrypted data processing. By default, Concrete ML uses parameters that guarantee the correctness of encrypted computations, so the results on encrypted data equals to those of simulations on clear data.

• Programmable Boostrapping (PBS) : Programmable Bootstrapping enables the homomorphic evaluation of any function of a ciphertext, with a controlled level of noise. Learn more about PBS in this paper.

For a deeper understanding of the cryptography behind the Concrete stack, refer to the whitepaper on TFHE and Programmable Boostrapping or this series of blogs.

Model accuracy considerations under FHE constraints

FHE requires all inputs, constants, and intermediate values to be integers of maximum 16 bits. To make machine learning models compatible with FHE, Concrete ML implements some techniques with accuracy considerations:

• Quantization: Concrete ML quantizes inputs, outputs, weights, and activations to meet FHE limitations. See the quantization documentation for details.

  • Accuracy trade-off: Quantization may reduce accuracy, but careful selection of quantization parameters or of the training approach can mitigate this. Concrete ML offers built-in quantized models; users only configure parameters like bit-width. For more details of quantization configurations, see the advanced quantization guide.

• Additional methods: Dimensionality reduction and pruning are additional ways to make programs compatible for FHE. See Poisson regression example for dimensionality reduction and built-in neural networks for pruning.

This page explains Concrete ML linear models for both classification and regression. These models are based on scikit-learn linear models.

Supported models for encrypted inference

The following models are supported for training on clear data and predicting on encrypted data. Their API is similar the one of scikit-learn. These models are also compatible with some of scikit-learn's main workflows, such as Pipeline() and GridSearch().

Supported models for encrypted training

In addition to predicting on encrypted data, the following models support training on encrypted data.

| SGDClassifier | SGDClassifier |

Quantization parameters

The n_bits parameter controls the bit-width of the inputs and weights of the linear models. Linear models do not use table lookups and thus alllows weight and inputs to be high precision integers.

For models with input dimensions up to 300, the parameter n_bits can be set to 8 or more. When the input dimensions are larger, n_bits must be reduced to 6-7. In many cases, quantized models can preserve all performance metrics compared to the non-quantized float models from scikit-learn when n_bits is down to 6. You should validate accuracy on held-out test sets and adjust n_bits accordingly.

Pre-trained models

You can convert an already trained scikit-learn linear model to a Concrete ML one by using the from_sklearn_model method. See the following example.

Example

The following example shows how to train a LogisticRegression model on a simple data-set and then use FHE to perform inference on encrypted data. You can find a more complete example in the LogisticRegression notebook.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

from concrete.ml.sklearn import LogisticRegression

# Create the data for classification:
X, y = make_classification(
    n_features=30,
    n_redundant=0,
    n_informative=2,
    random_state=2,
    n_clusters_per_class=1,
    n_samples=250,
)

# Retrieve train and test sets:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)

# Instantiate the model:
model = LogisticRegression(n_bits=8)

# Fit the model:
model.fit(X_train, y_train)

# Evaluate the model on the test set in clear:
y_pred_clear = model.predict(X_test)

# Compile the model:
model.compile(X_train)

# Perform the inference in FHE:
y_pred_fhe = model.predict(X_test, fhe="execute")

# Assert that FHE predictions are the same as the clear predictions:
print(
    f"{(y_pred_fhe == y_pred_clear).sum()} examples over {len(y_pred_fhe)} "
    "have an FHE inference equal to the clear inference."
)

# Output:
#  100 examples over 100 have an FHE inference equal to the clear inference

Model accuracy

The figure below compares the decision boundary of the FHE classifier and a scikit-learn model executed in clear. You can find the complete code in the LogisticRegression notebook.

The overall accuracy scores are identical (93%) between the scikit-learn model (executed in the clear) and the Concrete ML one (executed in FHE). In fact, quantization has little impact on the decision boundaries, as linear models can use large precision numbers when quantizing inputs and weights in Concrete ML. Additionally, as the linear models do not use Programmable Boostrapping, the FHE computations are always exact, irrespective of the PBS error tolerance setting. This ensures that the FHE predictions are always identical to the quantized clear ones.

Loading a pre-trained model

An alternative to the example above is to train a scikit-learn model in a separate step and then to convert it to Concrete ML.

from sklearn.linear_model import LogisticRegression as SKlearnLogisticRegression

# Instantiate the model:
model = SKlearnLogisticRegression()

# Fit the model:
model.fit(X_train, y_train)

cml_model = LogisticRegression.from_sklearn_model(model, X_train, n_bits=8)

# Compile the model:
cml_model.compile(X_train)

# Perform the inference in FHE:
y_pred_fhe = cml_model.predict(X_test, fhe="execute")

This document illustrate how Concrete ML model and DataFrames are deployed in client/server setting when creating privacy-preserving services in the cloud.

Once compiled to FHE, a Concrete ML model or DataFrame generates machine code that execute prediction, training or pre-processing on encrypted data. During this process, Concrete ML generates the private encryption keys and the pubic evaluation keys.

Communication protocols

The overall communications protocol to enable cloud deployment of machine learning services can be summarized in the following diagram:

The steps detailed above are:

1. Model Deployment: The model developer deploys the compiled machine learning model to the server. This model includes the cryptographic parameters. The server is now ready to provide private inference. Cryptographic parameters and compiled programs for DataFrames are included directly in Concrete ML.

2. Client request: The client requests the cryptographic parameters (client specs). Once the client receives them from the server, the secret and evaluation keys are generated.

3. Key exchanges: The client sends the evaluation key to the server. The server is now ready to accept requests from this client. The client sends their encrypted data. Serialized DataFrames include client evaluation keys.

4. Private inference: The server uses the evaluation key to securely run prediction, training and pre-processing on the user's data and sends back the encrypted result.

5. Decryption: The client now decrypts the result and can send back new requests.

For more information on how to implement this basic secure inference protocol, refer to the Production Deployment section and to the client/server example. For information on training on encrypted data, see the corresponding section.

Get started

Learn the basics of Concrete ML, set it up, and make it run with ease.

Build with Concrete ML

Start building with Concrete ML by exploring its core features, discovering essential guides, and learning more with user-friendly tutorials.

Explore more

Access to additional resources and join the Zama community.

References & Explanations

Refer to the API, review product architecture, and access additional resources for in-depth explanations while working with Concrete ML.

• Security and correctness

• API

• Quantization

• Pruning

• Compilation

• Advanced features

• Project architecture

Support channels

Ask technical questions and discuss with the community. Our team of experts usually answers within 24 hours in working days.

• Community channels

Developers

Collaborate with us to advance the FHE spaces and drive innovation together.

• Contribute to Concrete ML

• Check the latest release note

• Request a feature

• Report a bug

Concrete ML is an open source, privacy-preserving, machine learning framework based on Fully Homomorphic Encryption (FHE). It enables data scientists without any prior knowledge of cryptography to perform:

• Automatic model conversion: Use familiar APIs from scikit-learn and PyTorch to convert machine learning models to their FHE equivalent. This is applicable for , , and ).

• Encrypted data training: directly on encrypted data to maintain privacy.

• Encrypted data pre-processing: using a DataFrame paradigm.

Key features

Example usage

This example shows the typical flow of a Concrete ML model:

1. Training the model: Train the model on unencrypted (plaintext) data using scikit-learn. Since Fully Homomorphic Encryption (FHE) operates over integers, Concrete ML quantizes the model to use only integers during inference.

2. Compiling the model: Compile the quantized model to an FHE equivalent. Under the hood, the model is first converted to a Concrete Python program and then compiled.

It is also possible to call encryption, model prediction, and decryption functions separately as follows. Executing these steps separately is equivalent to calling predict_proba on the model instance.

Current limitations

• Precision and accuracy: In order to run models in FHE, Concrete ML requires models to be within the precision limit, currently 16-bit integers. Thus, machine learning models must be quantized and it sometimes leads to a loss of accuracy versus the original model that operates on plaintext.

• Models availability: Concrete ML currently only supports training on encrypted data for some models, while it supports inference for a large variety of models.

• Processing: Concrete currently doesn't support pre-processing model inputs and post-processing model outputs. These processing stages may involve:

  • Text-to-numerical feature transformation

  • Dimensionality reduction

  • KNN or clustering

  • Featurization

  • Normalization

  • The mixing of ensemble models' results.

These issues are currently being addressed, and significant improvements are expected to be released in the near future.

Concrete stack

Online demos and tutorials

If you have built awesome projects using Concrete ML, feel free to let us know and we'll link to your work!

Additional resources

Support

This document provides guides on how to install Concrete ML using PyPi or Docker.

Prerequisite

Before you start, determine your environment:

• Hardware platform

• Operating System (OS) version

• Python version

OS/HW support

Depending on your OS/HW, Concrete ML may be installed with Docker or with pip:

Python support

• Version: In the current release, Concrete ML supports only 3.8, 3.9, 3.10 and 3.11 versions of python.

• Linux requirement: The Concrete ML Python package requires glibc >= 2.28. On Linux, you can check your glibc version by running ldd --version.

• Kaggle installation: Concrete ML can be installed on Kaggle () and on Google Colab.

Most of these limits are shared with the rest of the Concrete stack (namely Concrete Python). Support for more platforms will be added in the future.

Installation using PyPi

Requirements

Installing Concrete ML using PyPi requires a Linux-based OS or macOS (both x86 and Apple Silicon CPUs are supported).

Installation

To install Concrete ML from PyPi, run the following:

This will automatically install all dependencies, notably Concrete.

Installation using Docker

You can install Concrete ML using Docker by either pulling the latest image or a specific version:

This will launch a Concrete ML enabled Jupyter server in Docker that can be accessed directly from a browser.

Alternatively, you can launch a shell in Docker, with or without volumes:

This document introduces how to construct and perform operations on encrypted DataFrames using Fully Homomorphic Encryption (FHE).

Introduction

Encrypted DataFrames are a storage format for encrypted tabular data. You can exchange encrypted DataFrames with third parties to collaborate without privacy risks. Potential applications include:

• Encrypt storage of tabular data-sets

• Joint data analysis efforts between multiple parties

• Data preparation steps before machine learning tasks, such as inference or training

• Secure outsourcing of data analysis to untrusted third parties

Encryption and decryption

To encrypt a pandas DataFrame, construct a ClientEngine that manages keys and then call the encrypt_from_pandas function:

from concrete.ml.pandas import ClientEngine
from io import StringIO
import pandas

data_left = """index,total_bill,tip,sex,smoker
1,12.54,2.5,Male,No
2,11.17,1.5,Female,No
3,20.29,2.75,Female,No
"""

# Load your pandas DataFrame
df = pandas.read_csv(StringIO(data_left))

# Obtain client object
client = ClientEngine(keys_path="my_keys")

# Encrypt the DataFrame
df_encrypted = client.encrypt_from_pandas(df)

# Decrypt the DataFrame to produce a pandas DataFrame
df_decrypted = client.decrypt_to_pandas(df_encrypted)

Supported data types and schema definition

• Integer: Integers are supported within a specific range determined by the encryption scheme's quantization parameters. Default range is 1 to 15. 0 being used for the NaN. Values outside this range will cause a ValueError to be raised during the pre-processing stage.

• Quantized Float: Floating-point numbers are quantized to integers within the supported range. This is achieved by computing a scale and zero point for each column, which are used to map the floating-point numbers to the quantized integer space.

• String Enum: String columns are mapped to integers starting from 1. This mapping is stored and later used for de-quantization. If the number of unique strings exceeds 15, a ValueError is raised.

Using a user-defined schema

Before encryption, the data is preprocessed. For example string enums first need to be mapped to integers, and floating point values must be quantized. By default, this mapping is done automatically. However, when two different clients encrypt their data separately, the automatic mappings may differ, possibly due to some missing values in one of the client's DataFrame. Thus the column can not be selected when merging encrypted DataFrames.

The encrypted DataFrame supports user-defined mappings. These schemas are defined as a dictionary where keys represent column names and values contain meta-data about the column. Supported column meta-data are:

• string columns: mapping between string values and integers.

• float columns: the min/max range that the column values lie in.

schema = {
    "string_column": {"abc": 1, "bcd": 2 },
    "float_column": {"min": 0.1, "max": 0.5 }
}

Supported operations

Encrypted DataFrame is designed to support a subset of operations that are available for pandas DataFrames. For now, only the merge operation is supported. More operations will be added in the future releases.

Merge operation

Merge operation allows you to left or right join two DataFrames.

[!NOTE] The merge operation on Encrypted DataFrames can be securely performed on a third-party server, meaning that the server can execute the merge without ever having access to the unencrypted data. The server only requires the encrypted DataFrames.

df_right = """index,day,time,size
2,Thur,Lunch,2
5,Sat,Dinner,3
9,Sun,Dinner,2"""

# Encrypt the DataFrame
df_encrypted2 = client.encrypt_from_pandas(pandas.read_csv(StringIO(df_right)))

df_encrypted_merged = df_encrypted.merge(df_encrypted2, how="left", on="index")

Serialization

You can serialize encrypted DataFrame objects to a file format for storage or transfer. When serialized, they contain the encrypted data and public evaluation keys necessary to perform computations.

[!NOTE] Serialized DataFrames do not contain any private encryption keys . The DataFrames can be exchanged with any third-party without any risk.

To save or load an encrypted DataFrame from a file, use the following commands:

from concrete.ml.pandas import load_encrypted_dataframe

# Save
df_encrypted_merged.save("df_encrypted_merged")

# Load
df_encrypted_merged = load_encrypted_dataframe("df_encrypted_merged")

# Decrypt the DataFrame
df_decrypted = client.decrypt_to_pandas(df_encrypted)

Error handling

During the pre-processing and post-processing stages, the ValueError can happen in the following situations:

• A column contains values outside the allowed range for integers

• Too many unique strings

• Unsupported data type by Concrete ML

• Unsupported data type by the operation attempted

Example workflow

An example workflow where two clients encrypt two DataFrame objects, perform a merge operation on the server side, and then decrypt the results is available in the notebook encrypted_pandas.ipynb.

Current limitations

While this API offers a new secure way to work on remotely stored and encrypted data, it has some strong limitations at the moment:

• Precision of Values: The precision for numerical values is limited to 4 bits.

• Supported Operations: The merge operation is the only one available.

• Index Handling: Index values are not preserved; users should move any relevant data from the index to a dedicated new column before encrypting.

• Integer Range: The range of integers that can be encrypted is between 1 and 15.

• Uniqueness for merge: The merge operation requires that the columns to merge on contain unique values. Currently this means that data-frames are limited to 15 rows.

• Metadata Security: Column names and the mapping of strings to integers are not encrypted and are sent to the server in clear text.

This document explains how to train SGD Logistic Regression on encrypted data.

Introduction

Training on encrypted data is done through an FHE program that is generated by Concrete ML, based on the characteristics of the data that are given to the fit function. Once the FHE program associated with the SGDClassifier object has fit the encrypted data, it performs specifically to that data's distribution and dimensionality.

When deploying encrypted training services, you need to consider the type of data that future users of your services will train on:

• The distribution of the data should match to achieve good accuracy

• The dimensionality of the data needs to match since the deployed FHE programs are compiled for a fixed number of dimensions.

See the deployment section for more details.

Example

The logistic regression training example shows logistic regression training on encrypted data in action.

The following snippet shows how to instantiate a logistic regression model that trains on encrypted data:

from concrete.ml.sklearn import SGDClassifier
parameters_range = (-1.0, 1.0)

model = SGDClassifier(
    random_state=42,
    max_iter=50,
    fit_encrypted=True,
    parameters_range=parameters_range,
)

To activate encrypted training, simply set fit_encrypted=True in the constructor. When the value is set, Concrete ML generates an FHE program which, when called through the fit function, processes encrypted training data, labels and initial weights and outputs trained model weights. If this value is not set, training is performed on clear data using scikit-learn gradient descent.

Next, to perform the training on encrypted data, call the fit function with the fhe="execute" argument:

model.fit(X_binary, y_binary, fhe="execute")

Training configuration

The max_iter parameter controls the number of batches that are processed by the training algorithm.

The parameters_range parameter determines the initialization of the coefficients and the bias of the logistic regression. It is recommended to give values that are close to the min/max of the training data. It is also possible to normalize the training data so that it lies in the range $[-1, 1]$.

Capabilities and Limitations

The trainable logistic model uses Stochastic Gradient Descent (SGD) and quantizes the data, weights, gradients and the error measure. It currently supports training 6-bit models, including g both the coefficients and the bias.

The SGDClassifier does not currently support training models with other bit-width values. The execution time to train a model is proportional to the number of features and the number of training examples in the batch. The SGDClassifier training does not currently support client/server deployment for training.

Deployment

Once you have tested an SGDClassifier that trains on encrypted data, you can build an FHE training service by deploying the FHE training program of the SGDClassifier. See the Production Deloyment page for more details on how to the Concrete ML deployment utility classes. To deploy an FHE training program, you must pass the mode='training' parameter to the FHEModelDev class.

This document introduces the nearest neighbors non-parametric classification models that Concrete ML provides with a scikit-learn interface through the KNeighborsClassifier class.

Example

from concrete.ml.sklearn import KNeighborsClassifier

concrete_classifier = KNeighborsClassifier(n_bits=2, n_neighbors=3)

Quantization parameters

The KNeighborsClassifier class quantizes the training data-set provided to .fit using the specified number of bits (n_bits). To comply with accumulator size constraints, you must keep this value low. The model's accuracy will depend significantly on a well-chosen n_bits value and the dimensionality of the data.

The predict method of the KNeighborsClassifier performs the following steps:

1. Quantize the test vectors on clear data

2. Compute the top-k class indices of the closest training set vector on encrypted data

3. Vote for the top-k class labels to find the class for each test vector, performed on clear data

Inference time considerations

The FHE inference latency of this model is heavily influenced by the n_bits and the dimensionality of the data. Additionally, the data-set size has a linear impact on the data complexity. The number of nearest neighbors (n_neighbors) also affects performance.

The KNN computation executes in FHE in $O(Nlog^2k)$ steps, where $N$ is the training data-set size and $k$ is n_neighbors. Each step requires several PBS operations, with their runtime affected by the factors listed above. These factors determine the precision needed to represent the distances between test vectors and training data-set vectors. The PBS input precision required by the circuit is related to the precision of the distance values.

This document introduces the simple built-in neural networks models that Concrete ML provides with a scikit-learn interface through the NeuralNetClassifier and NeuralNetRegressor classes.

Supported models

The neural network models are implemented with skorch, which provides a scikit-learn-like interface to Torch models (more here).

Concrete ML models are multi-layer, fully-connected, networks with customizable activation functions and have a number of neurons in each layer. This approach is similar to what is available in scikit-learn when using the MLPClassifier/MLPRegressor classes. The built-in models train easily with a single call to .fit(), which will automatically quantize weights and activations. These models use Quantization Aware Training, allowing good performance for low precision (down to 2-3 bits) weights and activations.

While NeuralNetClassifier and NeuralNetClassifier provide scikit-learn-like models, their architecture is somewhat restricted to make training easy and robust. If you need more advanced models, you can convert custom neural networks as described in the FHE-friendly models documentation.

Example

To create an instance of a Fully Connected Neural Network (FCNN), you need to instantiate one of the NeuralNetClassifier and NeuralNetRegressor classes and configure a number of parameters that are passed to their constructor.

Note that some parameters need to be prefixed by module__, while others don't. The parameters related to the model must have the prefix, such as the underlying nn.Module. The parameters related to training options do not require the prefix.

from concrete.ml.sklearn import NeuralNetClassifier
import torch.nn as nn

n_inputs = 10
n_outputs = 2
params = {
    "module__n_layers": 2,
    "max_epochs": 10,
}

concrete_classifier = NeuralNetClassifier(**params)

The Classifier Comparison notebook shows the behavior of built-in neural networks on several synthetic data-sets.

The folowing figure shows the Concrete ML neural network trained with Quantization Aware Training in an FHE-compatible configuration and compares it to the floating-point equivalent trained with scikit-learn.

Architecture parameters

• module__n_layers: number of layers in the FCNN.

  • This parameter must be at least 1. Note that this is the total number of layers. For a single, hidden layer NN model, set module__n_layers=2

• module__activation_function: can be one of the Torch activations (such as nn.ReLU)

  • See the full list of Torch activations here.

  • Neural networks with nn.ReLU activation benefit from specific optimizations that make them around 10x faster than networks with other activation functions.

Quantization parameters

• n_w_bits (default 3): number of bits for weights

• n_a_bits (default 3): number of bits for activations and inputs

• n_accum_bits: maximum accumulator bit-width that is desired

  • By default, this is unbounded, which, for weight and activation bit-width settings, may make the trained networks fail in compilation. When used, the implementation will attempt to keep accumulators under this bit-width through pruning (for example, setting some weights to zero).

• power_of_two_scaling (default True): forces quantization scales to be powers-of-two

  • When coupled with the ReLU activation, this optimize strongly the FHE inference time.

  • See this section in the quantization documentation for more details.

Training parameters (from skorch)

• max_epochs (default 10): The number of epochs to train the network

• verbose (default: False): Whether to log loss/metrics during training

• lr (default 0.001): Learning rate

You can find other parameters from skorch in the skorch documentation.

Advanced parameters

• module__n_hidden_neurons_multiplier (default 4): The number of hidden neurons.

  • This parameter will be automatically set proportional to the dimensionality of the input. It controls the proportionality factor. This value gives good accuracy while avoiding accumulator overflow.

  • See the pruning and quantization sections for more info.

Class weights

You can give weights to each class to use in training. Note that this must be supported by the underlying PyTorch loss function.

    from sklearn.utils.class_weight import compute_class_weight
    params["criterion__weight"] = compute_class_weight("balanced", classes=classes, y=y_train)

Overflow errors

The n_accum_bits parameter influences training accuracy by controlling the number of non-zero neurons allowed in each layer. You can increase n_accum_bits to improve accuracy, but must consider the precision limitations to avoid an overflow in the accumulator. The default value is a balanced choice that generally avoids overflow, but you may need to adjust it to reduce the network breadth if you encounter overflow errors.

The number of neurons in intermediate layers is controlled by the n_hidden_neurons_multiplier parameter. A value of 1 makes intermediate layers have the same number of neurons as the number as the input data dimensions.

This document introduces several scikit-learn's linear models for classification and regression tree models that Concrete ML provides.

Supported models

Concrete ML also supports XGBoost's XGBClassifier and XGBRegressor:

Pre-trained models

You can convert an already trained scikit-learn tree-based model to a Concrete ML one by using the from_sklearn_model method.

Example

Here's an example of how to use this model in FHE on a popular data-set using some of scikit-learn's pre-processing tools. You can find a more complete example in the XGBClassifier notebook.

from sklearn.datasets import load_breast_cancer
from sklearn.decomposition import PCA
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

from concrete.ml.sklearn.xgb import XGBClassifier


# Get data-set and split into train and test
X, y = load_breast_cancer(return_X_y=True)

# Split the train and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

# Define our model
model = XGBClassifier(n_jobs=1, n_bits=3)

# Define the pipeline
# We normalize the data and apply a PCA before fitting the model
pipeline = Pipeline(
    [("standard_scaler", StandardScaler()), ("pca", PCA(random_state=0)), ("model", model)]
)

# Define the parameters to tune
param_grid = {
    "pca__n_components": [2, 5, 10, 15],
    "model__max_depth": [2, 3, 5],
    "model__n_estimators": [5, 10, 20],
}

# Instantiate the grid search with 5-fold cross validation on all available cores
grid = GridSearchCV(pipeline, param_grid, cv=5, n_jobs=-1, scoring="accuracy")

# Launch the grid search
grid.fit(X_train, y_train)

# Print the best parameters found
print(f"Best parameters found: {grid.best_params_}")

# Output:
#  Best parameters found: {'model__max_depth': 5, 'model__n_estimators': 10, 'pca__n_components': 5}

# Currently we only focus on model inference in FHE
# The data transformation is done in clear (client machine)
# while the model inference is done in FHE on a server.
# The pipeline can be split into 2 parts:
#   1. data transformation
#   2. estimator
best_pipeline = grid.best_estimator_
data_transformation_pipeline = best_pipeline[:-1]
model = best_pipeline[-1]

# Transform test set
X_train_transformed = data_transformation_pipeline.transform(X_train)
X_test_transformed = data_transformation_pipeline.transform(X_test)

# Evaluate the model on the test set in clear
y_pred_clear = model.predict(X_test_transformed)
print(f"Test accuracy in clear: {(y_pred_clear == y_test).mean():0.2f}")

# In the output, the Test accuracy in clear should be > 0.9

# Compile the model to FHE
model.compile(X_train_transformed)

# Perform the inference in FHE
# Warning: this will take a while. It is recommended to run this with a very small batch of
# example first (e.g., N_TEST_FHE = 1)
# Note that here the encryption and decryption is done behind the scene.
N_TEST_FHE = 1
y_pred_fhe = model.predict(X_test_transformed[:N_TEST_FHE], fhe="execute")

# Assert that FHE predictions are the same as the clear predictions
print(f"{(y_pred_fhe == y_pred_clear[:N_TEST_FHE]).sum()} "
      f"examples over {N_TEST_FHE} have an FHE inference equal to the clear inference.")

# Output:
#  1 examples over 1 have an FHE inference equal to the clear inference

We can plot and compare the decision boundaries of the Concrete ML model and the classical XGBoost model executed in the clear. Here we show a 6-bit model to illustrate the impact of quantization on classification. You will find similar plots in the Classifier Comparison notebook.

Quantization parameters

When using a sufficiently high bit-width, quantization has little impact on the decision boundaries of the Concrete ML FHE decision tree model, as quantization is done individually on each input feature. It means FHE models can achieve similar accuracy levels as floating point models. Using 6 bits for quantization is effective in reaching or even exceeding floating point accuracy.

To adjust the number of bits for quantization, use the n_bits parameter. Setting n_bits to a low value may introduce artifacts, potentially reducing accuracy. However, the execution speed in FHE could improve. This adjustment allows you to manage the accuracy/speed trade-off. Additionally, you can recover some accuracy by increasing the n_estimators parameter.

The following graph shows that using 5-6 bits of quantization is usually sufficient to reach the performance of a non-quantized XGBoost model on floating point data. The metrics plotted are accuracy and F1-score on the spambase data-set.

FHE Inference time considerations

The inference time in FHE is strongly dependant on the maximum circuit bit-width. For trees, in most cases, the quantization bit-width will be the same as the circuit bit-width. Therefore, reducing the quantization bit-width to 4 or less will result in fast inference times. Adding more bits will increase FHE inference time exponentially.

In some rare cases, the bit-width of the circuit can be higher than the quantization bit-width. This could happen when the quantization bit-width is low but the tree-depth is high. In such cases, the circuit bit-width is upper bounded by ceil(log2(max_depth + 1) + 1).

For more information on the inference time of FHE decision trees and tree-ensemble models please see Privacy-Preserving Tree-Based Inference with Fully Homomorphic Encryption, arXiv:2303.01254.

This section explains how to fine-tune neural-network models and large language-models on private data. Small models can be be fine-tuned using a single-client/single-server setup. For optimal latency when fine-tuning larger models (e.g., GPT2 and bigger) you should consider distributed computation, with multiple worker nodes performing the training on encrypted data.

Overview

Concrete ML supports LORA, a parameter efficient fine-tuning (PEFT) approach, in the paradigm. LORA adds adapters, which contain a low number of fine-tunable weights, to the linear layers in an original model.

Concrete ML will outsource the logic of a model's original forward and backward passes to one or more remote servers. On the other hand, the forward and backward passes over the LORA weights, the loss computation and the weight updates are performed by the client side. As the number of LORA weights is low, this does not incur significant added computation time for the model training client machine. More than 99% of a model's weights can be outsourced for large LLMs.

The main benefit of hybrid-model LORA training is outsourcing the computation of the linear layers. In LLMs these layers have considerable size and performing inference and gradient computations for them requires significant hardware. Using Concrete ML, these computations can be securely outsourced, eliminating the memory bottleneck that previously constrained such operations.

Usage

Concrete ML integrates with the which adds LORA layer adapters to a model's linear layers. Here are the steps to convert a model to hybrid FHE LORA training.

1. Apply the peft LORA layers

The LoraConfig class from the peft package contains the various LORA parameters. It is possible to specify which layers have LORA adapters through the target_modules argument. Please refer to the documentation for a reference on the various config options.

2. Convert the LORA model to use custom Concrete ML layers

Concrete ML requires a conversion step for the peft model, adding FHE compatible layers. In this step the several fine-tuning parameters can be configured:

• the number of gradient accumulation steps: for LORA it is common to accumulate gradients over several gradient descent steps before updating weights.

• the optimizer parameters

• the loss function

3. Compile a hybrid FHE model for the LORA adapted PyTorch model

Next, a hybrid FHE model must be compiled in order to convert the selected outsourced layers to use FHE. Other layers will be executed on the client side. The back-and-forth communication of encrypted activations and gradients may require significant bandwidth.

4. Train the model on private data

Finally, the hybrid model can be trained, much in the same way a PyTorch model is trained. The client is responsible for generating and iterating on training data batches.

Additional options

Inference

One fine-tuned, the LORA hybrid FHE model can perform inference only, through the model.inference_model attribute of the hybrid FHE model.

Toggle LORA layers

To compare to the original model, it is possible to disable the LORA weights in order to use the original model for inference.

This document explains how to compile models in Concrete ML. This is particularly useful for importing models trained with Keras.

You can compile ONNX models by directly importing models that are already quantized with or by performing with Concrete ML.

Simple example

The following example shows how to compile an ONNX model using PTQ. The model was initially trained using Keras before being exported to ONNX. The training code is not shown here.

Quantization Aware Training

Models trained using QAT contain quantizers in the ONNX graph. These quantizers ensure that the inputs to the Linear/Dense and Conv layers are quantized. Since these QAT models have quantizers configured to a specific number of bits during training, you must import the ONNX graph using the same settings:

Supported operators

Concrete ML supports the following operators for evaluation and conversion to an equivalent FHE circuit. Other operators were not implemented either due to FHE constraints or because they are rarely used in PyTorch activations or scikit-learn models.

• Abs

• Acos

• Acosh

• Add

• Asin

• Asinh

• Atan

• Atanh

• AveragePool

• BatchNormalization

• Cast

• Celu

• Clip

• Concat

• Constant

• ConstantOfShape

• Conv

• Cos

• Cosh

• Div

• Elu

• Equal

• Erf

• Exp

• Expand

• Flatten

• Floor

• Gather

• Gemm

• Greater

• GreaterOrEqual

• HardSigmoid

• HardSwish

• Identity

• LeakyRelu

• Less

• LessOrEqual

• Log

• MatMul

• Max

• MaxPool

• Min

• Mul

• Neg

• Not

• OneHot

• Or

• PRelu

• Pad

• Pow

• ReduceSum

• Relu

• Reshape

• Round

• Selu

• Shape

• Sigmoid

• Sign

• Sin

• Sinh

• Slice

• Softplus

• Squeeze

• Sub

• Tan

• Tanh

• ThresholdedRelu

• Transpose

• Unfold

• Unsqueeze

• Where

• onnx.brevitas.Quant

This section provides a set of tools and guidelines to help users debug errors and build optimized models that are compatible with Fully Homomorphic Encryption (FHE).

Simulation

The of Concrete ML provides a way to evaluate, using clear data, the results that ML models produce on encrypted data. The simulation includes any probabilistic behavior FHE may induce. The simulation is implemented with .

The simulation mode can be useful when developing and iterating on an ML model implementation. As FHE non-linear models work with integers up to 16 bits, with a trade-off between the number of bits and the FHE execution speed, the simulation can help to find the optimal model design.

Simulation is much faster than FHE execution. This allows for faster debugging and model optimization. For example, this was used for the red/blue contours in the , as computing in FHE for the whole grid and all the classifiers would take significant time.

The following example shows how to use the simulation mode in Concrete ML.

Caching keys during debugging

It is possible to avoid re-generating the keys of the models you are debugging. This feature is unsafe and should not be used in production. Here is an example that shows how to enable key-caching:

Common compilation errors

1. TLU input maximum bit-width is exceeded

Error message: this [N]-bit value is used as an input to a table lookup

Possible solutions:

2. No crypto-parameters can be found

Error message: RuntimeError: NoParametersFound

Cause: This error occurs when cryptosystem parameters can not be found for the model bit-width, rounding mode and requested p_error, when using rounding_threshold_bits in the compile_torch_model function. With rounding_threshold_bits set, the 16-bit accumulator limit is relaxed, so the this [N]-bit value is used as an input to a table lookup does not occur. However, cryptosystem-parameters may still not exist for the model to be compiled.

3. Quantization import failed

Error message: Error occurred during quantization aware training (QAT) import [...] Could not determine a unique scale for the quantization!.

A common example is related to the concatenation operator. Suppose two tensors x and y are produced by two layers and need to be concatenated:

In the example above, the x and y layers need quantization before being concatenated.

Possible solutions:

1. If the error occurs for the first layer of the model: Add a QuantIdentity layer in your model and apply it on the input of the forward function, before the first layer is computed.

2. If the error occurs for a concatenation or addition layer: Add a new QuantIdentity layer in your model. Suppose it is called quant_concat. In the forward function, before concatenation of x and y, apply it to both tensors that are concatenated. The usage of a common Quantidentity layer to quantize both tensors that are concatenated ensures that they have the same scale:

PBS complexity and optimization

Furthermore, the cost of PBS depends on the bit-width of the compiled circuit. Every additional bit in the maximum bit-width significantly increase the complexity of the PBS. Therefore, it is important to determine the bit-width of the circuit and the amount of PBS it performs in order to optimize the performance.

To inspect the MLIR code produced by the compiler, use the following command:

Example output:

In the MLIR code, there are several calls to FHELinalg.apply_mapped_lookup_table and FHELinalg.apply_lookup_table. These calls apply PBS to the cells of their input tensors. For example, in the code above, the inputs are: tensor<1x5x!FHE.eint<15>> for both the first and last apply_mapped_lookup_table call. Thus, the PBS is applied 10 times, corresponding to the size of every encrypted tensor, which is 1x5 multiplied by 2.

To retrieve the bit-width of the circuit, use this command:

Reducing the number of bits and the number of PBS applications can significantly decrease the computation time of the compiled circuit.

This document explains how to implement machine learning models with Torch in Concrete ML, leveraging Fully Homomorphic Encryption (FHE).

Introduction

There are two approaches to build :

• : This method requires using custom layers to quantize weights and activations to low bit-widths. Concrete ML works with , a library that provides QAT support for PyTorch.

  • Use compile_brevitas_qat_model to compile models in this mode.

• : This method allows to compile a vanilla PyTorch model. However, accuracy may decrease significantly when quantizing weights and activations to fewer than 7 bits. On the other hand, depending on the model size, quantizing with 6-8 bits can be incompatible with FHE constraints. Thus you need to determine the trade-off between model accuracy and FHE compatibility.

  • Use compile_torch_model to compile models in this mode.

Both approaches require setting rounding_threshold_bits parameter accordingly. You should experiment to find the best values, starting with an initial value of 6. See for more details.

Quantization Aware training (QAT)

The following example uses a simple QAT PyTorch model that implements a fully connected neural network with two hidden layers. Due to its small size, making this model respect FHE constraints is relatively easy. To use QAT, Brevitas QuantIdentity nodes must be inserted in the PyTorch model, including one that quantizes the input of the forward function.

Post Training quantization (PTQ)

The following example demonstrates a simple PyTorch model that implements a fully connected neural network with two hidden layers. The model is compiled with compile_torch_model to use FHE.

Configuring quantization parameters

The quantization parameters, along with the number of neurons in each layer, determine the accumulator bit-width of the network. Larger accumulator bit-widths result in higher accuracy but slower FHE inference time.

QAT: Configure parameters such as bit_width and weight_bit_width. Set n_bits=None in the compile_brevitas_qat_model.

PTQ: Set the n_bits value in the compile_torch_model function. Manually determine the trade-off between accuracy, FHE compatibility, and latency.

Running encrypted inference

The model can now perform encrypted inference.

In this example, the input values x_test and the predicted values y_pred are floating points. The quantization (respectively de-quantization) step is done in the clear within the forward method, before (respectively after) any FHE computations.

Simulated FHE Inference in the clear

There are two approaches:

• quantized_module.forward(quantized_x, fhe="simulate"): This method simulates FHE execution taking into account Table Lookup errors. De-quantization must be done in a second step as for actual FHE execution. Simulation takes into account the p_error/global_p_error parameters

• quantized_module.forward(quantized_x, fhe="disable"): This method computes predictions in the clear on quantized data, and then de-quantize the result. The return value of this function contains the de-quantized (float) output of running the model in the clear. Calling this function on clear data is useful when debugging, but this does not perform actual FHE simulation.

Supported operators and activations

Concrete ML supports a variety of PyTorch operators that can be used to build fully connected or convolutional neural networks, with normalization and activation layers. Moreover, many element-wise operators are supported.

Operators

Univariate operators

Shape modifying operators

Tensor operators

Multi-variate operators: encrypted input and unencrypted constants

Concrete ML also supports some of their QAT equivalents from Brevitas.

• brevitas.nn.QuantLinear

• brevitas.nn.QuantConv1d

• brevitas.nn.QuantConv2d

Multi-variate operators: encrypted+unencrypted or encrypted+encrypted inputs

Quantizers

• brevitas.nn.QuantIdentity

Activation functions

This document introduces several approaches to reduce the overall latency of a neural network.

Introduction

Neural networks are challenging for encrypted inference. Each neuron in a network has to apply an activation function that requires a Programmable Bootstrapping(PBS) operation. The latency of a single PBS depends on the bit-width of its input.

Circuit bit-width optimization

Quantization Aware Training and pruning introduce specific hyper-parameters that influence the accumulator sizes. You can chose quantization and pruning configurations to reduce the accumulator size. To obtain a trade-off between latency and accuracy, you can manually set these hyper-parameters as described in the deep learning design guide.

Structured pruning

While using unstructured pruning ensures the accumulator bit-width stays low, structured pruning can eliminate entire neurons from the network as many neural networks are over-parametrized for easier training. You can apply structured pruning to a trained network as a fine-tuning step. This example demonstrates how to apply structured pruning to built-in neural networks using the prune helper function. To apply structured pruning to custom models, it is recommended to use the torch-pruning package.

Rounded activations and quantizers

Reducing the bit-width of inputs to the Table Lookup (TLU) operations significantly improves latency. Post-training, you can leverage properties of the fused activation and quantization functions in the TLUs to further reduce the accumulator size. This is achieved through the rounded PBS feature as described in the rounded activations and quantizers reference. Adjusting the rounding amount relative to the initial accumulator size can improve latency while maintaining accuracy.

TLU error tolerance adjustment

Finally, the TFHE scheme introduces a TLU error tolerance parameter that has an impact on crypto-system parameters that influence latency. A higher tolerance of TLU off-by-one errors results in faster computations but may reduce accuracy. You can think of the error of obtaining $T[x]$ as a Gaussian distribution centered on $x$: $TLU[x]$ is obtained with probability of 1 - p_error, while $T[x-1]$, $T[x+1]$ are obtained with much lower probability, etc. In Deep NNs, these type of errors can be tolerated up to some point. See the p_error documentation for details and more specifically the API for finding the best p_error.

This document explains how to perform encryption, execution, and decryption of Fully Homomorphic Encryption (FHE) using one function call of the Concrete ML API, or multiple function calls separately.

The APIs are different for the following:

• Built-in models

• Custom models

Built-in models

Using one function

All Concrete ML built-in models have a single predict method that performs the encryption, FHE execution, and decryption with only one function call.

The following example shows how to create a synthetic data-set and how to use it to train a LogisticRegression model from Concrete ML.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from concrete.ml.sklearn import LogisticRegression
import numpy

# Create a synthetic data-set for a classification problem
x, y = make_classification(n_samples=100, class_sep=2, n_features=3, n_informative=3, n_redundant=0, random_state=42)

# Split the data-set into a train and test set
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=42)

# Instantiate and train the model
model = LogisticRegression()
model.fit(x_train,y_train)

# Simulate the predictions in the clear (optional)
y_pred_clear = model.predict(x_test)

# Compile the model on a representative set
fhe_circuit = model.compile(x_train)

Concrete ML models follow the same API as scikit-learn models, transparently performing the steps related to encryption for convenience.

# Predict in FHE
y_pred_fhe = model.predict(x_test, fhe="execute")

Regarding this LogisticRegression model, as with scikit-learn, it is possible to predict the logits as well as the class probabilities by respectively using the decision_function or predict_proba methods instead.

Using separate functions

Alternatively, you can execute key generation, quantization, encryption, FHE execution and decryption separately.

# Generate the keys (set force to True in order to generate new keys at each execution)
fhe_circuit.keygen(force=True)

y_pred_fhe_step = []

for f_input in x_test:
    # Quantize an input (float)
    q_input = model.quantize_input([f_input])
    
    # Encrypt the input
    q_input_enc = fhe_circuit.encrypt(q_input)

    # Execute the linear product in FHE 
    q_y_enc = fhe_circuit.run(q_input_enc)

    # Decrypt the result (integer)
    q_y = fhe_circuit.decrypt(q_y_enc)

    # De-quantize the result
    y = model.dequantize_output(q_y)

    # Apply either the sigmoid if it is a binary classification task, which is the case in this 
    # example, or a softmax function in order to get the probabilities (in the clear)
    y_proba = model.post_processing(y)

    # Since this model does classification, apply the argmax to get the class predictions (in the clear)
    # Note that regression models won't need the following line
    y_class = numpy.argmax(y_proba, axis=1)

    y_pred_fhe_step += list(y_class)

y_pred_fhe_step = numpy.array(y_pred_fhe_step)

print("Predictions in clear:", y_pred_clear)
print("Predictions in FHE  :", y_pred_fhe_step)
print(f"Similarity: {int((y_pred_fhe_step == y_pred_clear).mean()*100)}%")

Custom models

For custom models, the API to execute inference in FHE or simulation is as follows:

from torch import nn
from brevitas import nn as qnn
from concrete.ml.torch.compile import compile_brevitas_qat_model

class FCSmall(nn.Module):
    """A small QAT NN."""

    def __init__(self, input_output):
        super().__init__()
        self.quant_input = qnn.QuantIdentity(bit_width=3)
        self.fc1 = qnn.QuantLinear(in_features=input_output, out_features=input_output, weight_bit_width=3, bias=True)
        self.act_f = nn.ReLU()
        self.fc2 = qnn.QuantLinear(in_features=input_output, out_features=input_output, weight_bit_width=3, bias=True)

    def forward(self, x):
        return self.fc2(self.act_f(self.fc1(self.quant_input(x))))

torch_model = FCSmall(3)

quantized_module = compile_brevitas_qat_model(
    torch_model,
    x_train,
)

x_test_q = quantized_module.quantize_input(x_test)
y_pred = quantized_module.quantized_forward(x_test_q, fhe="simulate")
y_pred = quantized_module.dequantize_output(y_pred)

y_pred = numpy.argmax(y_pred, axis=1)