Concrete-ML models can be easily deployed in a client/server setting, enabling the creation of privacy-preserving services in the cloud.

As seen in the concepts section, a Concrete-ML model, once compiled to FHE, generates machine code that performs the inference on private data. Furthermore, secret encryption keys are needed so that the user can securely encrypt their data and decrypt the inference result. An evaluation key is also needed for the server to securely process the user's encrypted data.

Keys are generated by the user once for each service they use, based on the model the service provides and its cryptographic parameters.

The overall communications protocol to enable cloud deployment of machine learning services can be summarized in the following diagram:

The steps detailed above are as follows:

1. The model developer deploys the compiled machine learning model to the server. This model includes the cryptographic parameters. The server is now ready to provide private inference.

2. The client requests the cryptographic parameters (also called "client specs"). Once it gets them from the server, the secret and evaluation keys are generated.

3. The client sends the evaluation key to the server. The server is now ready to accept requests from this client. The client sends their encrypted data.

4. The server uses the evaluation key to securely run inference on the user's data and sends back the encrypted result.

5. The client now decrypts the result and can send back new requests.

For more information on how to implement this basic secure inference protocol, refer to the Production Deployment section and to the client/server example.

| |

Concrete-ML is an open-source privacy-preserving machine learning inference framework based on fully homomorphic encryption (FHE). It enables data scientists without any prior knowledge of cryptography to automatically turn machine learning models into their FHE equivalent, using familiar APIs from Scikit-learn and PyTorch (see how it looks for , and ).

Fully Homomorphic Encryption (FHE) is an encryption technique that allows computing directly on encrypted data, without needing to decrypt it. With FHE, you can build private-by-design applications without compromising on features. You can learn more about FHE in , or by joining the community.

Example usage

Here is a simple example of classification on encrypted data using logistic regression. More examples can be found .

This example shows the typical flow of a Concrete-ML model:

• The model is trained on unencrypted (plaintext) data using scikit-learn. As FHE operates over integers, Concrete-ML quantizes the model to use only integers during inference.

• The quantized model is compiled to a FHE equivalent. Under the hood, the model is first converted to a Concrete-Numpy program, then compiled.

• Inference can then be done on encrypted data. The above example shows encrypted inference in the model development phase. Alternatively, in in a client/server setting, the data is encrypted by the client, processed securely by the server and then decrypted by the client.

Current limitations

To make a model work with FHE, the only constraint is to make it run within the supported precision limitations of Concrete-ML (currently 8-bit integers). Thus, machine learning models are required to be quantized, which sometimes leads to a loss of accuracy versus the original model operating on plaintext.

Additionally, Concrete-ML currently only supports FHE inference. On the other hand, training has to be done on unencrypted data, producing a model which is then converted to a FHE equivalent that can perform encrypted inference, i.e. prediction over encrypted data.

Finally, in Concrete-ML there is currently no support for pre-processing model inputs and for post-processing model outputs. These processing stages may involve text to numerical feature transformation, dimensionality reduction, KNN or clustering, featurization, normalization, and the mixing of results of ensemble models.

All of these issues are currently being addressed and significant improvements are expected to be released in the coming months.

Concrete Stack

Concrete-ML is built on top of Zama's Concrete framework. It uses , which itself uses the and the . To use these libraries directly, refer to the and documentations.

Online demos and tutorials.

Various tutorials are proposed for the and for . In addition, we also list standalone use-cases:

• : a Python and notebook showing a Quantization Aware Training (done with and following constraints of the package) and its corresponding use in Concrete-ML.

• : a notebook, which gives a solution to the . Done with XGBoost from Concrete-ML. It comes as a companion of , and was the subject of a blogpost in .

• : a gradio demo which predicts if a tweet / short message is positive, negative or neutral, in FHE of course! The demo is available on HuggingFace.

More generally, if you have built awesome projects using Concrete-ML, feel free to let us know and we'll link to it!

Additional resources

Looking for support? Ask our team!

• Support forum: (we answer in less than 24 hours).

• Live discussion on the FHE.org discord server: (inside the #concrete channel).

• Do you have a question about Zama? You can write us on or send us an email at: [email protected]

Depending on your OS, Concrete-ML may be installed with Docker or with pip:

Also, only some versions of python are supported: in the current release, these are 3.7 (linux only), 3.8 and 3.9.

Most of these limits are shared with the rest of the Concrete stack (namely Concrete-Numpy and Concrete-Compiler). Support for more platforms will be added in the future.

Using PyPi

Requirements

Installing Concrete-ML using PyPi requires a Linux-based OS or macOS running on an x86 CPU. For Apple Silicon, Docker is the only currently supported option (see ).

Installing on Windows can be done using Docker or WSL. On WSL, Concrete-ML will work as long as the package is not installed in the /mnt/c/ directory, which corresponds to the host OS filesystem.

Installation

To install Concrete-ML from PyPi, run the following:

This will automatically install all dependencies, notably Concrete-Numpy.

Using Docker

Concrete-ML can be installed using Docker by either pulling the latest image or a specific version:

The image can be used with Docker volumes, .

The image can then be used via the following command:

This will launch a Concrete-ML enabled Jupyter server in Docker that can be accessed directly from a browser.

Alternatively, a shell can be lauched in Docker, with or without volumes:

module concrete.ml.onnx

ONNX module.

Global Variables

• onnx_impl_utils

• ops_impl

• onnx_utils

• convert

• onnx_model_manipulations

Concrete-ML is built on top of Concrete-Numpy, which enables Numpy programs to be converted into FHE circuits.

Lifecycle of a Concrete-ML model

I. Model Development

1. Training. A model is trained using plaintext, non-encrypted, training data.

2. Quantization. The model is converted into an integer equivalent using quantization. Concrete-ML performs this step either during training (Quantization-Aware Training) or after training (Post-Training Quantization), depending on model type. Quantization converts inputs, model weights and all intermediate values of the inference computation to integers. More information is available .

3. Simulation using the Virtual Library. Testing FHE models on very large datasets can take a long time. Furthermore, not all models are compatible with FHE constraints out-of-the-box. Simulation using the Virtual Library allows you to execute a model that was quantized, to measure the accuracy it would have in FHE, but also to determine the modifications required to make it FHE compatible. Simulation is described in more details .

4. Compilation. Once the model is quantized, simulation can confirm it has good accuracy in FHE. The model then needs to be compiled using Concrete's FHE compiler to produce an equivalent FHE circuit. This circuit is represented as an MLIR program consisting of low level cryptographic operations. You can read more about FHE compilation , MLIR and about the low-level Concrete library .

5. Inference. The compiled model can then be executed on encrypted data, once the proper keys have been generated. The model can also be deployed to a server and used to run private inference on encrypted inputs.

You can see some examples of the model development workflow .

II. Model deployment

1. Client/Server deployment. In a client/server setting, the model can be exported in a way that:

   • allows the client to generate keys, encrypt and decrypt.

   • provides a compiled model that can run on the server to perform inference on encrypted data

2. Key generation. The data owner (client) needs to generate a pair of private keys (to encrypt/decrypt their data and results) and a public evaluation key (for the model's FHE evaluation on the server).

You can see an example of the model deployment workflow .

Cryptography concepts

Concrete-ML and Concrete-Numpy are tools that hide away the details of the underlying cryptography scheme, called TFHE. However, some cryptography concepts are still useful when using these two toolkits:

1. Encryption/Decryption. These operations transform plaintext, i.e. human-readable information, into ciphertext, i.e. data that contains a form of the original plaintext that is unreadable by a human or computer without the proper key to decrypt it. Encryption takes plaintext and an encryption key and produces ciphertext, while decryption is the inverse operation.

2. Encrypted inference. FHE allows a third party to execute (i.e. run inference or predict) a machine learning model on encrypted data (a ciphertext). The result of the inference is also encrypted and can only be read by the person who gets the decryption key.

3. Keys. A key is a series of bits used within an encryption algorithm for encrypting data so that the corresponding ciphertext appears random.

4. Key generation. Cryptographic keys need to be generated using random number generators. Their size may be large and key generation may take a long time. However, keys only need to be generated once for each model a client uses.

5. Guaranteed correctness of encrypted computations. To achieve security, TFHE, the underlying encryption scheme, adds random noise as ciphertexts. This can induce errors during processing of encrypted data, depending on noise parameters. By default, Concrete-ML uses parameters that ensure the correctness of the encrypted computation, so you do not need to take into account the noise parametrization. Therefore, results on encrypted data will be the same as the results of simulation on clear data.

While Concrete-ML users only need to understand the cryptography concepts above, for a deeper understanding of the cryptography behind the Concrete stack, please see the or .

Model accuracy considerations under FHE constraints

To respect FHE constraints, all numerical programs over encrypted data must have all inputs, constants and intermediate values represented with integers of a maximum of 8 bits.

Thus, Concrete-ML quantizes the input data and model outputs in the same way as weights and activations. The main levers to control accumulator bit-width are the numbers of bits used for the inputs, weights and activations of the model. These parameters are crucial to comply with the constraint on accumulator bit-widths. Please refer to for more details about how to develop models with quantization in Concrete-ML.

However, these methods may cause a reduction in the accuracy of the model since its representative power is diminished. Most importantly, carefully choosing a quantization approach can alleviate accuracy loss, all the while allowing compilation to FHE. Concrete-ML offers built-in models that already include quantization algorithms, and users only need to configure some of their parameters, such as the number of bits, discussed above. See for information about configuring these parameters for various models.

Additional specific methods can help to make models compatible with FHE constraints. For instance, dimensionality reduction can reduce the number of input features and, thus, the maximum accumulator bit-width reached within a circuit. Similarly, sparsity-inducing training methods, such as pruning, de-activate some features during inference, which also helps. For now, dimensionality reduction is considered as a pre-processing step, while pruning is used in the .

The configuration of model quantization parameters is illustrated in the advanced examples for and dimensionality reduction is shown in the .

Summary

The following table summarizes the examples in this section:

Examples

Concrete-ML provides functionality to deploy FHE machine learning models in a client/server setting. The deployment workflow and model serving pattern is as follows:

Deployment

The training of the model and its compilation to FHE are performed on a development machine. Three different files are created when saving the model:

• client.json contains the secure cryptographic parameters needed for the client to generate private and evaluation keys.

• server.json contains the compiled model. This file is sufficient to run the model on a server.

• serialized_processing.json contains the metadata about pre- and post-processing, such as quantization parameters to quantize the input and de-quantize the output.

The compiled model (server.zip) is deployed to a server and the cryptographic parameters (client.zip) along with the model meta data (serialized_processing.json) are shared with the clients.

Serving

The client obtains the cryptographic parameters (using client.zip) and generates a private encryption/decryption key as well as a set of public evaluation keys. The public evaluation keys are then sent to the server, while the secret key remains on the client.

The private data is then encrypted using serialized_processing.json by the client and sent to the server. Server-side, the FHE model inference is run on the encrypted inputs using the public evaluation keys.

The encrypted result is then returned by the server to the client, which decrypts it using its private key. Finally, the client performs any necessary post-processing of the decrypted result using serialized_processing.json.

Example notebook

For a complete example, see

module concrete.ml.common.debugging

Module for debugging.

Global Variables

• custom_assert

module concrete.ml.quantization

Modules for quantization.

Global Variables

• quantizers

• base_quantized_op

• quantized_module

• post_training

• quantized_ops

Using GitBook

Documentation with GitBook is done mainly by pushing content on GitHub. GitBook then pulls the docs from the repository and publishes. In most cases, GitBook is just a mirror of what is available in GitHub.

There are, however, some use-cases where documentation can be modified directly in GitBook (and then, push the modifications to GitHub), for example when the documentation is modified by a person outside of Zama. In this case, a GitHub branch is created, and a GitHub space is associated to it: modifications are done in this space and automatically pushed to the branch. Once the modifications are done, one can simply create a pull-request, to finally merge modifications on the main branch.

Using Sphinx

Documentation can alternatively be built using Sphinx:

The documentation contains both files written by hand by developers (the .md files) and files automatically created by parsing the source files.

Then to open it, go to docs/_build/html/index.html or use the follwing command:

To build and open the docs at the same time, use:

module concrete.ml.deployment

Module for deployment of the FHE model.

Global Variables

• fhe_client_server

Before you start this section, you must install Docker by following this official guide.

Building the image

Once you have access to this repository and the dev environment is installed on your host OS (via make setup_env once you followed the steps here), you should be able to launch the commands to build the dev Docker image with make docker_build.

Once you do that, you can get inside the Docker environment using the following command:

make docker_start

# or build and start at the same time
make docker_build_and_start

# or equivalently but shorter
make docker_bas

After you finish your work, you can leave Docker by using the exit command or by pressing CTRL + D.

module concrete.ml.version

File to manage the version of the package.

module concrete.ml.torch

Modules for torch to numpy conversion.

Global Variables

• numpy_module

module concrete.ml.common

Module for shared data structures and code.

Global Variables

• debugging

• check_inputs

• utils

Concrete-ML is a constant work-in-progress, and thus may contain bugs or suboptimal APIs.

Before opening an issue or asking for support, please read this documentation to understand common issues and limitations of Concrete-ML. You can also check the .

Furthermore, undefined behavior may occur if the input-set, which is internally used by the compilation core to set bit-widths of some intermediate data, is not sufficiently representative of the future user inputs. With all the inputs in the input-set, it appears that intermediate data can be represented as an n-bit integer. But, for a particular computation, this same intermediate data needs additional bits to be represented. The FHE execution for this computation will result in an incorrect output, as typically occurs in integer overflows in classical programs.

If you didn't find an answer, you can ask a question on the , or in the FHE.org .

Submitting an issue

When submitting an issue (), ideally include as much information as possible. In addition to the Python script, the following information is useful:

• the reproducibility rate you see on your side

• any insight you might have on the bug

• any workaround you have been able to find

If you would like to contribute to project and send pull requests, take a look at the guide.

Concrete-ML provides several of the most popular linear models for regression or classification that can be found in Scikit-learn:

Using these models in FHE is extremely similar to what can be done with scikit-learn's API, making it easy for data scientists who are used to this framework to get started with Concrete-ML.

Models are also compatible with some of scikit-learn's main workflows, such as Pipeline() or GridSearch().

Example

Here's an example of how to use this model in FHE on a simple data-set below. A more complete example can be found in the LogisticRegression notebook.

import numpy
from tqdm import tqdm
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

from concrete.ml.sklearn import LogisticRegression

# Create the data for classification
X, y = make_classification(
    n_features=2,
    n_redundant=0,
    n_informative=2,
    random_state=2,
    n_clusters_per_class=1,
    n_samples=100,
)

# Retrieve train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)

# Instantiate the model
model = LogisticRegression(n_bits=2)

# Fit the model
model.fit(X_train, y_train)

# Evaluate the model on the test set in clear
y_pred_clear = model.predict(X_test)

# Compile the model
model.compile(X_train)

# Perform the inference in FHE
# Note that here the encryption and decryption is done behind the scene.
# It is recommended to run this with a very small batch of
# examples first (e.g. N_TEST_FHE = 3)
N_TEST_FHE = 3
y_pred_fhe = numpy.array([
  model.predict([sample], execute_in_fhe=True)[0]
  for sample in tqdm(X_test[:N_TEST_FHE])
])

# Assert that FHE predictions are the same as the clear predictions
print(f"{(y_pred_fhe == y_pred_clear[:N_TEST_FHE]).sum()} "
      f"examples over {N_TEST_FHE} have a FHE inference equal to the clear inference.")

# Output:
#  3 examples over 3 have a FHE inference equal to the clear inference

We can then plot the decision boundary of the classifier and then compare those results with a scikit-learn model executed in clear. The complete code can be found in the LogisticRegression notebook.

We can clearly observe the impact of quantization over the decision boundaries in the FHE model, separating the initial lines into broken lines with steps. However, this does not change the overall score as both models output the same accuracy (90%).

In fact, the quantization process may sometimes create some artifacts that could lead to a decrease in performance. Still, the impact of those artifacts is often minor when considering linear models as FHE models reach similar scores as their equivalent clear ones.

Pruning is a method to reduce neural network complexity, usually applied in order to reduce the computation cost or memory size. Pruning is used in Concrete-ML to control the size of accumulators in neural networks, thus making them FHE-compatible. See here for an explanation of accumulator bit-width constraints.

Overview of pruning in Concrete-ML

Pruning is used in Concrete-ML for two types of neural networks:

1. Built-in neural networks include a pruning mechanism that can be parameterized by the user. The pruning type is based on L1-norm. To comply with FHE constraints, Concrete-ML uses unstructured pruning, as the aim is not to eliminate neurons or convolutional filters completely, but to decrease their accumulator bit-width.

2. Custom neural networks, to work well under FHE constraints, should include pruning. When implemented with PyTorch, you can use the framework's pruning mechanism (e.g.L1-Unstructured) to good effect.

Basics of pruning

In neural networks, a neuron computes a linear combination of inputs and learned weights, then applies an activation function.

The neuron computes:

$y_k = \phi\left(\sum_i w_ix_i\right)$

When building a full neural network, each layer will contain multiple neurons, which are connected to the neuron outputs of a previous layer or to the inputs.

For every neuron shown in each layer of the figure above, the linear combinations of inputs and learned weights are computed. Depending on the values of the inputs and weights, the sum $v_k = \sum_i w_ix_i$ - which for Concrete-ML neural networks is computed with integers - can take a range of different values.

To respect the bit-width constraint of the FHE Table Lookup, the values of the accumulator $v_k$ must remain small to be representable with only 8 bits. In other words, the values must be between 0 and 255.

Pruning a neural network entails fixing some of the weights $w_k$ to be zero during training. This is advantageous to meet FHE constraints, as irrespective of the distribution of $x_i$, multiplying these input values by 0 does not increase the accumulator value.

Fixing some of the weights to 0 makes the network graph look more similar to the following:

While pruning weights can reduce the prediction performance of the neural network, studies show that a high level of pruning (above 50%) can often be applied. See here how Concrete-ML uses pruning in Fully Connected Neural Networks.

Pruning in practice

In the formula above, in the worst-case, the maximum number of the input and weights that can make the result exceed $n$ bits is given by:

$\Omega = \mathsf{floor} \left( \frac{2^{n_{\mathsf{max}}} - 1}{(2^{n_{\mathsf{weights}}} - 1)(2^{n_{\mathsf{inputs}}} - 1)} \right)$

Here, $n_{\mathsf{max}} = 8$ is the maximum precision allowed.

For example, if $n_{\mathsf{weights}} = 2$ and $n_{\mathsf{inputs}} = 2$ with $n_{\mathsf{max}} = 8$, the worst case is where all inputs and weights are equal to their maximal value $2^2-1=3$. In this case, there can be at most $\Omega = 28$ elements in the multi-sums.

In practice, the distribution of the weights of a neural network is Gaussian, with many weights either 0 or having a small value. This enables exceeding the worst-case number of active neurons without having to risk overflowing the bit-width. In built-in neural networks, the parameter n_hidden_neurons_multiplier is multiplied with $\Omega$ to determine the total number of non-zero weights that should be kept in a neuron.

module concrete.ml.sklearn.tree_to_numpy

Implements the conversion of a tree model to a numpy function.

Global Variables

• MAXIMUM_TLU_BIT_WIDTH

• OPSET_VERSION_FOR_ONNX_EXPORT

• EXPECTED_NUMBER_OF_OUTPUTS_PER_TASK

function tree_to_numpy

tree_to_numpy(
    model: ModelProto,
    x: ndarray,
    framework: str,
    task: Task,
    output_n_bits: Optional[int] = 16
) → Tuple[Callable, List[UniformQuantizer], ModelProto]

Convert the tree inference to a numpy functions using Hummingbird.

Args:

• model (onnx.ModelProto): The model to convert.

• x (numpy.ndarray): The input data.

• framework (str): The framework from which the onnx_model is generated.

• (options: 'xgboost', 'sklearn')

• task (Task): The task the model is solving

• output_n_bits (int): The number of bits of the output.

Returns:

• Tuple[Callable, List[QuantizedArray], onnx.ModelProto]: A tuple with a function that takes a numpy array and returns a numpy array, QuantizedArray object to quantize and dequantize the output of the tree, and the ONNX model.

class Task

Task enumerate.

module concrete.ml.torch.numpy_module

A torch to numpy module.

Global Variables

• OPSET_VERSION_FOR_ONNX_EXPORT

class NumpyModule

General interface to transform a torch.nn.Module to numpy module.

Args:

• torch_model (Union[nn.Module, onnx.ModelProto]): A fully trained, torch model along with its parameters or the onnx graph of the model.

• dummy_input (Union[torch.Tensor, Tuple[torch.Tensor, ...]]): Sample tensors for all the module inputs, used in the ONNX export to get a simple to manipulate nn representation.

• debug_onnx_output_file_path: (Optional[Union[Path, str]], optional): An optional path to indicate where to save the ONNX file exported by torch for debug. Defaults to None.

method __init__

__init__(
    model: Union[Module, ModelProto],
    dummy_input: Optional[Tensor, Tuple[Tensor, ]] = None,
    debug_onnx_output_file_path: Optional[Path, str] = None
)

property onnx_model

Get the ONNX model.

.. # noqa: DAR201

Returns:

• _onnx_model (onnx.ModelProto): the ONNX model

method forward

forward(*args: ndarray) → Union[ndarray, Tuple[ndarray, ]]

Apply a forward pass on args with the equivalent numpy function only.

Args:

• *args: the inputs of the forward function

Returns:

• Union[numpy.ndarray, Tuple[numpy.ndarray, ...]]: result of the forward on the given inputs

module concrete.ml.sklearn

Import sklearn models.

Global Variables

• protocols

• tree_to_numpy

• base

• torch_module

• glm

• linear_model

• qnn

• rf

• svm

• tree

• xgb

module concrete.ml.common.check_inputs

Check and conversion tools.

Utils that are used to check (including convert) some data types which are compatible with scikit-learn to numpy types.

function check_array_and_assert

check_array_and_assert(X)

sklearn.utils.check_array with an assert.

Equivalent of sklearn.utils.check_array, with a final assert that the type is one which is supported by Concrete-ML.

Args:

• X (object): Input object to check / convert

Returns: The converted and validated array

function check_X_y_and_assert

check_X_y_and_assert(X, y, *args, **kwargs)

sklearn.utils.check_X_y with an assert.

Equivalent of sklearn.utils.check_X_y, with a final assert that the type is one which is supported by Concrete-ML.

Args:

• X (ndarray, list, sparse matrix): Input data

• y (ndarray, list, sparse matrix): Labels

• *args: The arguments to pass to check_X_y

• **kwargs: The keyword arguments to pass to check_X_y

Returns: The converted and validated arrays

Concrete-ML offers some features for advanced users that wish to adjust the cryptographic parameters that are generated by the Concrete stack for a certain machine learning model.

Approximate computations using the p_error parameter

Concrete-ML makes use of table lookup (TLU) to represent any non-linear operation (e.g. sigmoid). This TLU is implemented through the Programmable Bootstrapping (PBS) operation which will apply a non-linear operation in the cryptographic realm.

In Concrete-ML, the result of the TLU operation is obtained with a specific error probability:

DEFAULT_P_ERROR_PBS = 6.3342483999973e-05

A single PBS operation has 1 - DEFAULT_P_ERROR_PBS = 99.9936657516% chances of being correct. This number plays a role in the cryptographic parameters. As such, the lower the p_error, the more constraining the parameters will become. This has an impact on both key generation and, more importantly, on FHE execution time.

This number is set by default to be relatively low such that any user can build deep circuits without being impacted by this noise as described in the concepts section. However, there might be use cases and specific circuits where the Gaussian noise can increase without being too dramatic for the circuit accuracy. In that case, increasing the p_error can be relevant as it will reduce the execution time in FHE.

Here is a visualization of the effect of the p_error over a simple linear regression with a p_error = 0.1 vs the default p_error value:

The execution for the two models are 336 ms per example for the standard p_error and 253 ms per example for a p_error = 0.1 (on a 8 cores Intel CPU machine). Obviously, this speedup is very dependent on model complexity. To obtain a speedup while maintaining good accuracy, it is possible to search for a good value of p_error. Currently no heuristic has been proposed to find a good value a-priori.

Users have the possibility to change this p_error as they choose fit, by passing an argument to the compile function of any of the models. Here is an example:

from concrete.ml.sklearn import XGBoostClassifier
clf = XGBoostClassifier()
clf.fit(X_train, y_train)

# Here comes the p_error parameter
clf.compile(X_train, p_error = 0.1)

Hummingbird

is a third-party, open-source library that converts machine learning models into tensor computations, and it can export these models to ONNX. The list of supported models can be found in .

Concrete-ML allows the conversion of an ONNX inference to NumPy inference (note that NumPy is always the entry point to run models in FHE with Concrete ML).

Hummingbird exposes a convert function that can be imported as follows from the hummingbird.ml package:

This function can be used to convert a machine learning model to an ONNX as follows:

In theory, the resulting onnx_model could be used directly within Concrete-ML's get_equivalent_numpy_forward method (as long as all operators present in the ONNX model are implemented in NumPy) and get the NumPy inference.

In practice, there are some steps needed to clean the ONNX output and make the graph compatible with Concrete-ML, such as applying quantization where needed or deleting/replacing non-FHE friendly ONNX operators (such as Softmax and ArgMax).

Skorch

Concrete-ML uses to implement multi-layer, fully-connected PyTorch neural networks in a way that is compatible with the Scikit-learn API.

This wrapper implements Torch training boilerplate code, alleviating the work that needs to be done by the user. It is possible to add hooks during the training phase, for example once an epoch is finished.

Skorch allows the user to easily create a classifier or regressor around a neural network (NN), implemented in Torch as a nn.Module, which is used by Concrete-ML to provide a fully-connected multi-layer NN with a configurable number of layers and optional pruning (see and the for more information).

Under the hood, Concrete-ML uses a Skorch wrapper around a single PyTorch module, SparseQuantNeuralNetImpl. More information can be found .

Brevitas

is a quantization aware learning toolkit built on top of PyTorch. It provides quantization layers that are one-to-one equivalents to PyTorch layers, but also contain operations that perform the quantization during training.

While Brevitas provides many types of quantization, for Concrete-ML, a custom "mixed integer" quantization applies. This "mixed integer" quantization is much simpler than the "integer only" mode of Brevitas. The "mixed integer" network design is defined as:

• all weights and activations of convolutional, linear and pooling layers must be quantized (e.g. using Brevitas layers, QuantConv2D, QuantAvgPool2D, QuantLinear)

• PyTorch floating point versions of univariate functions can be used. E.g. torch.relu, nn.BatchNormalization2D, torch.max (encrypted vs. constant), torch.add, torch.exp. See the for a full list.

The "mixed integer" mode used in Concrete-ML neural networks is based on the that makes both weights and activations representable as integers during training. However, through the use of lookup tables in Concrete-ML, floating point univariate PyTorch functions are supported.

For "mixed integer" quantization to work, the first layer of a Brevitas nn.Module must be a QuantIdentity layer. However, you can then use functions such as torch.sigmoid on the result of such a quantizing operation.

For examples of such a "mixed integer" network design, please see the Quantization Aware Training examples:

or go to the .

You can also refer to the class which is the basis of the built-in NeuralNetworkClassifier.

Concrete-ML provides partial support for Pandas, with most available models (linear and tree-based models) usable on Pandas dataframes the same way they would be used with NumPy arrays.

The table below summarizes the current compatibility:

Example

The following example considers a LogisticRegression model on a simple classification problem. A more advanced example can be found in the , which considers a XGBClassifier.

module concrete.ml.common.debugging.custom_assert

Provide some variants of assert.

function assert_true

Provide a custom assert to check that the condition is True.

Args:

• condition (bool): the condition. If False, raise AssertionError

• on_error_msg (str): optional message for precising the error, in case of error

• error_type (Type[Exception]): the type of error to raise, if condition is not fulfilled. Default to AssertionError

function assert_false

Provide a custom assert to check that the condition is False.

Args:

• condition (bool): the condition. If True, raise AssertionError

• on_error_msg (str): optional message for precising the error, in case of error

• error_type (Type[Exception]): the type of error to raise, if condition is not fulfilled. Default to AssertionError

function assert_not_reached

Provide a custom assert to check that a piece of code is never reached.

Args:

• on_error_msg (str): message for precising the error

• error_type (Type[Exception]): the type of error to raise, if condition is not fulfilled. Default to AssertionError

There are three ways to contribute to Concrete-ML:

• You can open issues to report bugs and typos and to suggest ideas.

• You can ask to become an official contributor by emailing [email protected]. Only approved contributors can send pull requests (PR), so please make sure to get in touch before you do.

• You can also provide new tutorials or use-cases, showing what can be done with the library. The more examples we have, the better and clearer it is for the other users.

1. Creating a new branch

Concrete-ML uses a consistent branch naming scheme, and you are expected to follow it as well. Here is the format, along with some examples:

git checkout -b {feat|fix|refactor|test|benchmark|doc|style|chore}/short-description_$issue_id

e.g.

git checkout -b feat/explicit-tlu_11
git checkout -b fix/tracing_indexing_42

2. Before committing

2.1 Conformance

Each commit to Concrete-ML should conform to the standards of the project. You can let the development tools fix some issues automatically with the following command:

make conformance

Conformance can be checked using the following command:

make pcc

2.2 Testing

Your code must be well documented, containing tests and not breaking other tests:

make pytest

You need to make sure you get 100% code coverage. The make pytest command checks that by default and will fail with a coverage report at the end should some lines of your code not be executed during testing.

If your coverage is below 100%, you should write more tests and then create the pull request. If you ignore this warning and create the PR, GitHub actions will fail and your PR will not be merged.

There may be cases where covering your code is not possible (an exception that cannot be triggered in normal execution circumstances). In those cases, you may be allowed to disable coverage for some specific lines. This should be the exception rather than the rule, and reviewers will ask why some lines are not covered. If it appears they can be covered, then the PR won't be accepted in that state.

3. Committing

Concrete-ML uses a consistent commit naming scheme, and you are expected to follow it as well (the CI will make sure you do). The accepted format can be printed to your terminal by running:

make show_scope

e.g.

git commit -m "feat: implement bounds checking"
git commit -m "feat(debugging): add an helper function to draw intermediate representation"
git commit -m "fix(tracing): fix a bug that crashed PyTorch tracer"

To learn more about conventional commits, check this page. Just a reminder that commit messages are checked in the comformance step and are rejected if they don't follow the rules.

4. Rebasing

You should rebase on top of the main branch before you create your pull request. Merge commits are not allowed, so rebasing on main before pushing gives you the best chance of avoiding having to rewrite parts of your PR later if conflicts arise with other PRs being merged. After you commit your changes to your new branch, you can use the following commands to rebase:

# fetch the list of active remote branches
git fetch --all --prune

# checkout to main
git checkout main

# pull the latest changes to main (--ff-only is there to prevent accidental commits to main)
git pull --ff-only

# checkout back to your branch
git checkout $YOUR_BRANCH

# rebase on top of main branch
git rebase main

# If there are conflicts during the rebase, resolve them
# and continue the rebase with the following command
git rebase --continue

# push the latest version of the local branch to remote
git push --force

You can learn more about rebasing here.

5. Releases

Before any final release, Concrete-ML contributors go through a release candidate (RC) cycle. The idea is that once the codebase and documentations look ready for a release, you create an RC release by opening an issue with the release template here, starting with version vX.Y.Zrc1 and then with versions vX.Y.Zrc2, vX.Y.Zrc3...

Once the last RC is deemed ready, open an issue with the release template using the last RC version from which you remove the rc? part (i.e. v12.67.19 if your last RC version was v12.67.19-rc4) on github.

Concrete-ML provides several of the most popular classification and regression tree models that can be found in Scikit-learn:

In addition to support for scikit-learn, Concrete-ML also supports XGBoost 's XGBClassifier:

Example

Here's an example of how to use this model in FHE on a popular data-set using some of scikit-learn's pre-processing tools. A more complete example can be found in the XGBClassifier notebook.

from sklearn.datasets import load_breast_cancer
from sklearn.decomposition import PCA
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

from concrete.ml.sklearn.xgb import XGBClassifier


# Get data-set and split into train and test
X, y = load_breast_cancer(return_X_y=True)

# Split the train and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

# Define our model
model = XGBClassifier(n_jobs=1, n_bits=3)

# Define the pipeline
# We will normalize the data and apply a PCA before fitting the model
pipeline = Pipeline(
    [("standard_scaler", StandardScaler()), ("pca", PCA(random_state=0)), ("model", model)]
)

# Define the parameters to tune
param_grid = {
    "pca__n_components": [2, 5, 10, 15],
    "model__max_depth": [2, 3, 5],
    "model__n_estimators": [5, 10, 20],
}

# Instantiate the grid search with 5-fold cross validation on all available cores
grid = GridSearchCV(pipeline, param_grid, cv=5, n_jobs=-1, scoring="accuracy")

# Launch the grid search
grid.fit(X_train, y_train)

# Print the best parameters found
print(f"Best parameters found: {grid.best_params_}")

# Output:
#  Best parameters found: {'model__max_depth': 5, 'model__n_estimators': 10, 'pca__n_components': 5}

# Currently we only focus on model inference in FHE
# The data transformation will be done in clear (client machine)
# while the model inference will be done in FHE on a server.
# The pipeline can be split into 2 parts:
#   1. data transformation
#   2. estimator
best_pipeline = grid.best_estimator_
data_transformation_pipeline = best_pipeline[:-1]
model = best_pipeline[-1]

# Transform test set
X_train_transformed = data_transformation_pipeline.transform(X_train)
X_test_transformed = data_transformation_pipeline.transform(X_test)

# Evaluate the model on the test set in clear
y_pred_clear = model.predict(X_test_transformed)
print(f"Test accuracy in clear: {(y_pred_clear == y_test).mean():0.2f}")

# Output:
#  Test accuracy: 0.98

# Compile the model to FHE
model.compile(X_train_transformed)

# Perform the inference in FHE
# Warning: this will take a while. It is recommended to run this with a very small batch of
# example first (e.g. N_TEST_FHE = 1)
# Note that here the encryption and decryption is done behind the scene.
N_TEST_FHE = 1
y_pred_fhe = model.predict(X_test_transformed[:N_TEST_FHE], execute_in_fhe=True)

# Assert that FHE predictions are the same as the clear predictions
print(f"{(y_pred_fhe == y_pred_clear[:N_TEST_FHE]).sum()} "
      f"examples over {N_TEST_FHE} have a FHE inference equal to the clear inference.")

# Output:
#  1 examples over 1 have a FHE inference equal to the clear inference

Using the above example, we can then plot how the model classifies the inputs and then compare those results with the XGBoost model executed in clear. A 6-bits model is also given in order to better understand the impact of quantization on classification. Similar plots can be found in the Classifier Comparison notebook.

This graph shows the impact of quantization over the decision boundaries in the Concrete-ML FHE decision tree models. In the 3-bits model, only a rough, highly-discrete decision function is observed. This results in a small decrease of accuracy of about 7% compared to the initial XGBoost classifier. Besides, using 6-bits of quantization makes the model reach 93% accuracy, drastically reducing this difference to only 1.7 percentage points.

In fact, the quantization process may sometimes create some artifacts that could lead to a decrease in performance. Still, as the quantization is done individually on each input feature, the artifacts are minor when considering small tree-based models with 5-6 bits quantization. Thus, FHE tree-based models reach similar scores as their equivalent floating point ones.

The following graph shows that using 5-6 bits of quantization is usually sufficient to reach the performance of a non-quantized XGBoost model on floating point data. The metrics plotted are accuracy and F1-score on the spambase data-set.

module concrete.ml.sklearn.rf

Implements RandomForest models.

class RandomForestClassifier

Implements the RandomForest classifier.

method __init__

__init__(
    n_bits: int = 6,
    n_estimators=20,
    criterion='gini',
    max_depth=4,
    min_samples_split=2,
    min_samples_leaf=1,
    min_weight_fraction_leaf=0.0,
    max_features='sqrt',
    max_leaf_nodes=None,
    min_impurity_decrease=0.0,
    bootstrap=True,
    oob_score=False,
    n_jobs=None,
    random_state=None,
    verbose=0,
    warm_start=False,
    class_weight=None,
    ccp_alpha=0.0,
    max_samples=None
)

Initialize the RandomForestClassifier.

noqa: DAR101

property onnx_model

Get the ONNX model.

.. # noqa: DAR201

Returns:

• onnx.ModelProto: the ONNX model

class RandomForestRegressor

Implements the RandomForest regressor.

method __init__

__init__(
    n_bits: int = 6,
    n_estimators=20,
    criterion='squared_error',
    max_depth=4,
    min_samples_split=2,
    min_samples_leaf=1,
    min_weight_fraction_leaf=0.0,
    max_features='sqrt',
    max_leaf_nodes=None,
    min_impurity_decrease=0.0,
    bootstrap=True,
    oob_score=False,
    n_jobs=None,
    random_state=None,
    verbose=0,
    warm_start=False,
    ccp_alpha=0.0,
    max_samples=None
)

Initialize the RandomForestRegressor.

noqa: DAR101

property onnx_model

Get the ONNX model.

.. # noqa: DAR201

Returns:

• onnx.ModelProto: the ONNX model

module concrete.ml.onnx.onnx_utils

Utils to interpret an ONNX model with numpy.

Global Variables

• ATTR_TYPES

• ATTR_GETTERS

• ONNX_OPS_TO_NUMPY_IMPL

• ONNX_COMPARISON_OPS_TO_NUMPY_IMPL_FLOAT

• ONNX_COMPARISON_OPS_TO_NUMPY_IMPL_BOOL

• ONNX_OPS_TO_NUMPY_IMPL_BOOL

• IMPLEMENTED_ONNX_OPS

function get_attribute

get_attribute(attribute: AttributeProto) → Any

Get the attribute from an ONNX AttributeProto.

Args:

• attribute (onnx.AttributeProto): The attribute to retrieve the value from.

Returns:

• Any: The stored attribute value.

function get_op_name

get_op_name(node)

Construct the qualified name of the ONNX operator.

Args:

• node (Any): ONNX graph node

Returns:

• result (str): qualified name

function execute_onnx_with_numpy

execute_onnx_with_numpy(graph: GraphProto, *inputs: ndarray) → Tuple[ndarray, ]

Execute the provided ONNX graph on the given inputs.

Args:

• graph (onnx.GraphProto): The ONNX graph to execute.

• *inputs: The inputs of the graph.

Returns:

• Tuple[numpy.ndarray]: The result of the graph's execution.

module concrete.ml.common.utils

Utils that can be re-used by other pieces of code in the module.

Global Variables

• DEFAULT_P_ERROR_PBS

function replace_invalid_arg_name_chars

replace_invalid_arg_name_chars(arg_name: str) → str

Sanitize arg_name, replacing invalid chars by _.

This does not check that the starting character of arg_name is valid.

Args:

• arg_name (str): the arg name to sanitize.

Returns:

• str: the sanitized arg name, with only chars in _VALID_ARG_CHARS.

function generate_proxy_function

generate_proxy_function(
    function_to_proxy: Callable,
    desired_functions_arg_names: Iterable[str]
) → Tuple[Callable, Dict[str, str]]

Generate a proxy function for a function accepting only *args type arguments.

This returns a runtime compiled function with the sanitized argument names passed in desired_functions_arg_names as the arguments to the function.

Args:

• function_to_proxy (Callable): the function defined like def f(*args) for which to return a function like f_proxy(arg_1, arg_2) for any number of arguments.

• desired_functions_arg_names (Iterable[str]): the argument names to use, these names are sanitized and the mapping between the original argument name to the sanitized one is returned in a dictionary. Only the sanitized names will work for a call to the proxy function.

Returns:

• Tuple[Callable, Dict[str, str]]: the proxy function and the mapping of the original arg name to the new and sanitized arg names.

function get_onnx_opset_version

get_onnx_opset_version(onnx_model: ModelProto) → int

Return the ONNX opset_version.

Args:

• onnx_model (onnx.ModelProto): the model.

Returns:

• int: the version of the model

Concrete-ML provides simple neural networks models with a Scikit-learn interface through the NeuralNetClassifier and NeuralNetRegressor classes.

The neural network models are built with , which provides a scikit-learn like interface to Torch models (more ).

These models use a stack of linear layers and the activation function and the number of neurons in each layer is configurable. This approach is similar to what is available in Scikit-learn using the MLPClassifier/MLPRegressor classes. The built-in, fully connected neural network (FCNN) models train easily with a single call to .fit(), which will automatically quantize the weights and activations. These models use Quantization Aware Training, allowing good performance for low precision (down to 2-3 bit) weights and activations.

While NeuralNetClassifier and NeuralNetClassifier provide scikit-learn like models, their architecture is somewhat restricted in order to make training easy and robust. If you need more advanced models, you can convert custom neural networks as described in the .

Example usage

To create an instance of a Fully Connected Neural Network you need to instantiate one of the NeuralNetClassifier and NeuralNetRegressor classes and configure a number of parameters that are passed to their constructor. Note that some parameters need to be prefixed by module__, while others don't. Basically, the parameters that are related to the model, i.e. the underlying nn.Module, must have the prefix. The parameters that are related to training options do not require the prefix.

The shows the behavior of built-in neural networks on several synthetic datasets.

The figure above shows, on the right, the Concrete-ML neural network, trained with Quantization Aware Training, in a FHE-compatible configuration. The figure compares this network to the floating point equivalent, trained with scikit-learn.

Architecture parameters

• module__n_layers: number of layers in the FCNN, must be at least 1. Note that this is the total number of layers. For a single hidden layer NN model, set module__n_layers=2

• module__n_outputs: number of outputs (classes or targets)

• module__input_dim: dimensionality of the input

• module__activation_function: can be one of the Torch activations (e.g. nn.ReLU, see the full list )

Quantization parameters

• n_w_bits (default 3): number of bits for weights

• n_a_bits (default 3): number of bits for activations and inputs

• n_accum_bits (default 8): maximum accumulator bit-width that is desired. The implementation will attempt to keep accumulators under this bit-width through , i.e. setting some weights to zero

Training parameters (from Skorch)

• max_epochs: The number of epochs to train the network (default 10)

• verbose: Whether to log loss/metrics during training (default: False)

• lr: Learning rate (default 0.001)

Other parameters from skorch are in the

Advanced parameters

• module__n_hidden_neurons_multiplier: The number of hidden neurons will be automatically set proportional to the dimensionality of the input (i.e. the value for module__input_dim). This parameter controls the proportionality factor and is set to 4 by default. This value gives good accuracy while avoiding accumulator overflow. See the and sections for more info.

Network input/output

When you have training data in the form of a NumPy array, and targets in a NumPy 1d array, you can set:

Class weights

You can give weights to each class to use in training. Note that this must be supported by the underlying PyTorch loss function.

Overflow errors

The n_hidden_neurons_multiplier parameter influences training accuracy as it controls the number of non-zero neurons that are allowed in each layer. Increasing n_hidden_neurons_multiplier improves accuracy, but should take into account precision limitations to avoid overflow in the accumulator. The default value is a good compromise that avoids overflow, in most cases, but you may want to change the value of this parameter to reduce the breadth of the network if you have overflow errors. A value of 1 should be completely safe with respect to overflow.

The following table summarizes the various examples in this section, along with their accuracies.

A * means that FHE accuracy was calculated on a subset of the validation set.

Concrete-ML models

Comparison of classifiers

Quantizing data

Concrete-ML has support for quantized ML models and also provides quantization tools for Quantization Aware Training and Post-Training Quantization. The core of this functionality is the conversion of floating point values to integers and back. This is done using QuantizedArray in concrete.ml.quantization.

The class takes several arguments that determine how float values are quantized:

• n_bits that defines the precision of the quantization

• values are floating point values that will be converted to integers

• is_signed determines if the quantized integer values should allow negative values

• is_symmetric determines if the range of floating point values to be quantized should be taken as symmetric around zero

See also the reference for more information:

It is also possible to use symmetric quantization, where the integer values are centered around 0:

In the following example, showing the de-quantization of model outputs, the QuantizedArray class is used in a different way. Here it uses pre-quantized integer values and has the scale and zero-point set explicitly. Once the QuantizedArray is constructed, calling dequant() will compute the floating point values corresponding to the integer values qvalues, which are the output of the forward_fhe.encrypt_run_decrypt(..) call.

Quantized modules

Machine learning models are implemented with a diverse set of operations, such as convolution, linear transformations, activation functions and element-wise operations. When working with quantized values, these operations cannot be carried out in an equivalent way as for floating point values. With quantization, it is necessary to re-scale the input and output values of each operation to fit in the quantization domain.

In Concrete-ML, the quantized equivalent of a scikit-learn model or a PyTorch nn.Module is the QuantizedModule. Note that only inference is implemented in the QuantizedModule, and it is built through a conversion of the inference function of the corresponding scikit-learn or PyTorch module.

Built-in neural networks expose the quantized_module member, while a QuantizedModule is also the result of the compilation of custom models through compile_torch_model and compile_brevitas_qat_model.

The quantized versions of floating point model operations are stored in the QuantizedModule. The ONNX_OPS_TO_QUANTIZED_IMPL dictionary maps ONNX floating point operators (e.g. Gemm) to their quantized equivalent (e.g. QuantizedGemm). For more information on implementing these operations, please see the .

The computation graph is taken from the corresponding floating point ONNX graph exported from scikit-learn , or from the ONNX graph exported by PyTorch. Calibration is used to obtain quantized parameters for the operations in the QuantizedModule. Parameters are also determined for the quantization of inputs during model deployment.

That QuantizedModule generates the Concrete-Numpy function that is compiled to FHE. The compilation will succeed if the intermediate values conform to the 8-bits precision limit of the Concrete stack. See for details.

Resources

• Lei Mao's blog on quantization:

• Google paper on neural network quantization and integer-only inference:

Compilation of a model produces machine code that executes the model on encrypted data. In some cases, notably in the client/server setting, the compilation can be done by the server when loading the model for serving.

As FHE execution is much slower than execution on non-encrypted data, Concrete-ML has a simulation mode, using an execution mode named the Virtual Library. Since, by default, the cryptographic parameters are chosen such that the results obtained in FHE are the same as those on clear data, the Virtual Library allows you to benchmark models quickly during development.

Compilation

Concrete-ML implements machine model inference using Concrete-Numpy as a backend. In order to execute in FHE, a numerical program written in Concrete-Numpy needs to be compiled. This functionality is , and Concrete-ML hides away most of the complexity of this step. The entire compilation process is done by Concrete-Numpy.

From the perspective of the Concrete-ML user, the compilation process performed by Concrete-Numpy can be broken up into 3 steps:

1. Numpy program tracing and creation of a Concrete-Numpy op-graph

2. checking that the op-graph is FHE compatible

3. producing machine code for the op-graph. This step automatically determines cryptographic parameters

Additionally, the packages the result of the last step in a way that allows the deployment of the encrypted circuit to a server and key generation, encryption and decryption on the client side.

Simulation with the Virtual Library

The first step in the list above takes a Python function implemented using the Concrete-Numpy and transforms it into an executable operation graph.

The result of this single step of the compilation pipeline allows the:

• execution of the op-graph, which includes TLUs, on clear non-encrypted data. This is, of course, not secure, but it is much faster than executing in FHE. This mode is useful for debugging, i.e. to find the appropriate hyper-parameters. This mode is called the Virtual Library.

• verification of the maximum bit-width of the op-graph, to determine FHE compatibility, without actually compiling the circuit to machine code.

Enabling Virtual Library execution requires the definition of a compilation Configuration. As simulation does not execute in FHE, this can be considered unsafe:

Next, the following code uses the simulation mode for built-in models:

And finally, for custom models, it is possible to enable simulation using the following syntax:

Obtaining the simulated predictions of the models using the Virtual Library has the same syntax as execution in FHE:

Moreover, the maximum accumulator bit-width is determined as follows:

A simple Concrete-Numpy example

While Concrete-ML hides away all the Concrete-Numpy code that performs model inference, it can be useful to understand how Concrete-Numpy code works. Here is an toy example for a simple linear regression model on integers. Note that this is just an example to illustrate compilation concepts. Generally, it is recommended to use the , which provide linear regression out of the box.

Internally, Concrete-ML uses operators as intermediate representation (or IR) for manipulating machine learning models produced through export for , and .

As ONNX is becoming the standard exchange format for neural networks, this allows Concrete-ML to be flexible while also making model representation manipulation quite easy. In addition, it allows for straight-forward mapping to NumPy operators, supported by Concrete-Numpy to use Concrete stack's FHE conversion capabilities.

Torch to NumPy conversion using ONNX

The diagram below gives an overview of the steps involved in the conversion of an ONNX graph to a FHE compatible format, i.e. a format that can be compiled to FHE through Concrete-Numpy.

All Concrete-ML built-in models follow the same pattern for FHE conversion:

1. The models are trained with sklearn or PyTorch

2. All models have a PyTorch implementation for inference. This implementation is provided either by a third-party tool such as or implemented directly in Concrete-ML.

3. The PyTorch model is exported to ONNX. For more information on the use of ONNX in Concrete-ML, see .

4. The Concrete-ML ONNX parser checks that all the operations in the ONNX graph are supported and assigns reference NumPy operations to them. This step produces a NumpyModule.

5. Quantization is performed on the , producing a . Two steps are performed: calibration and assignment of equivalent objects to each ONNX operation. The QuantizedModule class is the quantized counterpart of the NumpyModule.

6. Once the QuantizedModule is built, Concrete-Numpy is used to trace the ._forward() function of the QuantizedModule.

Moreover, by passing a user provided nn.Module to step 2 of the above process, Concrete-ML supports custom user models. See the associated for instructions about working with such models.

Once an ONNX model is imported, it is converted to a NumpyModule, then to a QuantizedModule and, finally, to a FHE circuit. However, as the diagram shows, it is perfectly possible to stop at the NumpyModule level if you just want to run the PyTorch model as NumPy code without doing quantization.

Inspecting the ONNX models

In order to better understand how Concrete-ML works under the hood, it is possible to access each model in their ONNX format and then either print it or visualize it by importing the associated file in . For example, with LogisticRegression:

module concrete.ml.sklearn.tree

Implement the sklearn tree models.

class DecisionTreeClassifier

Implements the sklearn DecisionTreeClassifier.

method __init__

Initialize the DecisionTreeClassifier.

noqa: DAR101

property onnx_model

Get the ONNX model.

.. # noqa: DAR201

Returns:

• onnx.ModelProto: the ONNX model

class DecisionTreeRegressor

Implements the sklearn DecisionTreeClassifier.

method __init__

Initialize the DecisionTreeRegressor.

noqa: DAR101

property onnx_model

Get the ONNX model.

.. # noqa: DAR201

Returns:

• onnx.ModelProto: the ONNX model

module concrete.ml.onnx.onnx_model_manipulations

Some code to manipulate models.

function simplify_onnx_model

simplify_onnx_model(onnx_model: ModelProto)

Simplify an ONNX model, removes unused Constant nodes and Identity nodes.

Args:

• onnx_model (onnx.ModelProto): the model to simplify.

function remove_unused_constant_nodes

remove_unused_constant_nodes(onnx_model: ModelProto)

Remove unused Constant nodes in the provided onnx model.

Args:

• onnx_model (onnx.ModelProto): the model for which we want to remove unused Constant nodes.

function remove_identity_nodes

remove_identity_nodes(onnx_model: ModelProto)

Remove identity nodes from a model.

Args:

• onnx_model (onnx.ModelProto): the model for which we want to remove Identity nodes.

function keep_following_outputs_discard_others

keep_following_outputs_discard_others(
    onnx_model: ModelProto,
    outputs_to_keep: Iterable[str]
)

Keep the outputs given in outputs_to_keep and remove the others from the model.

Args:

• onnx_model (onnx.ModelProto): the ONNX model to modify.

• outputs_to_keep (Iterable[str]): the outputs to keep by name.

function remove_node_types

remove_node_types(onnx_model: ModelProto, op_types_to_remove: List[str])

Remove unnecessary nodes from the ONNX graph.

Args:

• onnx_model (onnx.ModelProto): The ONNX model to modify.

• op_types_to_remove (List[str]): The node types to remove from the graph.

Raises:

• ValueError: Wrong replacement by an Identity node.

function clean_graph_after_node_name

clean_graph_after_node_name(
    onnx_model: ModelProto,
    node_name: str,
    fail_if_not_found: bool = True
)

Clean the graph of the onnx model by removing nodes after the given node name.

Args:

• onnx_model (onnx.ModelProto): The onnx model.

• node_name (str): The node's name whose following nodes will be removed.

• fail_if_not_found (bool): If true, abort if the node name is not found

Raises:

• ValueError: if the node name is not found and if fail_if_not_found is set

function clean_graph_after_node_op_type

clean_graph_after_node_op_type(
    onnx_model: ModelProto,
    node_op_type: str,
    fail_if_not_found: bool = True
)

Clean the graph of the onnx model by removing nodes after the given node type.

Args:

• onnx_model (onnx.ModelProto): The onnx model.

• node_op_type (str): The node's op_type whose following nodes will be removed.

• fail_if_not_found (bool): If true, abort if the node op_type is not found

Raises:

• ValueError: if the node op_type is not found and if fail_if_not_found is set

Quantization is the process of constraining an input from a continuous or otherwise large set of values (such as real numbers) to a discrete set (such as integers).

This means that some accuracy in the representation is lost (e.g. a simple approach is to eliminate least-significant bits). However, in many cases in machine learning, it is possible to adapt the models to give meaningful results while using these smaller data types. This significantly reduces the number of bits necessary for intermediary results during the execution of these machine learning models.

Since FHE is currently limited to 8-bit integers, it is necessary to quantize models to make them compatible. As a general rule, the smaller the precision models, the better the FHE performance.

Overview of quantization in Concrete-ML

Quantization implemented in Concrete-ML is applied in two ways:

1. Built-in models apply quantization internally and the user only needs to configure some quantization parameters. This approach requires little work by the user but may not be a one-size-fits-all solution for all types of models. The final quantized model is FHE friendly and ready to predict over encrypted data. In this setting, Post-Training Quantization (PTQ) is for linear models, data quantization is used for tree-based models and, finally, Quantization Aware Training (QAT) is included in the built-in neural network models.

2. For custom neural networks with more complex topology, obtaining FHE-compatible models with good accuracy requires QAT. Concrete-ML offers the possibility for the user to perform quantization before compiling to FHE. This can be achieved through a third-party library that offers QAT tools, such as for PyTorch. In this approach, the user is responsible for implementing a full-integer model, respecting FHE constraints. Please refer to the for tips on designing FHE neural networks.

Basics of quantization

Let be the range of a value to quantize where is the minimum and is the maximum. To quantize a range of floating point values (in ) to integer values (in ), the first step is to choose the data type that is going to be used. Concrete, the framework used by Concrete-ML, is currently limited to 8-bit integers, so this will be the value used in this example. Knowing the number of bits that can be used for a value in the range , the scale can be computed :

where is the number of bits (). For the sake of example, let's take .

In practice, the quantization scale is then . This means the gap between consecutive representable values cannot be smaller than , which, in turn, means there can be a substantial loss of precision. Every interval of length will be represented by a value within the range .

The other important parameter from this quantization schema is the zero point value. This essentially brings the 0 floating point value to a specific integer. If the quantization scheme is asymmetric (quantized values are not centered in 0), the resulting integer will be in .

When using quantized values in a matrix multiplication or convolution, the equations for computing the result become more complex. The IntelLabs distiller quantization documentation provides a more of the maths used to quantize values and how to keep computations consistent.

Configuring model quantization parameters

Built-in models provide a simple interface for configuring quantization parameters, most notably the number of bits used for inputs, model weights, intermediary and output values.

For , the quantization is done post-training. Thus, the model is trained in floating point, and then, the best integer weight representations are found, depending on the distribution of inputs and weights. For these models, the user can select the value of the n_bits parameter.

For linear models, n_bits is used to quantize both model inputs and weights. Depending on the number of features, you can use a single integer value for the n_bits parameter, e.g. a value between 2 and 7. When the number of features is high, the n_bits parameter should be decreased if you encounter compilation errors. It is also possible to quantize inputs and weights with different number of bits by passing a dictionary to n_bits , containing the op_inputs and op_weights keys.

For , the training and test data is quantized. The maximum accumulator bit-width for a model trained with n_bits=n for this type of model is known beforehand: it will need n+1 bits. Thus, as Concrete-ML only supports up to 8-bit integers, n should be less than 8. Through experimentation, it was determined that in many cases a value of 5 or 6 bits gives the same accuracy as training in floating point.

For the built-in , several linear layers are used. Thus, the outputs of a layer are used as inputs to a new layer. Built-in neural networks use Quantization Aware Training. The parameters controlling the maximum accumulator bit-width are the number of weights and activation bits ( module__n_w_bits, module__n_a_bits ), but also the pruning factor. This factor is determined automatically by specifying a desired accumulator bit-width module__n_accum_bits and, optionally, a multiplier factor, module__n_hidden_neurons_multiplier.

Quantizing model inputs and outputs

The models implemented in Concrete-ML provide features to let the user quantize the input data and de-quantize the output data.

In a client/server setting, the client is responsible for quantizing inputs before sending them, encrypted, to the server. Further, the client must de-quantize the encrypted integer results received from the server. See the section for more details.

Here is a simple example showing how to perform inference, starting from float values and ending up with float values. Note that the FHE engine that is compiled for the ML models does not support data batching.

Resources

• IntelLabs distiller explanation of quantization:

In addition to Concrete-ML models and to custom models in torch, it is also possible to directly compile ONNX models. This can be particularly appealing, notably to import models trained with Keras.

ONNX models can be compiled by directly importing models that are already quantized with Quantization Aware Training (QAT). or by performing Post-Training Quantization (PTQ) with Concrete-ML.

Simple example

The following example shows how to compile an ONNX model using PTQ. The model was initially trained using Keras before being exported to ONNX. The training code is not shown here.

import numpy
import onnx
import tensorflow
import tf2onnx

from concrete.ml.torch.compile import compile_onnx_model
from concrete.numpy.compilation import Configuration


class FC(tensorflow.keras.Model):
    """A fully-connected model."""

    def __init__(self):
        super().__init__()
        hidden_layer_size = 10
        output_size = 5

        self.dense1 = tensorflow.keras.layers.Dense(
            hidden_layer_size,
            activation=tensorflow.nn.relu,
        )
        self.dense2 = tensorflow.keras.layers.Dense(output_size, activation=tensorflow.nn.relu6)
        self.flatten = tensorflow.keras.layers.Flatten()

    def call(self, inputs):
        """Forward function."""
        x = self.flatten(inputs)
        x = self.dense1(x)
        x = self.dense2(x)
        return self.flatten(x)


n_bits = 6
input_output_feature = 2
input_shape = (input_output_feature,)
num_inputs = 1
n_examples = 5000

# Define the Keras model
keras_model = FC()
keras_model.build((None,) + input_shape)
keras_model.compute_output_shape(input_shape=(None, input_output_feature))

# Create random input
input_set = numpy.random.uniform(-100, 100, size=(n_examples, *input_shape))

# Convert to ONNX
tf2onnx.convert.from_keras(keras_model, opset=14, output_path="tmp.model.onnx")

onnx_model = onnx.load("tmp.model.onnx")
onnx.checker.check_model(onnx_model)

# Compile
quantized_numpy_module = compile_onnx_model(
    onnx_model, input_set, n_bits=2
)

# Create test data from the same distribution and quantize using
# learned quantization parameters during compilation
x_test = tuple(numpy.random.uniform(-100, 100, size=(1, *input_shape)) for _ in range(num_inputs))
qtest = quantized_numpy_module.quantize_input(x_test)

y_clear = quantized_numpy_module(*qtest)
y_fhe = quantized_numpy_module.forward_fhe.encrypt_run_decrypt(*qtest)

print("Execution in clear: ", y_clear)
print("Execution in FHE:   ", y_fhe)
print("Equality:           ", numpy.sum(y_clear == y_fhe), "over", numpy.size(y_fhe), "values")

Quantization Aware Training

QAT models contain quantizers in the ONNX graph. These quantizers ensure that the inputs to the Linear/Dense and Conv layers are quantized. Since these QAT models have quantizers that are configured during training to a specific number of bits, the ONNX graph will need to be imported using the same settings:

n_bits_qat = 3  # number of bits for weights and activations during training

quantized_numpy_module = compile_onnx_model(
    onnx_model,
    input_set,
    n_bits=n_bits_qat,
)

Supported operators

The following operators are supported for evaluation and conversion to an equivalent FHE circuit. Other operators were not implemented either due to FHE constraints, or because they are rarely used in PyTorch activations or scikit-learn models.

• Abs

• Acos

• Acosh

• Add

• Asin

• Asinh

• Atan

• Atanh

• AveragePool

• BatchNormalization

• Cast

• Celu

• Clip

• Constant

• Conv

• Cos

• Cosh

• Div

• Elu

• Equal

• Erf

• Exp

• Flatten

• Floor

• Gemm

• Greater

• GreaterOrEqual

• HardSigmoid

• HardSwish

• Identity

• LeakyRelu

• Less

• LessOrEqual

• Log

• MatMul

• Max

• Min

• Mul

• Neg

• Not

• Or

• PRelu

• Pad

• Pow

• ReduceSum

• Relu

• Reshape

• Round

• Selu

• Sigmoid

• Sign

• Sin

• Sinh

• Softplus

• Sub

• Tan

• Tanh

• ThresholdedRelu

• Transpose

• Where

• onnx.brevitas.Quant

The ONNX import section gave an overview of the conversion of a generic ONNX graph to a FHE compatible Concrete-ML op-graph. This section describes the implementation of operations in the Concrete-ML op-graph and the way floating point can be used in some parts of the op-graphs through table lookup operations.

Float vs. quantized operations

Concrete, the underlying implementation of TFHE that powers Concrete-ML, enables two types of operations on integers:

• arithmetic operations: the addition of two encrypted values and multiplication of encrypted values with clear scalars. These are used, for example, in dot-products, matrix multiplication (linear layers) and convolution.

• table lookup operations (TLU): using an encrypted value as an index, return the value of a lookup table at that index. This is implemented using Programmable Bootstrapping. This operation is used to perform any non-linear computation such as activation functions, quantization and normalization.

Since machine learning models use floating point inputs and weights, they first need to be converted to integers using quantization.

Alternatively, it is possible to use a table lookup to avoid the quantization of the entire graph, by converting floating-point ONNX subgraphs into lambdas and computing their corresponding lookup tables to be evaluated directly in FHE. This operator-fusion technique only requires the input and output of the lambdas to be integers.

For example, in the following graph there is a single input, which must be an encrypted integer tensor. The following series of univariate functions is then fed into a matrix multiplication (MatMul) and fused into a single table lookup with integer inputs and outputs.

ONNX operations

Concrete-ML implements ONNX operations using Concrete-Numpy, which can handle floating point operations, as long as they can be fused to an integer lookup table. The ONNX operations implementations are based on the QuantizedOp class.

There are two modes of creation of a single table lookup for a chain of ONNX operations:

• float mode: when the operation can be fused

• mixed float/integer: when the ONNX operation needs to perform arithmetic operations

Thus, QuantizedOp instances may need to quantize their inputs or the result of their computation, depending on their position in the graph.

The QuantizedOp class provides a generic implementation of an ONNX operation, including the quantization of inputs and outputs, with the computation implemented in NumPy in ops_impl.py. It is possible to picture at the architecture of the QuantizedOp as the following structure:

This figure shows that the QuantizedOp has a body that implements the computation of the operation, following the ONNX spec. The operation's body can take either integer or float inputs and can output float or integer values. Two quantizers are attached to the operation: one that takes float inputs and produces integer inputs and one that does the same for the output.

Operations that can fuse to a TLU

Depending on the position of the op in the graph and its inputs, the QuantizedOp can be fully fused to a TLU.

Many ONNX ops are trivially univariate, as they multiply variable inputs with constants or apply univariate functions such as ReLU, Sigmoid, etc. This includes operations between the input and the MatMul in the graph above (subtraction, comparison, multiplication, etc. between inputs and constants).

Operations that work on integers

Operations, such as matrix multiplication of encrypted inputs with a constant matrix or convolution with constant weights, require that the encrypted inputs be integers. In this case, the input quantizer of the QuantizedOp is applied. These types of operations are implemented with a class that derives from QuantizedOp and implements q_impl, such as QuantizedGemm and QuantizedConv.

Operations that produce graph outputs

Finally, some operations produce graph outputs, which must be integers. These operations need to quantize their outputs as follows:

The diagram above shows that both float ops and integer ops need to quantize their outputs to integers, when placed at the end of the graph.

Putting it all together

To chain the operation types described above following the ONNX graph, Concrete-ML constructs a function that calls the q_impl of the QuantizedOp instances in the graph in sequence, and uses Concrete-Numpy to trace the execution and compile to FHE. Thus, in this chain of function calls, all groups of that instruction that operate in floating point will be fused to TLUs. In FHE, this lookup table is computed with a PBS.

The red contours show the groups of elementary Concrete-Numpy instructions that will be converted to TLUs.

Note that the input is slightly different from the QuantizedOp. Since the encrypted function takes integers as inputs, the input needs to be de-quantized first.

Implementing a QuantizedOp

QuantizedOp is the base class for all ONNX-quantized operators. It abstracts away many things to allow easy implementation of new quantized ops.

Determining if the operation can be fused

The QuantizedOp class exposes a function can_fuse that

• helps to determine the type of implementation that will be traced

• determines whether operations further in the graph, that depend on the results of this operation, can fuse

In most cases, ONNX ops have a single variable input and one or more constant inputs.

When the op implements elementwise operations between the inputs and constants (addition, subtract, multiplication, etc), the operation can be fused to a TLU. Thus, by default in QuantizedOp, the can_fuse function returns True.

When the op implements operations that mix the various scalars in the input encrypted tensor, the operation cannot fuse, as table lookups are univariate. Thus, operations such as QuantizedGemm, QuantizedConv return False in can_fuse.

Some operations may be found in both settings above. A mechanism is implemented in Concrete-ML to determine if the inputs of a QuantizedOp are produced by a unique integer tensor. Therefore, the can_fuse function of some QuantizedOp types (addition, subtraction) will allow fusion to take place if both operands are produced by a unique integer tensor:

def can_fuse(self) -> bool:
    return len(self._int_input_names) == 1

Case 1: A floating point version of the op is sufficient

You can check ops_impl.py to see how some operations are implemented in NumPy. The declaration convention for these operations is as follows:

• The required inputs should be positional arguments only before the /, which marks the limit of the positional arguments

• The optional inputs should be positional or keyword arguments between the / and *, which marks the limits of positional or keyword arguments

• The operator attributes should be keyword arguments only after the *

The proper use of positional/keyword arguments is required to allow the QuantizedOp class to properly populate metadata automatically. It uses Python inspect modules and stores relevant information for each argument related to its positional/keyword status. This allows using the Concrete-NumPy implementation as specifications for QuantizedOp, which removes some data duplication and allows having a single source of truth for QuantizedOp and ONNX-NumPy implementations.

In that case (unless the quantized implementation requires special handling like QuantizedGemm), you can just set _impl_for_op_named to the name of the ONNX op for which the quantized class is implemented (this uses the mapping ONNX_OPS_TO_NUMPY_IMPL in onnx_utils.py to get the correct implementation).

Case 2: An integer implementation of the op is necessary

Providing an integer implementation requires sub-classing QuantizedOp to create a new operation. This sub-class must override q_impl in order to provide an integer implementation. QuantizedGemm is an example of such a case where quantized matrix multiplication requires proper handling of scales and zero points. The q_impl of that class reflects this.

In the body of q_impl, in order to obtain quantized integer values you can use the _prepare_inputs_with_constants function as such:

from concrete.ml.quantization import QuantizedArray

def q_impl(
    self,
    *q_inputs: QuantizedArray,
    **attrs,
) -> QuantizedArray:

    # Retrieve the quantized inputs
    prepared_inputs = self._prepare_inputs_with_constants(
        *q_inputs, calibrate=False, quantize_actual_values=True
    )

Here, prepared_inputs will contain one or more QuantizedArray of which the qvalues are the quantized integers.

Once the required integer processing code is implemented, the output of the q_impl function must be implemented as a single QuantizedArray. Most commonly, this is built using the de-quantized results of the processing done in q_impl.

    result = (
        sum_result.astype(numpy.float32) - q_input.quantizer.zero_point
    ) * q_input.quantizer.scale

    return QuantizedArray(
        self.n_bits,
        result,
        value_is_float=True,
        options=self.input_quant_opts,
        stats=self.output_quant_stats,
        params=self.output_quant_params,
    )

Case 3: Both a floating point and an integer implementation are necessary

In this case, in q_impl you can check whether the current operation can be fused by calling self.can_fuse(). You can then have both a floating point and an integer implementation. The traced execution path will depend on can_fuse():

def q_impl(
    self,
    *q_inputs: QuantizedArray,
    **attrs,
) -> QuantizedArray:

    execute_in_float = len(self.constant_inputs) > 0 or self.can_fuse()

    # a floating point implementation that can fuse
    if execute_in_float:
        prepared_inputs = self._prepare_inputs_with_constants(
            *q_inputs, calibrate=False, quantize_actual_values=False
        )

        result = prepared_inputs[0] + self.b_sign * prepared_inputs[1]
        return QuantizedArray(
            self.n_bits,
            result,
            # ......
        )
    else:
        prepared_inputs = self._prepare_inputs_with_constants(
            *q_inputs, calibrate=False, quantize_actual_values=True
        )
        # an integer implementation follows, see Case 2
        # ....

module concrete.ml.torch.compile

torch compilation function.

Global Variables

• MAXIMUM_TLU_BIT_WIDTH

• DEFAULT_P_ERROR_PBS

• OPSET_VERSION_FOR_ONNX_EXPORT

function convert_torch_tensor_or_numpy_array_to_numpy_array

convert_torch_tensor_or_numpy_array_to_numpy_array(
    torch_tensor_or_numpy_array: Union[Tensor, ndarray]
) → ndarray

Convert a torch tensor or a numpy array to a numpy array.

Args:

• torch_tensor_or_numpy_array (Tensor): the value that is either a torch tensor or a numpy array.

Returns:

• numpy.ndarray: the value converted to a numpy array.

function compile_torch_model

compile_torch_model(
    torch_model: Module,
    torch_inputset: Union[Tensor, ndarray, Tuple[Union[Tensor, ndarray], ]],
    import_qat: bool = False,
    configuration: Optional[Configuration] = None,
    compilation_artifacts: Optional[DebugArtifacts] = None,
    show_mlir: bool = False,
    n_bits=16,
    use_virtual_lib: bool = False,
    p_error: Optional[float] = 6.3342483999973e-05
) → QuantizedModule

Compile a torch module into an FHE equivalent.

Take a model in torch, turn it to numpy, quantize its inputs / weights / outputs and finally compile it with Concrete-Numpy

Args:

• torch_model (torch.nn.Module): the model to quantize

• torch_inputset (Dataset): the inputset, can contain either torch tensors or numpy.ndarray, only datasets with a single input are supported for now.

• import_qat (bool): Set to True to import a network that contains quantizers and was trained using quantization aware training

• configuration (Configuration): Configuration object to use during compilation

• compilation_artifacts (DebugArtifacts): Artifacts object to fill during compilation

• show_mlir (bool): if set, the MLIR produced by the converter and which is going to be sent to the compiler backend is shown on the screen, e.g., for debugging or demo

• n_bits: the number of bits for the quantization

• use_virtual_lib (bool): set to use the so called virtual lib simulating FHE computation. Defaults to False

• p_error (Optional[float]): probability of error of a PBS

Returns:

• QuantizedModule: The resulting compiled QuantizedModule.

function compile_onnx_model

compile_onnx_model(
    onnx_model: ModelProto,
    torch_inputset: Union[Tensor, ndarray, Tuple[Union[Tensor, ndarray], ]],
    import_qat: bool = False,
    configuration: Optional[Configuration] = None,
    compilation_artifacts: Optional[DebugArtifacts] = None,
    show_mlir: bool = False,
    n_bits=16,
    use_virtual_lib: bool = False,
    p_error: Optional[float] = 6.3342483999973e-05
) → QuantizedModule

Compile a torch module into an FHE equivalent.

Take a model in torch, turn it to numpy, quantize its inputs / weights / outputs and finally compile it with Concrete-Numpy

Args:

• onnx_model (onnx.ModelProto): the model to quantize

• torch_inputset (Dataset): the inputset, can contain either torch tensors or numpy.ndarray, only datasets with a single input are supported for now.

• import_qat (bool): Flag to signal that the network being imported contains quantizers in in its computation graph and that Concrete ML should not requantize it.

• configuration (Configuration): Configuration object to use during compilation

• compilation_artifacts (DebugArtifacts): Artifacts object to fill during compilation

• show_mlir (bool): if set, the MLIR produced by the converter and which is going to be sent to the compiler backend is shown on the screen, e.g., for debugging or demo

• n_bits: the number of bits for the quantization

• use_virtual_lib (bool): set to use the so called virtual lib simulating FHE computation. Defaults to False.

• p_error (Optional[float]): probability of error of a PBS

Returns:

• QuantizedModule: The resulting compiled QuantizedModule.

function compile_brevitas_qat_model

compile_brevitas_qat_model(
    torch_model: Module,
    torch_inputset: Union[Tensor, ndarray, Tuple[Union[Tensor, ndarray], ]],
    n_bits: Union[int, dict],
    configuration: Optional[Configuration] = None,
    compilation_artifacts: Optional[DebugArtifacts] = None,
    show_mlir: bool = False,
    use_virtual_lib: bool = False,
    p_error: Optional[float] = 6.3342483999973e-05,
    output_onnx_file: Union[Path, str] = None
) → QuantizedModule

Compile a Brevitas Quantization Aware Training model.

The torch_model parameter is a subclass of torch.nn.Module that uses quantized operations from brevitas.qnn. The model is trained before calling this function. This function compiles the trained model to FHE.

Args:

• torch_model (torch.nn.Module): the model to quantize

• torch_inputset (Dataset): the inputset, can contain either torch tensors or numpy.ndarray, only datasets with a single input are supported for now.

• n_bits (Union[int,dict]): the number of bits for the quantization

• configuration (Configuration): Configuration object to use during compilation

• compilation_artifacts (DebugArtifacts): Artifacts object to fill during compilation

• show_mlir (bool): if set, the MLIR produced by the converter and which is going to be sent to the compiler backend is shown on the screen, e.g., for debugging or demo

• use_virtual_lib (bool): set to use the so called virtual lib simulating FHE computation, defaults to False.

• p_error (Optional[float]): probability of error of a PBS

• output_onnx_file (str): temporary file to store ONNX model. If None a temporary file is generated

Returns:

• QuantizedModule: The resulting compiled QuantizedModule.

module concrete.ml.sklearn.xgb

Implements XGBoost models.

class XGBClassifier

Implements the XGBoost classifier.

method __init__

__init__(
    n_bits: int = 6,
    max_depth: Optional[int] = 3,
    learning_rate: Optional[float] = 0.1,
    n_estimators: Optional[int] = 20,
    objective: Optional[str] = 'binary:logistic',
    booster: Optional[str] = None,
    tree_method: Optional[str] = None,
    n_jobs: Optional[int] = None,
    gamma: Optional[float] = None,
    min_child_weight: Optional[float] = None,
    max_delta_step: Optional[float] = None,
    subsample: Optional[float] = None,
    colsample_bytree: Optional[float] = None,
    colsample_bylevel: Optional[float] = None,
    colsample_bynode: Optional[float] = None,
    reg_alpha: Optional[float] = None,
    reg_lambda: Optional[float] = None,
    scale_pos_weight: Optional[float] = None,
    base_score: Optional[float] = None,
    missing: float = nan,
    num_parallel_tree: Optional[int] = None,
    monotone_constraints: Optional[Dict[str, int], str] = None,
    interaction_constraints: Optional[str, List[Tuple[str]]] = None,
    importance_type: Optional[str] = None,
    gpu_id: Optional[int] = None,
    validate_parameters: Optional[bool] = None,
    predictor: Optional[str] = None,
    enable_categorical: bool = False,
    use_label_encoder: bool = False,
    random_state: Optional[RandomState, int] = None,
    verbosity: Optional[int] = None
)

property onnx_model

Get the ONNX model.

.. # noqa: DAR201

Returns:

• onnx.ModelProto: the ONNX model

method post_processing

post_processing(y_preds: ndarray) → ndarray

Apply post-processing to the predictions.

Args:

• y_preds (numpy.ndarray): The predictions.

Returns:

• numpy.ndarray: The post-processed predictions.

class XGBRegressor

Implements the XGBoost regressor.

method __init__

__init__(
    n_bits: int = 6,
    max_depth: Optional[int] = 3,
    learning_rate: Optional[float] = 0.1,
    n_estimators: Optional[int] = 20,
    objective: Optional[str] = 'reg:squarederror',
    booster: Optional[str] = None,
    tree_method: Optional[str] = None,
    n_jobs: Optional[int] = None,
    gamma: Optional[float] = None,
    min_child_weight: Optional[float] = None,
    max_delta_step: Optional[float] = None,
    subsample: Optional[float] = None,
    colsample_bytree: Optional[float] = None,
    colsample_bylevel: Optional[float] = None,
    colsample_bynode: Optional[float] = None,
    reg_alpha: Optional[float] = None,
    reg_lambda: Optional[float] = None,
    scale_pos_weight: Optional[float] = None,
    base_score: Optional[float] = None,
    missing: float = nan,
    num_parallel_tree: Optional[int] = None,
    monotone_constraints: Optional[Dict[str, int], str] = None,
    interaction_constraints: Optional[str, List[Tuple[str]]] = None,
    importance_type: Optional[str] = None,
    gpu_id: Optional[int] = None,
    validate_parameters: Optional[bool] = None,
    predictor: Optional[str] = None,
    enable_categorical: bool = False,
    use_label_encoder: bool = False,
    random_state: Optional[RandomState, int] = None,
    verbosity: Optional[int] = None
)

property onnx_model

Get the ONNX model.

.. # noqa: DAR201

Returns:

• onnx.ModelProto: the ONNX model

method fit

fit(X, y, **kwargs) → Any

Fit the tree-based estimator.

Args:

• X : training data By default, you should be able to pass: * numpy arrays * torch tensors * pandas DataFrame or Series

• y (numpy.ndarray): The target data.

• **kwargs: args for super().fit

Returns:

• Any: The fitted model.

method post_processing

post_processing(y_preds: ndarray) → ndarray

Apply post-processing to the predictions.

Args:

• y_preds (numpy.ndarray): The predictions.

Returns:

• numpy.ndarray: The post-processed predictions.

module concrete.ml.sklearn.torch_module

Implement torch module.

This section provides a set of tools and guidelines to help users build optimized FHE-compatible models.

Virtual library

The Virtual Lib in Concrete-ML is a prototype that provides drop-in replacements for Concrete-Numpy's compiler, allowing users to simulate what would happen when converting a model to FHE without the current bit-width constraint. Additionally, it quickly simulates the behavior with 8 bits or less without actually doing the FHE computations.

The Virtual Lib can be useful when developing and iterating on an ML model implementation. For example, you can check that your model is compatible in terms of operands (all integers) with the Virtual Lib compilation. Then, you can check how many bits your ML model would require, which can give you hints as to how it should be modified if you want to compile it to an actual FHE Circuit (not a simulated one) that only supports 8 bits of integer precision.

The Virtual Lib, being pure Python and not requiring crypto key generation, can be much faster than the actual compilation and FHE execution, thus allowing for faster iterations, debugging and FHE simulation, regardless of the bit-width used. For example, this was used for the red/blue contours in the , as computing in FHE for the whole grid and all the classifiers would take significant time.

The following example shows how to use the Virtual Lib in Concrete-ML. Simply add use_virtual_lib = True and enable_unsafe_features = True in a Configuration. The result of the compilation will then be a simulated circuit that allows for more precision or simulated FHE execution.

Compilation debugging

The following example produces a neural network that is not FHE-compatible:

Upon execution, the compiler will raise the following error:

Knowing that a linear/dense layer is implemented as a matrix multiplication, it can determine which parts of the op-graph listing in the exception message above correspond to which layers.

Layer weights initialization:

Input processing and quantization:

First dense layer and activation function:

Second dense layer and activation function:

Third dense layer and output quantization:

We can see here that the error is in the second layer. Reducing the number of neurons in this layer will resolve the error and make the network FHE-compatible:

Complexity analysis

In FHE, univariate functions are encoded as table lookups, which are then implemented using Programmable Bootstrapping (PBS). PBS is a powerful technique but will require significantly more computing resources, and thus time, than simpler encrypted operations such matrix multiplications, convolution or additions.

Furthermore, the cost of PBS will depend on the bit-width of the compiled circuit. Every additional bit in the maximum bit-width raises the complexity of the PBS by a significant factor. It may be of interest to the model developer, then, to determine the bit-width of the circuit and the amount of PBS it performs.

This can be done by inspecting the MLIR code produced by the compiler:

Concrete-ML Model

Compiled MLIR model

There are several calls to FHELinalg.apply_mapped_lookup_table and FHELinalg.apply_lookup_table. These calls apply PBS to the cells of their input tensors. Their inputs in the listing above are: tensor<1x2x!FHE.eint<8>> for the first and last call and tensor<1x50x!FHE.eint<8>> for the two calls in the middle. Thus, PBS is applied 104 times.

Getting the bit-width of the circuit is then simply:

Decreasing the number of bits and the number of PBS induces large reductions in the computation time of the compiled circuit.

module concrete.ml.quantization.quantized_module

QuantizedModule API.

Global Variables

• DEFAULT_P_ERROR_PBS

class QuantizedModule

Inference for a quantized model.

method __init__

property fhe_circuit

Get the FHE circuit.

Returns:

• Circuit: the FHE circuit

property is_compiled

Return the compiled status of the module.

Returns:

• bool: the compiled status of the module.

property onnx_model

Get the ONNX model.

.. # noqa: DAR201

Returns:

• _onnx_model (onnx.ModelProto): the ONNX model

property post_processing_params

Get the post-processing parameters.

Returns:

• Dict[str, Any]: the post-processing parameters

method compile

Compile the forward function of the module.

Args:

• q_inputs (Union[Tuple[numpy.ndarray, ...], numpy.ndarray]): Needed for tracing and building the boundaries.

• configuration (Optional[Configuration]): Configuration object to use during compilation

• compilation_artifacts (Optional[DebugArtifacts]): Artifacts object to fill during

• show_mlir (bool): if set, the MLIR produced by the converter and which is going to be sent to the compiler backend is shown on the screen, e.g., for debugging or demo. Defaults to False.

• use_virtual_lib (bool): set to use the so called virtual lib simulating FHE computation. Defaults to False.

• p_error (Optional[float]): probability of error of a PBS.