Depending on your OS, Concrete ML may be installed with Docker or with pip:

Only some versions of python are supported: In the current release, these are 3.8, 3.9 and 3.10. The Concrete ML Python package requires glibc >= 2.28. On Linux, you can check your glibc version by running ldd --version.

Concrete ML can be installed on Kaggle (see question on community for more details) and on Google Colab.

Most of these limits are shared with the rest of the Concrete stack (namely Concrete-Python). Support for more platforms will be added in the future.

Using PyPi

Requirements

Installing Concrete ML using PyPi requires a Linux-based OS or macOS (both x86 and Apple Silicon CPUs are supported).

Installing on Windows can be done using Docker or WSL. On WSL, Concrete ML will work as long as the package is not installed in the /mnt/c/ directory, which corresponds to the host OS filesystem.

Installation

To install Concrete ML from PyPi, run the following:

pip install -U pip wheel setuptools
pip install concrete-ml

This will automatically install all dependencies, notably Concrete.

If you encounter any issue during installation on Apple Silicon mac, please visit this troubleshooting guide on community.

Using Docker

Concrete ML can be installed using Docker by either pulling the latest image or a specific version:

docker pull zamafhe/concrete-ml:latest
# or
docker pull zamafhe/concrete-ml:v0.4.0

The image can be used with Docker volumes, see the Docker documentation here.

The image can then be used via the following command:

# Without local volume:
docker run --rm -it -p 8888:8888 zamafhe/concrete-ml

# With local volume to save notebooks on host:
docker run --rm -it -p 8888:8888 -v /host/path:/data zamafhe/concrete-ml

This will launch a Concrete ML enabled Jupyter server in Docker that can be accessed directly from a browser.

Alternatively, a shell can be lauched in Docker, with or without volumes:

docker run --rm -it zamafhe/concrete-ml /bin/bash

Concrete ML provides simple built-in neural networks models with a scikit-learn interface through the NeuralNetClassifier and NeuralNetRegressor classes.

The neural network models are implemented with skorch, which provides a scikit-learn-like interface to Torch models (more here).

Concrete ML models are multi-layer, fully-connected, networks with customizable activation functions and have a number of neurons in each layer. This approach is similar to what is available in scikit-learn when using the MLPClassifier/MLPRegressor classes. The built-in models train easily with a single call to .fit(), which will automatically quantize weights and activations. These models use Quantization Aware Training, allowing good performance for low precision (down to 2-3 bits) weights and activations.

While NeuralNetClassifier and NeuralNetClassifier provide scikit-learn-like models, their architecture is somewhat restricted to make training easy and robust. If you need more advanced models, you can convert custom neural networks as described in the FHE-friendly models documentation.

Example usage

To create an instance of a Fully Connected Neural Network (FCNN), you need to instantiate one of the NeuralNetClassifier and NeuralNetRegressor classes and configure a number of parameters that are passed to their constructor. Note that some parameters need to be prefixed by module__, while others don't. The parameters related to the model (i.e., the underlying nn.Module), must have the prefix. The parameters related to training options do not require the prefix.

from concrete.ml.sklearn import NeuralNetClassifier
import torch.nn as nn

n_inputs = 10
n_outputs = 2
params = {
    "module__n_layers": 2,
    "max_epochs": 10,
}

concrete_classifier = NeuralNetClassifier(**params)

The Classifier Comparison notebook shows the behavior of built-in neural networks on several synthetic data-sets.

The figure above right shows the Concrete ML neural network, trained with Quantization Aware Training in an FHE-compatible configuration. The figure compares this network to the floating-point equivalent, trained with scikit-learn.

Architecture parameters

• module__n_layers: number of layers in the FCNN, must be at least 1. Note that this is the total number of layers. For a single, hidden layer NN model, set module__n_layers=2

• module__activation_function: can be one of the Torch activations (e.g., nn.ReLU, see the full list here). Neural networks with nn.ReLU activation benefit from specific optimizations that make them around 10x faster than networks with other activation functions.

Quantization parameters

• n_w_bits (default 3): number of bits for weights

• n_a_bits (default 3): number of bits for activations and inputs

• n_accum_bits: maximum accumulator bit-width that is desired. By default, this is unbounded, which, for weight and activation bit-width settings, may make the trained networks fail in compilation. When used, the implementation will attempt to keep accumulators under this bit-width through pruning (i.e., setting some weights to zero)

• power_of_two_scaling (default True): forces quantization scales to be powers-of-two, which, when coupled with the ReLU activation, benefits from strong FHE inference time optimization. See this section in the quantization documentation for more details.

Training parameters (from skorch)

• max_epochs: The number of epochs to train the network (default 10)

• verbose: Whether to log loss/metrics during training (default: False)

• lr: Learning rate (default 0.001)

Other parameters from skorch can be found in the skorch documentation.

Advanced parameters

• module__n_hidden_neurons_multiplier: The number of hidden neurons will be automatically set proportional to the dimensionality of the input. This parameter controls the proportionality factor and is set to 4 by default. This value gives good accuracy while avoiding accumulator overflow. See the pruning and quantization sections for more info.

Class weights

You can give weights to each class to use in training. Note that this must be supported by the underlying PyTorch loss function.

    from sklearn.utils.class_weight import compute_class_weight
    params["criterion__weight"] = compute_class_weight("balanced", classes=classes, y=y_train)

Overflow errors

The n_accum_bits parameter influences training accuracy as it controls the number of non-zero neurons that are allowed in each layer. Increasing n_accum_bits improves accuracy, but should take into account precision limitations to avoid an overflow in the accumulator. The default value is a good compromise that avoids an overflow in most cases, but you may want to change the value of this parameter to reduce the breadth of the network if you have overflow errors.

Furthermore, the number of neurons on intermediate layers is controlled through the n_hidden_neurons_multiplier parameter - a value of 1 will make intermediate layers have the same number of neurons as the number of dimensions of the input data.

Concrete ML models and data-frames can be easily deployed in a client/server setting, enabling the creation of privacy-preserving services in the cloud.

As seen in the concepts section, once compiled to FHE, a Concrete ML model or data-frame generates machine code that execute prediction, training or pre-processing on encrypted data. Secret encryption keys are needed so that the user can securely encrypt their data and decrypt the execution result. An evaluation key is also needed for the server to securely process the user's encrypted data.

Keys are generated by the user once for each service they use, based on the model the service provides and its cryptographic parameters.

The overall communications protocol to enable cloud deployment of machine learning services can be summarized in the following diagram:

The steps detailed above are:

1. The model developer deploys the compiled machine learning model to the server. This model includes the cryptographic parameters. The server is now ready to provide private inference. Crypto-graphic parameters and compiled programs for data-frames are included directly in Concrete ML.

2. The client requests the cryptographic parameters (also called "client specs"). Once it receives them from the server, the secret and evaluation keys are generated.

3. The client sends the evaluation key to the server. The server is now ready to accept requests from this client. The client sends their encrypted data. Serialized data-frames include client evaluation keys.

4. The server uses the evaluation key to securely run prediction, training and pre-processing on the user's data and sends back the encrypted result.

5. The client now decrypts the result and can send back new requests.

For more information on how to implement this basic secure inference protocol, refer to the Production Deployment section and to the client/server example. For information on training on encrypted data, see the corresponding section.

Get started

Learn the basics of Concrete ML, set it up, and make it run with ease.

Build with Concrete ML

Start building with Concrete ML by exploring its core features, discovering essential guides, and learning more with user-friendly tutorials.

Explore more

Access to additional resources and join the Zama community.

References & Explanations

Refer to the API, review product architecture, and access additional resources for in-depth explanations while working with Concrete ML.

Support channels

Ask technical questions and discuss with the community. Our team of experts usually answers within 24 hours in working days.

Developers

Collaborate with us to advance the FHE spaces and drive innovation together.

Concrete ML provides several of the most popular linear models for regression and classification that can be found in :

Using these models in FHE is extremely similar to what can be done with scikit-learn's , making it easy for data scientists who have used this framework to get started with Concrete ML.

Models are also compatible with some of scikit-learn's main workflows, such as Pipeline() and GridSearch().

Pre-trained models

It is possible to convert an already trained scikit-learn linear model to a Concrete ML one by using the method. See . This functionality is only available for linear models.

Quantization parameters

The n_bits parameter controls the bit-width of the inputs and weights of the linear models. When non-linear mapping is applied by the model, such as exp or sigmoid, Concrete ML applies it on the client-side, on clear-text values that are the decrypted output of the linear part of the model. Thus, Linear Models do not use table lookups, and can, therefore, use high precision integers for weight and inputs.

The n_bits parameter can be set to 8 or more bits for models with up to 300 input dimensions. When the input has more dimensions, n_bits must be reduced to 6-7. All performance metrics are preserved down to n_bits=6, compared to the non-quantized float models from scikit-learn.

Example

The overall accuracy scores are identical (93%) between the scikit-learn model (executed in the clear) and the Concrete ML one (executed in FHE). In fact, quantization has little impact on the decision boundaries, as linear models are able to consider large precision numbers when quantizing inputs and weights in Concrete ML. Additionally, as the linear models do not use PBS, the FHE computations are always exact. This means that the FHE predictions are always identical to the quantized clear ones.

Loading a pre-trained model

An alternative to the example above is to train a scikit-learn model in a separate step and then to convert it to Concrete ML.

In addition to the built-in models, Concrete ML supports generic machine learning models implemented with Torch, or exported as ONNX graphs.

As Quantization Aware Training (QAT) is the most appropriate method of training neural networks that are compatible with FHE constraints, Concrete ML works with Brevitas, a library providing QAT support for PyTorch.

The following example uses a simple QAT PyTorch model that implements a fully connected neural network with two hidden layers. Due to its small size, making this model respect FHE constraints is relatively easy.

import brevitas.nn as qnn
import torch.nn as nn
import torch

N_FEAT = 12
n_bits = 3

class QATSimpleNet(nn.Module):
    def __init__(self, n_hidden):
        super().__init__()

        self.quant_inp = qnn.QuantIdentity(bit_width=n_bits, return_quant_tensor=True)
        self.fc1 = qnn.QuantLinear(N_FEAT, n_hidden, True, weight_bit_width=n_bits, bias_quant=None)
        self.quant2 = qnn.QuantIdentity(bit_width=n_bits, return_quant_tensor=True)
        self.fc2 = qnn.QuantLinear(n_hidden, n_hidden, True, weight_bit_width=n_bits, bias_quant=None)
        self.quant3 = qnn.QuantIdentity(bit_width=n_bits, return_quant_tensor=True)
        self.fc3 = qnn.QuantLinear(n_hidden, 2, True, weight_bit_width=n_bits, bias_quant=None)

    def forward(self, x):
        x = self.quant_inp(x)
        x = self.quant2(torch.relu(self.fc1(x)))
        x = self.quant3(torch.relu(self.fc2(x)))
        x = self.fc3(x)
        return x

Once the model is trained, calling the compile_brevitas_qat_model from Concrete ML will automatically perform conversion and compilation of a QAT network. Here, 3-bit quantization is used for both the weights and activations. The compile_brevitas_qat_model function automatically identifies the number of quantization bits used in the Brevitas model.

from concrete.ml.torch.compile import compile_brevitas_qat_model
import numpy

torch_input = torch.randn(100, N_FEAT)
torch_model = QATSimpleNet(30)
quantized_module = compile_brevitas_qat_model(
    torch_model, # our model
    torch_input, # a representative input-set to be used for both quantization and compilation
)

Configuring quantization parameters

The PyTorch/Brevitas models, created following the example above, require the user to configure quantization parameters such as bit_width (activation bit-width) and weight_bit_width. The quantization parameters, along with the number of neurons on each layer, will determine the accumulator bit-width of the network. Larger accumulator bit-widths result in higher accuracy but slower FHE inference time.

The following configurations were determined through experimentation for convolutional and dense layers.

Using the templates above, the probability of obtaining the target accumulator bit-width, for a single layer, was determined experimentally by training 10 models for each of the following data-sets.

Note that the accuracy on larger data-sets, when the accumulator size is low, is also reduced strongly.

Running encrypted inference

The model can now perform encrypted inference.

x_test = numpy.array([numpy.random.randn(N_FEAT)])

y_pred = quantized_module.forward(x_test, fhe="execute")

In this example, the input values x_test and the predicted values y_pred are floating points. The quantization (resp. de-quantization) step is done in the clear within the forward method, before (resp. after) any FHE computations.

Simulated FHE Inference in the clear

The user can also perform the inference on clear data. Two approaches exist:

• quantized_module.forward(quantized_x, fhe="simulate"): simulates FHE execution taking into account Table Lookup errors. De-quantization must be done in a second step as for actual FHE execution. Simulation takes into account the p_error/global_p_error parameters

• quantized_module.forward(quantized_x, fhe="disable"): computes predictions in the clear on quantized data, and then de-quantize the result. The return value of this function contains the de-quantized (float) output of running the model in the clear. Calling this function on clear data is useful when debugging, but this does not perform actual FHE simulation.

Generic Quantization Aware Training import

While the example above shows how to import a Brevitas/PyTorch model, Concrete ML also provides an option to import generic QAT models implemented in PyTorch or through ONNX. Deep learning models made with TensorFlow or Keras should be usable by preliminary converting them to ONNX.

QAT models contain quantizers in the PyTorch graph. These quantizers ensure that the inputs to the Linear/Dense and Conv layers are quantized.

Suppose that n_bits_qat is the bit-width of activations and weights during the QAT process. To import a PyTorch QAT network, you can use the compile_torch_model library function, passing import_qat=True:

from concrete.ml.torch.compile import compile_torch_model
n_bits_qat = 3

quantized_module = compile_torch_model(
    torch_model,
    torch_input,
    import_qat=True,
    n_bits=n_bits_qat,
)

Alternatively, if you want to import an ONNX model directly, please see the ONNX guide. The compile_onnx_model also supports the import_qat parameter.

Supported operators and activations

Concrete ML supports a variety of PyTorch operators that can be used to build fully connected or convolutional neural networks, with normalization and activation layers. Moreover, many element-wise operators are supported.

Operators

Univariate operators

• torch.nn.identity

• torch.clip

• torch.clamp

• torch.round

• torch.floor

• torch.min

• torch.max

• torch.abs

• torch.neg

• torch.sign

• torch.logical_or, torch.Tensor operator ||

• torch.logical_not

• torch.gt, torch.greater

• torch.ge, torch.greater_equal

• torch.lt, torch.less

• torch.le, torch.less_equal

• torch.eq

• torch.where

• torch.exp

• torch.log

• torch.pow

• torch.sum

• torch.mul, torch.Tensor operator *

• torch.div, torch.Tensor operator /

• torch.nn.BatchNorm2d

• torch.nn.BatchNorm3d

• torch.erf, torch.special.erf

• torch.nn.functional.pad

Shape modifying operators

• torch.reshape

• torch.Tensor.view

• torch.flatten

• torch.unsqueeze

• torch.squeeze

• torch.transpose

• torch.concat, torch.cat

• torch.nn.Unfold

Tensor operators

• torch.Tensor.expand

• torch.Tensor.to -- for casting to dtype

Multi-variate operators: encrypted input and unencrypted constants

• torch.nn.Linear

• torch.conv1d, torch.nn.Conv1D

• torch.conv2d, torch.nn.Conv2D

• torch.nn.AvgPool2d

• torch.nn.MaxPool2d

Concrete ML also supports some of their QAT equivalents from Brevitas.

• brevitas.nn.QuantLinear

• brevitas.nn.QuantConv1d

• brevitas.nn.QuantConv2d

Multi-variate operators: encrypted+unencrypted or encrypted+encrypted inputs

• torch.add, torch.Tensor operator +

• torch.sub, torch.Tensor operator -

• torch.matmul

Quantizers

• brevitas.nn.QuantIdentity

Activation functions

• torch.nn.CELU

• torch.nn.ELU

• torch.nn.GELU

• torch.nn.Hardshrink

• torch.nn.HardSigmoid

• torch.nn.Hardswish

• torch.nn.HardTanh

• torch.nn.LeakyReLU

• torch.nn.LogSigmoid

• torch.nn.Mish

• torch.nn.PReLU

• torch.nn.ReLU6

• torch.nn.ReLU

• torch.nn.SELU

• torch.nn.Sigmoid

• torch.nn.SiLU

• torch.nn.Softplus

• torch.nn.Softshrink

• torch.nn.Softsign

• torch.nn.Tanh

• torch.nn.Tanhshrink

• torch.nn.Threshold -- partial support

In addition to Concrete ML models and custom models in torch, it is also possible to directly compile ONNX models. This can be particularly appealing, notably to import models trained with Keras.

ONNX models can be compiled by directly importing models that are already quantized with Quantization Aware Training (QAT) or by performing Post-Training Quantization (PTQ) with Concrete ML.

Simple example

The following example shows how to compile an ONNX model using PTQ. The model was initially trained using Keras before being exported to ONNX. The training code is not shown here.

import numpy
import onnx
import tensorflow
import tf2onnx

from concrete.ml.torch.compile import compile_onnx_model
from concrete.fhe.compilation import Configuration


class FC(tensorflow.keras.Model):
    """A fully-connected model."""

    def __init__(self):
        super().__init__()
        hidden_layer_size = 10
        output_size = 5

        self.dense1 = tensorflow.keras.layers.Dense(
            hidden_layer_size,
            activation=tensorflow.nn.relu,
        )
        self.dense2 = tensorflow.keras.layers.Dense(output_size, activation=tensorflow.nn.relu6)
        self.flatten = tensorflow.keras.layers.Flatten()

    def call(self, inputs):
        """Forward function."""
        x = self.flatten(inputs)
        x = self.dense1(x)
        x = self.dense2(x)
        return self.flatten(x)


n_bits = 6
input_output_feature = 2
input_shape = (input_output_feature,)
num_inputs = 1
n_examples = 5000

# Define the Keras model
keras_model = FC()
keras_model.build((None,) + input_shape)
keras_model.compute_output_shape(input_shape=(None, input_output_feature))

# Create random input
input_set = numpy.random.uniform(-100, 100, size=(n_examples, *input_shape))

# Convert to ONNX
tf2onnx.convert.from_keras(keras_model, opset=14, output_path="tmp.model.onnx")

onnx_model = onnx.load("tmp.model.onnx")
onnx.checker.check_model(onnx_model)

# Compile
quantized_module = compile_onnx_model(
    onnx_model, input_set, n_bits=2
)

# Create test data from the same distribution and quantize using
# learned quantization parameters during compilation
x_test = tuple(numpy.random.uniform(-100, 100, size=(1, *input_shape)) for _ in range(num_inputs))

y_clear = quantized_module.forward(*x_test, fhe="disable")
y_fhe = quantized_module.forward(*x_test, fhe="execute")

print("Execution in clear: ", y_clear)
print("Execution in FHE:   ", y_fhe)
print("Equality:           ", numpy.sum(y_clear == y_fhe), "over", numpy.size(y_fhe), "values")

Quantization Aware Training

Models trained using Quantization Aware Training contain quantizers in the ONNX graph. These quantizers ensure that the inputs to the Linear/Dense and Conv layers are quantized. Since these QAT models have quantizers that are configured during training to a specific number of bits, the ONNX graph will need to be imported using the same settings:

# Define the number of bits to use for quantizing weights and activations during training
n_bits_qat = 3  

quantized_numpy_module = compile_onnx_model(
    onnx_model,
    input_set,
    import_qat=True,
    n_bits=n_bits_qat,
)

Supported operators

The following operators are supported for evaluation and conversion to an equivalent FHE circuit. Other operators were not implemented, either due to FHE constraints or because they are rarely used in PyTorch activations or scikit-learn models.

• Abs

• Acos

• Acosh

• Add

• Asin

• Asinh

• Atan

• Atanh

• AveragePool

• BatchNormalization

• Cast

• Celu

• Clip

• Concat

• Constant

• ConstantOfShape

• Conv

• Cos

• Cosh

• Div

• Elu

• Equal

• Erf

• Exp

• Expand

• Flatten

• Floor

• Gather

• Gemm

• Greater

• GreaterOrEqual

• HardSigmoid

• HardSwish

• Identity

• LeakyRelu

• Less

• LessOrEqual

• Log

• MatMul

• Max

• MaxPool

• Min

• Mul

• Neg

• Not

• Or

• PRelu

• Pad

• Pow

• ReduceSum

• Relu

• Reshape

• Round

• Selu

• Shape

• Sigmoid

• Sign

• Sin

• Sinh

• Slice

• Softplus

• Squeeze

• Sub

• Tan

• Tanh

• ThresholdedRelu

• Transpose

• Unfold

• Unsqueeze

• Where

• onnx.brevitas.Quant

This guide provides a complete example of converting a PyTorch neural network into its FHE-friendly, quantized counterpart. It focuses on Quantization Aware Training a simple network on a synthetic data-set.

In general, quantization can be carried out in two different ways: either during Quantization Aware Training (QAT) or after the training phase with Post-Training Quantization (PTQ).

Regarding FHE-friendly neural networks, QAT is the best way to reach optimal accuracy under FHE constraints. This technique allows weights and activations to be reduced to very low bit-widths (e.g., 2-3 bits), which, combined with pruning, can keep accumulator bit-widths low.

Concrete ML uses the third-party library Brevitas to perform QAT for PyTorch NNs, but options exist for other frameworks such as Keras/Tensorflow.

Several demos and tutorials that use Brevitas are available in the Concrete ML library, such as the CIFAR classification tutorial.

This guide is based on a notebook tutorial, from which some code blocks are documented.

For a more formal description of the usage of Brevitas to build FHE-compatible neural networks, please see the Brevitas usage reference.

Baseline PyTorch model

In PyTorch, using standard layers, a fully connected neural network (FCNN) would look like this:

import torch
from torch import nn

IN_FEAT = 2
OUT_FEAT = 2

class SimpleNet(nn.Module):
    """Simple MLP with PyTorch"""

    def __init__(self, n_hidden = 30):
        super().__init__()
        self.fc1 = nn.Linear(in_features=IN_FEAT, out_features=n_hidden)
        self.fc2 = nn.Linear(in_features=n_hidden, out_features=n_hidden)
        self.fc3 = nn.Linear(in_features=n_hidden, out_features=OUT_FEAT)


    def forward(self, x):
        """Forward pass."""
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = self.fc3(x)
        return x

The notebook tutorial, example shows how to train a FCNN, similarly to the one above, on a synthetic 2D data-set with a checkerboard grid pattern of 100 x 100 points. The data is split into 9500 training and 500 test samples.

Once trained, this PyTorch network can be imported using the compile_torch_model function. This function uses simple PTQ.

The network was trained using different numbers of neurons in the hidden layers, and quantized using 3-bits weights and activations. The mean accumulator size shown below is measured as the mean over 10 runs of the experiment. An accumulator of 6.6 means that 4 times out of 10 the accumulator measured was 6 bits while 6 times it was 7 bits.

This shows that the fp32 accuracy and accumulator size increases with the number of hidden neurons, while the 3-bits accuracy remains low irrespective of the number of neurons. While all the configurations tried here were FHE-compatible (accumulator < 16 bits), it is often preferable to have a lower accumulator size in order to speed up inference time.

Quantization Aware Training:

Quantization Aware Training using Brevitas is the best way to guarantee a good accuracy for Concrete ML compatible neural networks.

Brevitas provides a quantized version of almost all PyTorch layers (Linear layer becomes QuantLinear, ReLU layer becomes QuantReLU and so on), plus some extra quantization parameters, such as :

• bit_width: precision quantization bits for activations

• act_quant: quantization protocol for the activations

• weight_bit_width: precision quantization bits for weights

• weight_quant: quantization protocol for the weights

In order to use FHE, the network must be quantized from end to end, and thanks to the Brevitas's QuantIdentity layer, it is possible to quantize the input by placing it at the entry point of the network. Moreover, it is also possible to combine PyTorch and Brevitas layers, provided that a QuantIdentity is placed after this PyTorch layer. The following table gives the replacements to be made to convert a PyTorch NN for Concrete ML compatibility.

Some PyTorch operators (from the PyTorch functional API), require a brevitas.quant.QuantIdentity to be applied on their inputs.

With Brevitas, the network above becomes:

from brevitas import nn as qnn
from brevitas.core.quant import QuantType
from brevitas.quant import Int8ActPerTensorFloat, Int8WeightPerTensorFloat

N_BITS = 3
IN_FEAT = 2
OUT_FEAT = 2

class QuantSimpleNet(nn.Module):
    def __init__(
        self,
        n_hidden,
        qlinear_args={
            "weight_bit_width": N_BITS,
            "weight_quant": Int8WeightPerTensorFloat,
            "bias": True,
            "bias_quant": None,
            "narrow_range": True
        },
        qidentity_args={"bit_width": N_BITS, "act_quant": Int8ActPerTensorFloat},
    ):
        super().__init__()

        self.quant_inp = qnn.QuantIdentity(**qidentity_args)
        self.fc1 = qnn.QuantLinear(IN_FEAT, n_hidden, **qlinear_args)
        self.relu1 = qnn.QuantReLU(bit_width=qidentity_args["bit_width"])
        self.fc2 = qnn.QuantLinear(n_hidden, n_hidden, **qlinear_args)
        self.relu2 = qnn.QuantReLU(bit_width=qidentity_args["bit_width"])
        self.fc3 = qnn.QuantLinear(n_hidden, OUT_FEAT, **qlinear_args)

        for m in self.modules():
            if isinstance(m, qnn.QuantLinear):
                torch.nn.init.uniform_(m.weight.data, -1, 1)

    def forward(self, x):
        x = self.quant_inp(x)
        x = self.relu1(self.fc1(x))
        x = self.relu2(self.fc2(x))
        x = self.fc3(x)
        return x       

Training this network with pruning (see below) with 30 out of 100 total non-zero neurons gives good accuracy while keeping the accumulator size low.

Pruning using Torch

Considering that FHE only works with limited integer precision, there is a risk of overflowing in the accumulator, which will make Concrete ML raise an error.

This fact can be leveraged to train a network with more neurons, while not overflowing the accumulator, using a technique called pruning where the developer can impose a number of zero-valued weights. Torch provides support for pruning out of the box.

The following code shows how to use pruning in the previous example:

import torch.nn.utils.prune as prune

class PrunedQuantNet(SimpleNet):
    """Simple MLP with PyTorch"""

    pruned_layers = set()

    def prune(self, max_non_zero):
        # Linear layer weight has dimensions NumOutputs x NumInputs
        for name, layer in self.named_modules():
            if isinstance(layer, nn.Linear):
                print(name, layer)
                num_zero_weights = (layer.weight.shape[1] - max_non_zero) * layer.weight.shape[0]
                if num_zero_weights <= 0:
                    continue
                print(f"Pruning layer {name} factor {num_zero_weights}")
                prune.l1_unstructured(layer, "weight", amount=num_zero_weights)
                self.pruned_layers.add(name)

    def unprune(self):
        for name, layer in self.named_modules():
            if name in self.pruned_layers:
                prune.remove(layer, "weight")
                self.pruned_layers.remove(name)

Results with PrunedQuantNet, a pruned version of the QuantSimpleNet with 100 neurons on the hidden layers, are given below, showing a mean accumulator size measured over 10 runs of the experiment:

This shows that the fp32 accuracy has been improved while maintaining constant mean accumulator size.

When pruning a larger neural network during training, it is easier to obtain a low bit-width accumulator while maintaining better final accuracy. Thus, pruning is more robust than training a similar, smaller network.

Concrete ML offers the possibility to train SGD Logistic Regression on encrypted data. The logistic regression training example shows this feature in action.

This example shows how to instantiate a logistic regression model that trains on encrypted data:

from concrete.ml.sklearn import SGDClassifier
parameters_range = (-1.0, 1.0)

model = SGDClassifier(
    random_state=42,
    max_iter=50,
    fit_encrypted=True,
    parameters_range=parameters_range,
)

To activate encrypted training simply set fit_encrypted=True in the constructor. If this value is not set, training is performed on clear data using scikit-learn gradient descent.

Next, to perform the training on encrypted data, call the fit function with the fhe="execute" argument:

model.fit(X_binary, y_binary, fhe="execute")

Training configuration

The max_iter parameter controls the number of batches that are processed by the training algorithm.

The parameters_range parameter determines the initialization of the coefficients and the bias of the logistic regression. It is recommended to give values that are close to the min/max of the training data. It is also possible to normalize the training data so that it lies in the range $[-1, 1]$.

Capabilities and Limitations

The logistic model that can be trained uses Stochastic Gradient Descent (SGD) and quantizes for data, weights, gradients and the error measure. It currently supports training 6-bit models, training both the coefficients and the bias.

The SGDClassifier does not currently support training models with other values for the bit-widths. The execution time to train a model is proportional to the number of features and the number of training examples in the batch. The SGDClassifier training does not currently support client/server deployment for training.

Concrete ML offers nearest neighbors non-parametric classification models with a scikit-learn interface through the KNeighborsClassifier class.

Example usage

from concrete.ml.sklearn import KNeighborsClassifier

concrete_classifier = KNeighborsClassifier(n_bits=2, n_neighbors=3)

The KNeighborsClassifier class quantizes the training data-set that is given to .fit with the specified number of bits, n_bits. As this value must be kept low to comply with accumulator size constraints the accuracy of the model will depend heavily a well-chosen value n_bits and the dimensionality of the data.

The predict method of the KNeighborsClassifier performs the following steps:

• quantization of the test vectors, performed in the clear

• computation of the top-k class indices of the closest training set vector, on encrypted data

• majority vote of the top-k class labels to find the class for each test vector, performed in the clear

Inference time considerations

The FHE inference latency of this model is heavily influenced by the n_bits, the dimensionality of the data. Furthermore, the size of the data-set has a linear impact on the complexity of the data and the number of nearest neighbors, n_neighbors, also plays a role.

The KNN computation executes in FHE in $O(Nlog^2k)$ steps, where $N$ is the training data-set size and $k$ is n_neighbors. Each step requires several PBS, but the run-time of each of these PBS is influenced by the factors listed above. These factors combine to give the precision required to represent the distances between test vectors and the training data-set vectors. The PBS input precision required by the circuit is related to the one of the distance values.

Concrete ML builds upon the pandas data-frame functionality by introducing the capability to construct and perform operations on encrypted data-frames using FHE. This API ensures data scientists can leverage well-known pandas-like operations while maintaining privacy throughout the whole process.

Encrypted data-frames are a storage format for encrypted tabular data and they can be exchanged with third-parties without security risks.

Potential applications include:

• Encrypted storage of tabular datasets

• Joint data analysis efforts between multiple parties

• Data preparation steps before machine learning tasks, such as inference or training

• Secure outsourcing of data analysis to untrusted third parties

Encrypt and Decrypt a DataFrame

To encrypt a pandas DataFrame, you must construct a ClientEngine which manages keys. Then call the encrypt_from_pandas function:

Supported Data Types and Schema Definition

Concrete ML's encrypted DataFrame operations support a specific set of data types:

• Integer: Integers are supported within a specific range determined by the encryption scheme's quantization parameters. Default range is 1 to 15. 0 being used for the NaN. Values outside this range will cause a ValueError to be raised during the pre-processing stage.

• Quantized Float: Floating-point numbers are quantized to integers within the supported range. This is achieved by computing a scale and zero point for each column, which are used to map the floating-point numbers to the quantized integer space.

• String Enum: String columns are mapped to integers starting from 1. This mapping is stored and later used for de-quantization. If the number of unique strings exceeds 15, a ValueError is raised.

Supported Operations on Encrypted Data-frames

Outsourced execution: The merge operation on Encrypted DataFrames can be securely performed on a third-party server. This means that the server can execute the merge without ever having access to the unencrypted data. The server only requires the encrypted DataFrames.

Encrypted DataFrames support a subset of operations that are available for pandas DataFrames. The following operations are currently supported:

• merge: left or right join two data-frames

Serialization of Encrypted Data-frames

Security: Serialized data-frames do not contain any secret keys. The data-frames can be exchanged with any third-party without any risk.

Saving and loading Data-frames

To save or load an encrypted DataFrame from a file, use the following commands:

Error Handling

The library is designed to raise specific errors when encountering issues during the pre-processing and post-processing stages:

• ValueError: Raised when a column contains values outside the allowed range for integers, when there are too many unique strings, or when encountering an unsupported data type. Raised also when an operation is attempted on a data type that is not supported by the operation.

Example Workflow

Current Limitations

While this API offers a new secure way to work on remotely stored and encrypted data, it has some strong limitations at the moment:

• Precision of Values: The precision for numerical values is limited to 4 bits.

• Supported Operations: The merge operation is the only one available.

• Index Handling: Index values are not preserved; users should move any relevant data from the index to a dedicated new column before encrypting.

• Integer Range: The range of integers that can be encrypted is between 1 and 15.

• Uniqueness for merge: The merge operation requires that the columns to merge on contain unique values. Currently this means that data-frames are limited to 15 rows.

• Metadata Security: Column names and the mapping of strings to integers are not encrypted and are sent to the server in clear text.

This section provides a set of tools and guidelines to help users build optimized FHE-compatible models. It discusses FHE simulation, the key-cache functionality that helps speed-up FHE result debugging, and gives a guide to evaluate circuit complexity.

Simulation

The of Concrete ML provides a way to evaluate, using clear data, the results that ML models produce on encrypted data. The simulation includes any probabilistic behavior FHE may induce. The simulation is implemented with .

The simulation mode can be useful when developing and iterating on an ML model implementation. As FHE non-linear models work with integers up to 16 bits, with a trade-off between the number of bits and the FHE execution speed, the simulation can help to find the optimal model design.

Simulation is much faster than FHE execution. This allows for faster debugging and model optimization. For example, this was used for the red/blue contours in the , as computing in FHE for the whole grid and all the classifiers would take significant time.

The following example shows how to use the simulation mode in Concrete ML.

Caching keys during debugging

It is possible to avoid re-generating the keys of the models you are debugging. This feature is unsafe and should not be used in production. Here is an example that shows how to enable key-caching:

Compilation error debugging

Compilation errors that signal that the ML model is not FHE compatible are usually of two types:

1. TLU input maximum bit-width is exceeded

2. No crypto-parameters can be found for the ML model: RuntimeError: NoParametersFound is raised by the compiler

The following produces a neural network that is not FHE-compatible:

Upon execution, the Compiler will raise the following error within the graph representation:

Fixing compilation errors

To make this network FHE-compatible one can apply several techniques:

1. reduce the accumulator bit-width of the second layer named fc2. To do this, a simple solution is to reduce the number of neurons, as it is proportional to the bit-width.

Complexity analysis

In FHE, univariate functions are encoded as table lookups, which are then implemented using Programmable Bootstrapping (PBS). PBS is a powerful technique but will require significantly more computing resources, and thus time, compared to simpler encrypted operations such as matrix multiplications, convolution, or additions.

Furthermore, the cost of PBS will depend on the bit-width of the compiled circuit. Every additional bit in the maximum bit-width raises the complexity of the PBS by a significant factor. It may be of interest to the model developer, then, to determine the bit-width of the circuit and the amount of PBS it performs.

This can be done by inspecting the MLIR code produced by the Compiler:

There are several calls to FHELinalg.apply_mapped_lookup_table and FHELinalg.apply_lookup_table. These calls apply PBS to the cells of their input tensors. Their inputs in the listing above are: tensor<1x2x!FHE.eint<8>> for the first and last call and tensor<1x50x!FHE.eint<8>> for the two calls in the middle. Thus, PBS is applied 104 times.

Retrieving the bit-width of the circuit is then simply:

Decreasing the number of bits and the number of PBS applications induces large reductions in the computation time of the compiled circuit.

Neural networks pose unique challenges with regards to encrypted inference. Each neuron in a network applies an activation function that requires a PBS operation. The latency of a single PBS depends on the bit-width of the input of the PBS.

Several approaches can be used to reduce the overall latency of a neural network.

Circuit bit-width optimization

and introduce specific hyper-parameters that influence the accumulator sizes. It is possible to chose quantization and pruning configurations that reduce the accumulator size. A trade-off between latency and accuracy can be obtained by varying these hyper-parameters as described in the .

Structured pruning

While un-structured pruning is used to ensure the accumulator bit-width stays low, can eliminate entire neurons from the network. Many neural networks are over-parametrized (since this enables easier training) and some neurons can be removed. Structured pruning, applied to a trained network as a fine-tuning step, can be applied to built-in neural networks using the helper function as shown in . To apply structured pruning to custom models, it is recommended to use the package.

Rounded activations and quantizers

Reducing the bit-width of the inputs to the Table Lookup (TLU) operations is a major source of improvements in the latency. Post-training, it is possible to leverage some properties of the fused activation and quantization functions expressed in the TLUs to further reduce the accumulator. This is achieved through the rounded PBS feature as described in the . Adjusting the rounding amount, relative to the initial accumulator size, can bring large improvements in latency while maintaining accuracy.

TLU error tolerance adjustment

Finally, the TFHE scheme exposes a TLU error tolerance parameter that has an impact on crypto-system parameters that influence latency. A higher tolerance of TLU off-by-one errors results in faster computations but may reduce accuracy. One can think of the error of obtaining as a Gaussian distribution centered on : is obtained with probability of 1 - p_error, while , are obtained with much lower probability, etc. In Deep NNs, these type of errors can be tolerated up to some point. See the and more specifically the usage example of .

FHE enables cloud applications to process private user data without running the risk of data leaks. Furthermore, deploying ML models in the cloud is advantageous as it eases model updates, allows to scale to large numbers of users by using large amounts of compute power, and protects model IP by keeping the model on a trusted server instead of the client device.

However, not all applications can be easily converted to FHE computation and the computation cost of FHE may make a full conversion exceed latency requirements.

Hybrid models provide a balance between on-device deployment and cloud-based deployment. This approach entails executing parts of the model directly on the client side, while other parts are securely processed with FHE on the server side. Concrete ML facilitates the hybrid deployment of various neural network models, including MLP (multilayer perceptron), CNN (convolutional neural network), and Large Language Models.

The hybrid model deployment API provides an easy way to integrate the into neural network style models that are compiled with or .

Compilation

To use hybrid model deployment, the first step is to define what part of the PyTorch neural network model must be executed in FHE. The model part must be a nn.Module and is identified by its key in the original model's .named_modules().

Server Side Deployment

Client Side

A client application that deploys a model with hybrid deployment can be developed in a very similar manner to on-premise deployment: the model is loaded normally with PyTorch, but an extra step is required to specify the remote endpoint and the model parts that are to be executed remotely.

When the client application is ready to make inference requests to the server, it must set the operation mode of the HybridFHEModel instance to HybridFHEMode.REMOTE:

When performing inference with the HybridFHEModel instance, hybrid_model, only the regular forward method is called, as if the model was fully deployed locally:

When calling forward, the HybridFHEModel handles, for each model part that is deployed remotely, all the necessary intermediate steps: quantizing the data, encrypting it, makes the request to the server using requests Python module, decrypting and de-quantizing the result.

Concrete ML provides functionality to deploy FHE machine learning models in a client/server setting. The deployment workflow and model serving pattern is as follows:

Deployment

The diagram above shows the steps that a developer goes through to prepare a model for encrypted inference in a client/server setting. The training of the model and its compilation to FHE are performed on a development machine. Three different files are created when saving the model:

• client.zip contains client.specs.json which lists the secure cryptographic parameters needed for the client to generate private and evaluation keys. It also contains serialized_processing.json which describes the pre-processing and post-processing required by the machine learning model, such as quantization parameters to quantize the input and de-quantize the output.

• server.zip contains the compiled model. This file is sufficient to run the model on a server. The compiled model is machine-architecture specific (i.e., a model compiled on x86 cannot run on ARM).

The compiled model (server.zip) is deployed to a server and the cryptographic parameters (client.zip) are shared with the clients. In some settings, such as a phone application, the client.zip can be directly deployed on the client device and the server does not need to host it.

Note that for built-in models, the server output + post-processing adheres to the following guidelines: if the model is a regressor, the output follows the format of the scikit-learn .predict() method; if the model is a classifier, the output follows the format of the scikit-learn .predict_proba() method.

Serving

The client-side deployment of a secured inference machine learning model follows the schema above. First, the client obtains the cryptographic parameters (stored in client.zip) and generates a private encryption/decryption key as well as a set of public evaluation keys. The public evaluation keys are then sent to the server, while the secret key remains on the client.

The private data is then encrypted by the client as described in the serialized_processing.json file in client.zip, and it is then sent to the server. Server-side, the FHE model inference is run on encrypted inputs using the public evaluation keys.

The encrypted result is then returned by the server to the client, which decrypts it using its private key. Finally, the client performs any necessary post-processing of the decrypted result as specified in serialized_processing.json (part of client.zip).

The server-side implementation of a Concrete ML model follows the diagram above. The public evaluation keys sent by clients are stored. They are then retrieved for the client that is querying the service and used to evaluate the machine learning model stored in server.zip. Finally, the server sends the encrypted result of the computation back to the client.

Example notebook

AWS

Once this first setup is done you can launch python src/concrete/ml/deployment/deploy_to_aws.py --path-to-model <path_to_your_serialized_model> from the root of the repository to create an instance that runs a FastAPI server serving the model.

Docker

Running Docker with the latest version of Concrete ML will require you to build a Docker image. To do this, run the following command: poetry build && mkdir pkg && cp dist/* pkg/ && make release_docker. You will need to have make, poetry and docker installed on your system. To test locally there is a dedicated script: python src/concrete/ml/deployment/deploy_to_docker.py --path-to-model <path_to_your_serialized_model> whoch should be run from the root of the repository in order to create a Docker that runs a FastAPI server serving the model.

Quantization is the process of constraining an input from a continuous or otherwise large set of values (such as real numbers) to a discrete set (such as integers).

This means that some accuracy in the representation is lost (e.g., a simple approach is to eliminate least-significant bits). In many cases in machine learning, it is possible to adapt the models to give meaningful results while using these smaller data types. This significantly reduces the number of bits necessary for intermediary results during the execution of these machine learning models.

Since FHE is currently limited to 16-bit integers, it is necessary to quantize models to make them compatible. As a general rule, the smaller the bit-width of integer values used in models, the better the FHE performance. This trade-off should be taken into account when designing models, especially neural networks.

Overview of quantization in Concrete ML

Quantization implemented in Concrete ML is applied in two ways:

1. Built-in models apply quantization internally and the user only needs to configure some quantization parameters. This approach requires little work by the user but may not be a one-size-fits-all solution for all types of models. The final quantized model is FHE-friendly and ready to predict over encrypted data. In this setting, Post-Training Quantization (PTQ) is used for linear models, data quantization is used for tree-based models and, finally, Quantization Aware Training (QAT) is included in the built-in neural network models.

2. For custom neural networks with more complex topology, obtaining FHE-compatible models with good accuracy requires QAT. Concrete ML offers the possibility for the user to perform quantization before compiling to FHE. This can be achieved through a third-party library that offers QAT tools, such as Brevitas for PyTorch. In this approach, the user is responsible for implementing a full-integer model, respecting FHE constraints. Please refer to the advanced QAT tutorial for tips on designing FHE neural networks.

Basics of quantization

Let $[\alpha, \beta ]$ be the range of a value to quantize where $\alpha$ is the minimum and $\beta$ is the maximum. To quantize a range of floating point values (in $\mathbb{R}$) to integer values (in $\mathbb{Z}$), the first step is to choose the data type that is going to be used. Many ML models work with weights and activations represented as 8-bit integers, so this will be the value used in this example. Knowing the number of bits that can be used for a value in the range $[\alpha, \beta ]$, the scale $S$ can be computed :

$S = \frac{\beta - \alpha}{2^n - 1}$

where $n$ is the number of bits ($n \leq 8$). In the following, $n = 8$ is assumed.

In practice, the quantization scale is then $S = \frac{\beta - \alpha}{255}$. This means the gap between consecutive representable values cannot be smaller than $S$, which, in turn, means there can be a substantial loss of precision. Every interval of length $S$ will be represented by a value within the range $[0..255]$.

The other important parameter from this quantization schema is the zero point $Z_p$ value. This essentially brings the 0 floating point value to a specific integer. If the quantization scheme is asymmetric (quantized values are not centered in 0), the resulting $Z_p$ will be in $\mathbb{Z}$.

$Z_p = \mathtt{round} \left(- \frac{\alpha}{S} \right)$

When using quantized values in a matrix multiplication or convolution, the equations for computing the result become more complex. The IntelLabs Distiller documentation provides a more detailed explanation of the maths used to quantize values and how to keep computations consistent.

Quantization special cases

Machine learning acceleration solutions are often based on integer computation of activations. To make quantization computations hardware-friendly, a popular approach is to ensure that scales are powers-of-two, which allows the replacement of the division in the equations above with a shift-right operation. TFHE also has a fast primitive for right bit-shift that enables acceleration in the special case of power-of-two scales.

Configuring model quantization parameters

Built-in models provide a simple interface for configuring quantization parameters, most notably the number of bits used for inputs, model weights, intermediary values, and output values.

For linear models, the quantization is done post-training. Thus, the model is trained in floating point, and then, the best integer weight representations are found, depending on the distribution of inputs and weights. For these models, the user selects the value of the n_bits parameter.

For linear models, n_bits is used to quantize both model inputs and weights. Depending on the number of features, you can use a single integer value for the n_bits parameter (e.g., a value between 2 and 7). When the number of features is high, the n_bits parameter should be decreased if you encounter compilation errors. It is also possible to quantize inputs and weights with different numbers of bits by passing a dictionary to n_bits containing the op_inputs and op_weights keys.

For tree-based models, the training and test data is quantized. The maximum accumulator bit-width for a model trained with n_bits=n for this type of model is known beforehand: It will need n+1 bits. Through experimentation, it was determined that, in many cases, a value of 5 or 6 bits gives the same accuracy as training in floating point and values above n=7 do not increase model performance (but rather induce a strong slowdown).

For built-in neural networks, several linear layers are used. Thus, the outputs of a layer are used as inputs to a new layer. Built-in neural networks use Quantization Aware Training. The parameters controlling the maximum accumulator bit-width are the number of weights and activation bits ( module__n_w_bits, module__n_a_bits ), but also the pruning factor. This factor is determined automatically by specifying a desired accumulator bit-width module__n_accum_bits and, optionally, a multiplier factor, module__n_hidden_neurons_multiplier.

Quantizing model inputs and outputs

The models implemented in Concrete ML provide features to let the user quantize the input data and de-quantize the output data.

In a client/server setting, the client is responsible for quantizing inputs before sending them, encrypted, to the server. The client must then de-quantize the encrypted integer results received from the server. See the Production Deployment section for more details.

Here is a simple example showing how to perform inference, starting from float values and ending up with float values. The FHE engine that is compiled for ML models does not support data batching.

# Assume 
#   quantized_module : QuantizedModule
#   x: numpy.ndarray (of float)

# Quantization is done in the clear
x_q = quantized_module.quantize_input(x)

# Forward in FHE (here with simulation)
q_y_proba = quantized_module.quantized_forward(x_q, fhe="simulate")

# De-quantization is done in the clear
y_proba = quantized_module.dequantize_output(q_y_proba)

# For classifiers with multi-class outputs, the arg max is done in the clear
y_pred = np.argmax(y_proba, 1)

Alternatively, the forward method groups the quantization, FHE execution and de-quantization steps all together.

# Assume 
#   quantized_module : QuantizedModule
#   x: numpy.ndarray (of float)

# Forward in FHE (here with simulation). Quantization and de-quantization steps are still done in 
# the clear 
y_proba = quantized_module.forward(x, fhe="simulate")

# For classifiers with multi-class outputs, the arg max is done in the clear
y_pred = np.argmax(y_proba, 1)

Resources

• IntelLabs distiller explanation of quantization: Distiller documentation

Start here

• Build-in model examples

• Deep learning examples

Go further

Live demos on Hugging Face:

• Credit card approval: Predicting credit scoring card approval application in which sensitive data can be shared and analyzed without exposing the actual information to neither the three parties involved, nor the server processing it.

  • Check the code here

• Sentiment analysis with transformers: predicting if an encrypted tweet / short message is positive, negative or neutral, using FHE.

  • Check the code here and the blog post

• Health diagnosis: giving a diagnosis using FHE to preserve the privacy of the patient based on a patient's symptoms, history and other health factors.

  • Check the code here

• Encrypted image filtering: filtering encrypted images by applying filters such as black-and-white, ridge detection, or your own filter.

  • Check the code here

Code examples on Github:

• GPT-2 in FHE: Privacy-preserving text generation based on a user's prompt

• Titanic: Train an XGB classifier that can perform encrypted prediction for the Kaggle Titanic competition

• Federated learning and private inference: Use federated learning to train a Logistic Regression while preserving training data confidentiality. Import the model into Concrete ML and perform encrypted prediction

• Neutral network fine-tuning: Fine-tune a VGG network to classify the CIFAR image data-sets and predict on encrypted data

• Encrypted sentiment analysis:A Hugging Face space that securely analyzes the sentiment expressed in a short text

• Credit scoring: Predict the chance of a given loan applicant defaulting on loan repayment

Blog tutorials:

• Build an end-to-end encrypted Shazam application using Concrete ML - February 2024

• Linear regression over encrypted data with homomorphic encryption - June 2023

• Comparison of Concrete ML regressors - June 2023

• How to deploy a machine learning model with Concrete ML - May 2023

• Encrypted image filtering using homomorphic encryption - February 2023

• Sentiment analysis over encrypted data - November 2022

• Titanic Competition with Privacy Preserving Machine Learning - August 2022

Video tutorials

• Train a linear classifier on encrypted data using Concrete ML and Fully Homomorphic Encryption (FHE) - February 2024

• How to convert a scikit-learn model into its homomorphic equivalent - June 2023

Modules

• concrete.ml.common: Module for shared data structures and code.

• concrete.ml.common.check_inputs: Check and conversion tools.

• concrete.ml.common.debugging: Module for debugging.

• concrete.ml.common.debugging.custom_assert: Provide some variants of assert.

• concrete.ml.common.serialization: Serialization module.

• concrete.ml.common.serialization.decoder: Custom decoder for serialization.

• concrete.ml.common.serialization.dumpers: Dump functions for serialization.

• concrete.ml.common.serialization.encoder: Custom encoder for serialization.

• concrete.ml.common.serialization.loaders: Load functions for serialization.

• concrete.ml.common.utils: Utils that can be re-used by other pieces of code in the module.

• concrete.ml.deployment: Module for deployment of the FHE model.

• concrete.ml.deployment.deploy_to_aws: Methods to deploy a client/server to AWS.

• concrete.ml.deployment.deploy_to_docker: Methods to deploy a server using Docker.

• concrete.ml.deployment.fhe_client_server: APIs for FHE deployment.

• concrete.ml.deployment.server: Deployment server.

• concrete.ml.deployment.utils: Utils.

• concrete.ml.onnx: ONNX module.

• concrete.ml.onnx.convert: ONNX conversion related code.

• concrete.ml.onnx.onnx_impl_utils: Utility functions for onnx operator implementations.

• concrete.ml.onnx.onnx_model_manipulations: Some code to manipulate models.

• concrete.ml.onnx.onnx_utils: Utils to interpret an ONNX model with numpy.

• concrete.ml.onnx.ops_impl: ONNX ops implementation in Python + NumPy.

• concrete.ml.pandas: Public API for encrypted data-frames.

• concrete.ml.pandas.client_engine: Define the framework used for managing keys (encrypt, decrypt) for encrypted data-frames.

• concrete.ml.pandas.dataframe: Define the encrypted data-frame framework.

• concrete.ml.pytest: Module which is used to contain common functions for pytest.

• concrete.ml.pytest.torch_models: Torch modules for our pytests.

• concrete.ml.pytest.utils: Common functions or lists for test files, which can't be put in fixtures.

• concrete.ml.quantization: Modules for quantization.

• concrete.ml.quantization.base_quantized_op: Base Quantized Op class that implements quantization for a float numpy op.

• concrete.ml.quantization.post_training: Post Training Quantization methods.

• concrete.ml.quantization.quantized_module: QuantizedModule API.

• concrete.ml.quantization.quantized_module_passes: Optimization passes for QuantizedModules.

• concrete.ml.quantization.quantized_ops: Quantized versions of the ONNX operators for post training quantization.

• concrete.ml.quantization.quantizers: Quantization utilities for a numpy array/tensor.

• concrete.ml.search_parameters: Modules for p_error search.

• concrete.ml.search_parameters.p_error_search: p_error binary search for classification and regression tasks.

• concrete.ml.sklearn: Import sklearn models.

• concrete.ml.sklearn.base: Base classes for all estimators.

• concrete.ml.sklearn.glm: Implement sklearn's Generalized Linear Models (GLM).

• concrete.ml.sklearn.linear_model: Implement sklearn linear model.

• concrete.ml.sklearn.neighbors: Implement sklearn neighbors model.

• concrete.ml.sklearn.qnn: Scikit-learn interface for fully-connected quantized neural networks.

• concrete.ml.sklearn.qnn_module: Sparse Quantized Neural Network torch module.

• concrete.ml.sklearn.rf: Implement RandomForest models.

• concrete.ml.sklearn.svm: Implement Support Vector Machine.

• concrete.ml.sklearn.tree: Implement DecisionTree models.

• concrete.ml.sklearn.tree_to_numpy: Implements the conversion of a tree model to a numpy function.

• concrete.ml.sklearn.xgb: Implements XGBoost models.

• concrete.ml.torch: Modules for torch to numpy conversion.

• concrete.ml.torch.compile: torch compilation function.

• concrete.ml.torch.hybrid_model: Implement the conversion of a torch model to a hybrid fhe/torch inference.

• concrete.ml.torch.numpy_module: A torch to numpy module.

• concrete.ml.version: File to manage the version of the package.

Classes

• decoder.ConcreteDecoder: Custom json decoder to handle non-native types found in serialized Concrete ML objects.

• encoder.ConcreteEncoder: Custom json encoder to handle non-native types found in serialized Concrete ML objects.

• utils.FheMode: Enum representing the execution mode.

• deploy_to_aws.AWSInstance: AWSInstance.

• fhe_client_server.FHEModelClient: Client API to encrypt and decrypt FHE data.

• fhe_client_server.FHEModelDev: Dev API to save the model and then load and run the FHE circuit.

• fhe_client_server.FHEModelServer: Server API to load and run the FHE circuit.

• ops_impl.ONNXMixedFunction: A mixed quantized-raw valued onnx function.

• ops_impl.RawOpOutput: Type construct that marks an ndarray as a raw output of a quantized op.

• client_engine.ClientEngine: Define a framework that manages keys.

• dataframe.EncryptedDataFrame: Define an encrypted data-frame framework that supports Pandas operators and parameters.

• torch_models.AddNet: Torch model that performs a simple addition between two inputs.

• torch_models.BranchingGemmModule: Torch model with some branching and skip connections.

• torch_models.BranchingModule: Torch model with some branching and skip connections.

• torch_models.CNN: Torch CNN model for the tests.

• torch_models.CNNGrouped: Torch CNN model with grouped convolution for compile torch tests.

• torch_models.CNNInvalid: Torch CNN model for the tests.

• torch_models.CNNMaxPool: Torch CNN model for the tests with a max pool.

• torch_models.CNNOther: Torch CNN model for the tests.

• torch_models.ConcatFancyIndexing: Concat with fancy indexing.

• torch_models.Conv1dModel: Small model that uses a 1D convolution operator.

• torch_models.DoubleQuantQATMixNet: Torch model that with two different quantizers on the input.

• torch_models.EncryptedMatrixMultiplicationModel: PyTorch module for performing matrix multiplication between two encrypted values.

• torch_models.ExpandModel: Minimalist network that expands the input tensor to a larger size.

• torch_models.FC: Torch model for the tests.

• torch_models.FCSeq: Torch model that should generate MatMul->Add ONNX patterns.

• torch_models.FCSeqAddBiasVec: Torch model that should generate MatMul->Add ONNX patterns.

• torch_models.FCSmall: Torch model for the tests.

• torch_models.ManualLogisticRegressionTraining: PyTorch module for performing SGD training.

• torch_models.MultiInputNN: Torch model to test multiple inputs forward.

• torch_models.MultiInputNNConfigurable: Torch model to test multiple inputs forward.

• torch_models.MultiInputNNDifferentSize: Torch model to test multiple inputs with different shape in the forward pass.

• torch_models.MultiOpOnSingleInputConvNN: Network that applies two quantized operations on a single input.

• torch_models.MultiOutputModel: Multi-output model.

• torch_models.NetWithConcatUnsqueeze: Torch model to test the concat and unsqueeze operators.

• torch_models.NetWithConstantsFoldedBeforeOps: Torch QAT model that does not quantize the inputs.

• torch_models.NetWithLoops: Torch model, where we reuse some elements in a loop.

• torch_models.PaddingNet: Torch QAT model that applies various padding patterns.

• torch_models.PartialQATModel: A model with a QAT Module.

• torch_models.QATTestModule: Torch model that implements a simple non-uniform quantizer.

• torch_models.QuantCustomModel: A small quantized network with Brevitas, trained on make_classification.

• torch_models.ShapeOperationsNet: Torch QAT model that reshapes the input.

• torch_models.SimpleNet: Fake torch model used to generate some onnx.

• torch_models.SimpleQAT: Torch model implements a step function that needs Greater, Cast and Where.

• torch_models.SingleMixNet: Torch model that with a single conv layer that produces the output, e.g., a blur filter.

• torch_models.StepActivationModule: Torch model implements a step function that needs Greater, Cast and Where.

• torch_models.TinyCNN: A very small CNN.

• torch_models.TinyQATCNN: A very small QAT CNN to classify the sklearn digits data-set.

• torch_models.TorchCustomModel: A small network with Brevitas, trained on make_classification.

• torch_models.TorchSum: Torch model to test the ReduceSum ONNX operator in a leveled circuit.

• torch_models.UnivariateModule: Torch model that calls univariate and shape functions of torch.

• base_quantized_op.QuantizedMixingOp: An operator that mixes (adds or multiplies) together encrypted inputs.

• base_quantized_op.QuantizedOp: Base class for quantized ONNX ops implemented in numpy.

• base_quantized_op.QuantizedOpUnivariateOfEncrypted: An univariate operator of an encrypted value.

• post_training.ONNXConverter: Base ONNX to Concrete ML computation graph conversion class.

• post_training.PostTrainingAffineQuantization: Post-training Affine Quantization.

• post_training.PostTrainingQATImporter: Converter of Quantization Aware Training networks.

• quantized_module.QuantizedModule: Inference for a quantized model.

• quantized_module_passes.PowerOfTwoScalingRoundPBSAdapter: Detect neural network patterns that can be optimized with round PBS.

• quantized_ops.ONNXConstantOfShape: ConstantOfShape operator.

• quantized_ops.ONNXGather: Gather operator.

• quantized_ops.ONNXShape: Shape operator.

• quantized_ops.ONNXSlice: Slice operator.

• quantized_ops.QuantizedAbs: Quantized Abs op.

• quantized_ops.QuantizedAdd: Quantized Addition operator.

• quantized_ops.QuantizedAvgPool: Quantized Average Pooling op.

• quantized_ops.QuantizedBatchNormalization: Quantized Batch normalization with encrypted input and in-the-clear normalization params.

• quantized_ops.QuantizedBrevitasQuant: Brevitas uniform quantization with encrypted input.

• quantized_ops.QuantizedCast: Cast the input to the required data type.

• quantized_ops.QuantizedCelu: Quantized Celu op.

• quantized_ops.QuantizedClip: Quantized clip op.

• quantized_ops.QuantizedConcat: Concatenate operator.

• quantized_ops.QuantizedConv: Quantized Conv op.

• quantized_ops.QuantizedDiv: Div operator /.

• quantized_ops.QuantizedElu: Quantized Elu op.

• quantized_ops.QuantizedEqual: Comparison operator ==.

• quantized_ops.QuantizedErf: Quantized erf op.

• quantized_ops.QuantizedExp: Quantized Exp op.

• quantized_ops.QuantizedExpand: Expand operator for quantized tensors.

• quantized_ops.QuantizedFlatten: Quantized flatten for encrypted inputs.

• quantized_ops.QuantizedFloor: Quantized Floor op.

• quantized_ops.QuantizedGemm: Quantized Gemm op.

• quantized_ops.QuantizedGreater: Comparison operator >.

• quantized_ops.QuantizedGreaterOrEqual: Comparison operator >=.

• quantized_ops.QuantizedHardSigmoid: Quantized HardSigmoid op.

• quantized_ops.QuantizedHardSwish: Quantized Hardswish op.

• quantized_ops.QuantizedIdentity: Quantized Identity op.

• quantized_ops.QuantizedLeakyRelu: Quantized LeakyRelu op.

• quantized_ops.QuantizedLess: Comparison operator <.

• quantized_ops.QuantizedLessOrEqual: Comparison operator <=.

• quantized_ops.QuantizedLog: Quantized Log op.

• quantized_ops.QuantizedMatMul: Quantized MatMul op.

• quantized_ops.QuantizedMax: Quantized Max op.

• quantized_ops.QuantizedMaxPool: Quantized Max Pooling op.

• quantized_ops.QuantizedMin: Quantized Min op.

• quantized_ops.QuantizedMul: Multiplication operator.

• quantized_ops.QuantizedNeg: Quantized Neg op.

• quantized_ops.QuantizedNot: Quantized Not op.

• quantized_ops.QuantizedOr: Or operator ||.

• quantized_ops.QuantizedPRelu: Quantized PRelu op.

• quantized_ops.QuantizedPad: Quantized Padding op.

• quantized_ops.QuantizedPow: Quantized pow op.

• quantized_ops.QuantizedReduceSum: ReduceSum with encrypted input.

• quantized_ops.QuantizedRelu: Quantized Relu op.

• quantized_ops.QuantizedReshape: Quantized Reshape op.

• quantized_ops.QuantizedRound: Quantized round op.

• quantized_ops.QuantizedSelu: Quantized Selu op.

• quantized_ops.QuantizedSigmoid: Quantized sigmoid op.

• quantized_ops.QuantizedSign: Quantized Neg op.

• quantized_ops.QuantizedSoftplus: Quantized Softplus op.

• quantized_ops.QuantizedSqueeze: Squeeze operator.

• quantized_ops.QuantizedSub: Subtraction operator.

• quantized_ops.QuantizedTanh: Quantized Tanh op.

• quantized_ops.QuantizedTranspose: Transpose operator for quantized inputs.

• quantized_ops.QuantizedUnfold: Quantized Unfold op.

• quantized_ops.QuantizedUnsqueeze: Unsqueeze operator.

• quantized_ops.QuantizedWhere: Where operator on quantized arrays.

• quantizers.MinMaxQuantizationStats: Calibration set statistics.

• quantizers.QuantizationOptions: Options for quantization.

• quantizers.QuantizedArray: Abstraction of quantized array.

• quantizers.UniformQuantizationParameters: Quantization parameters for uniform quantization.

• quantizers.UniformQuantizer: Uniform quantizer.

• p_error_search.BinarySearch: Class for p_error hyper-parameter search for classification and regression tasks.

• base.BaseClassifier: Base class for linear and tree-based classifiers in Concrete ML.

• base.BaseEstimator: Base class for all estimators in Concrete ML.

• base.BaseTreeClassifierMixin: Mixin class for tree-based classifiers.

• base.BaseTreeEstimatorMixin: Mixin class for tree-based estimators.

• base.BaseTreeRegressorMixin: Mixin class for tree-based regressors.

• base.QuantizedTorchEstimatorMixin: Mixin that provides quantization for a torch module and follows the Estimator API.

• base.SklearnKNeighborsClassifierMixin: A Mixin class for sklearn KNeighbors classifiers with FHE.

• base.SklearnKNeighborsMixin: A Mixin class for sklearn KNeighbors models with FHE.

• base.SklearnLinearClassifierMixin: A Mixin class for sklearn linear classifiers with FHE.

• base.SklearnLinearModelMixin: A Mixin class for sklearn linear models with FHE.

• base.SklearnLinearRegressorMixin: A Mixin class for sklearn linear regressors with FHE.

• base.SklearnSGDClassifierMixin: A Mixin class for sklearn SGD classifiers with FHE.

• base.SklearnSGDRegressorMixin: A Mixin class for sklearn SGD regressors with FHE.

• glm.GammaRegressor: A Gamma regression model with FHE.

• glm.PoissonRegressor: A Poisson regression model with FHE.

• glm.TweedieRegressor: A Tweedie regression model with FHE.

• linear_model.ElasticNet: An ElasticNet regression model with FHE.

• linear_model.Lasso: A Lasso regression model with FHE.

• linear_model.LinearRegression: A linear regression model with FHE.

• linear_model.LogisticRegression: A logistic regression model with FHE.

• linear_model.Ridge: A Ridge regression model with FHE.

• linear_model.SGDClassifier: An FHE linear classifier model fitted with stochastic gradient descent.

• linear_model.SGDRegressor: An FHE linear regression model fitted with stochastic gradient descent.

• neighbors.KNeighborsClassifier: A k-nearest neighbors classifier model with FHE.

• qnn.NeuralNetClassifier: A Fully-Connected Neural Network classifier with FHE.

• qnn.NeuralNetRegressor: A Fully-Connected Neural Network regressor with FHE.

• qnn_module.SparseQuantNeuralNetwork: Sparse Quantized Neural Network.

• rf.RandomForestClassifier: Implements the RandomForest classifier.

• rf.RandomForestRegressor: Implements the RandomForest regressor.

• svm.LinearSVC: A Classification Support Vector Machine (SVM).

• svm.LinearSVR: A Regression Support Vector Machine (SVM).

• tree.DecisionTreeClassifier: Implements the sklearn DecisionTreeClassifier.

• tree.DecisionTreeRegressor: Implements the sklearn DecisionTreeClassifier.

• xgb.XGBClassifier: Implements the XGBoost classifier.

• xgb.XGBRegressor: Implements the XGBoost regressor.

• hybrid_model.HybridFHEMode: Simple enum for different modes of execution of HybridModel.

• hybrid_model.HybridFHEModel: Convert a model to a hybrid model.

• hybrid_model.HybridFHEModelServer: Hybrid FHE Model Server.

• hybrid_model.LoggerStub: Placeholder type for a typical logger like the one from loguru.

• hybrid_model.RemoteModule: A wrapper class for the modules to be evaluated remotely with FHE.

• numpy_module.NumpyModule: General interface to transform a torch.nn.Module to numpy module.