# Dark Market with Integer API

Last updated

Last updated

In this tutorial, we are going to build a dark market application using TFHE-rs. A dark market is a marketplace where buy and sell orders are not visible to the public before they are filled. Different algorithms aim to solve this problem, we are going to implement the algorithm defined in this paper with TFHE-rs.

We will first implement the algorithm in plain Rust and then we will see how to use TFHE-rs to implement the same algorithm with FHE.

In addition, we will also implement a modified version of the algorithm that allows for more concurrent operations which improves the performance in hardware where there are multiple cores.

Specifications

Inputs:

A list of sell orders where each sell order is only defined in volume terms, it is assumed that the price is fetched from a different source.

A list of buy orders where each buy order is only defined in volume terms, it is assumed that the price is fetched from a different source.

Input constraints:

The sell and buy orders are within the range [1,100].

The maximum number of sell and buy orders is 500, respectively.

Outputs:

There is no output returned at the end of the algorithm. Instead, the algorithm makes changes on the given input lists. The number of filled orders is written over the original order count in the respective lists. If it is not possible to fill the orders, the order count is set to zero.

Example input and output:

**Example 1:**

Sell | Buy | |
---|---|---|

Last three indices of the filled sell orders are zero because there is no buy orders to match them.

**Example 2:**

Last three indices of the filled buy orders are zero because there is no sell orders to match them.

Plain Implementation

Calculate the total sell volume and the total buy volume.

Find the total volume that will be transacted. In the paper, this amount is calculated with the formula:

When closely observed, we can see that this formula can be replaced with the `min`

function. Therefore, we calculate this value by taking the minimum of the total sell volume and the total buy volume.

Beginning with the first item, start filling the sell orders one by one. We apply the

`min`

function replacement also here.

The number of orders that are filled is indicated by modifying the input list. For example, if the first sell order is 1000 and the total volume is 500, then the first sell order will be modified to 500 and the second sell order will be modified to 0.

Do the fill operation also for the buy orders.

The complete algorithm in plain Rust:

FHE Implementation

For the FHE implementation, we first start with finding the right bit size for our algorithm to work without overflows.

The variables that are declared in the algorithm and their maximum values are described in the table below:

As we can observe from the table, we need **16 bits of message space** to be able to run the algorithm without overflows. TFHE-rs provides different presets for the different bit sizes. Since we need 16 bits of message, we are going to use the `integer`

module to implement the algorithm.

Here are the input types of our algorithm:

`sell_orders`

is of type`Vec<tfhe::integer::RadixCipherText>`

`buy_orders`

is of type`Vec<tfhe::integer::RadixCipherText>`

`server_key`

is of type`tfhe::integer::ServerKey`

Now, we can start implementing the algorithm with FHE:

Calculate the total sell volume and the total buy volume.

Find the total volume that will be transacted by taking the minimum of the total sell volume and the total buy volume.

Beginning with the first item, start filling the sell and buy orders one by one. We can create

`fill_orders`

closure to reduce code duplication since the code for filling buy orders and sell orders are the same.

The complete algorithm in TFHE-rs:

Optimizing the implementation

TFHE-rs provides parallelized implementations of the operations. We can use these parallelized implementations to speed up the algorithm. For example, we can use

`smart_add_assign_parallelized`

instead of`smart_add_assign`

.We can parallelize vector sum with Rayon and

`reduce`

operation.

We can run vector summation on

`buy_orders`

and`sell_orders`

in parallel since these operations do not depend on each other.

We can match sell and buy orders in parallel since the matching does not depend on each other.

Optimized algorithm

Modified Algorithm

When observed closely, there is only a small amount of concurrency introduced in the `fill_orders`

part of the algorithm. The reason is that the `volume_left_to_transact`

is shared between all the orders and should be modified sequentially. This means that the orders cannot be filled in parallel. If we can somehow remove this dependency, we can fill the orders in parallel.

In order to do so, we closely observe the function of `volume_left_to_transact`

variable in the algorithm. We can see that it is being used to check whether we can fill the current order or not. Instead of subtracting the current order value from `volume_left_to_transact`

in each loop, we can add this value to the next order index and check the availability by comparing the current order value with the total volume. If the current order value (now representing the sum of values before this order plus this order) is smaller than the total number of matching orders, we can safely fill all the orders and continue the loop. If not, we should partially fill the orders with what is left from matching orders.

We will call the new list the "prefix sum" of the array.

The new version for the plain `fill_orders`

is as follows:

To write this new function we need transform the conditional code into a mathematical expression since FHE does not support conditional operations.

New `fill_order`

function requires a prefix sum array. We are going to calculate this prefix sum array in parallel with the algorithm described here.

The sample code in the paper is written in CUDA. When we try to implement the algorithm in Rust we see that the compiler does not allow us to do so. The reason for that is while the algorithm does not access the same array element in any of the threads(the index calculations using `d`

and `k`

values never overlap), Rust compiler cannot understand this and does not let us share the same array between threads. So we modify how the algorithm is implemented, but we don't change the algorithm itself.

Here is the modified version of the algorithm in TFHE-rs:

Running the tutorial

The plain, FHE and parallel FHE implementations can be run by providing respective arguments as described below.

Conclusion

In this tutorial, we've learned how to implement the volume matching algorithm described in this paper in plain Rust and in TFHE-rs. We've identified the right bit size for our problem at hand, used operations defined in `TFHE-rs`

, and introduced concurrency to the algorithm to increase its performance.

Sell | Buy | |
---|---|---|

Variable | Maximum Value | Bit Size |
---|---|---|