Using GitBook

Documentation with GitBook is done mainly by pushing content on GitHub. GitBook then pulls the docs from the repository and publishes. In most cases, GitBook is just a mirror of what is available in GitHub.

There are, however, some use-cases where documentation can be modified directly in GitBook (and, then, push the modifications to GitHub), for example when the documentation is modified by a person outside of Zama. In this case, a GitHub branch is created, and a GitHub space is associated to it: modifications are done in this space and automatically pushed to the branch. Once the modifications have been completed, one can simply create a pull-request, to finally merge modifications on the main branch.

Using Sphinx

Documentation can alternatively be built using Sphinx:

The documentation contains both files written by hand by developers (the .md files) and files automatically created by parsing the source files.

Then to open it, go to docs/_build/html/index.html or use the follwing command:

To build and open the docs at the same time, use:

Concrete ML is a constant work-in-progress, and thus may contain bugs or suboptimal APIs.

Before opening an issue or asking for support, please read this documentation to understand common issues and limitations of Concrete ML. You can also check the outstanding issues on github.

Furthermore, undefined behavior may occur if the input-set, which is internally used by the compilation core to set bit-widths of some intermediate data, is not sufficiently representative of the future user inputs. With all the inputs in the input-set, it appears that intermediate data can be represented as an n-bit integer. But, for a particular computation, this same intermediate data needs additional bits to be represented. The FHE execution for this computation will result in an incorrect output, as typically occurs in integer overflows in classical programs.

If you didn't find an answer, you can ask a question on the Zama forum or in the FHE.org Discord.

Submitting an issue

When submitting an issue (here), ideally include as much information as possible. In addition to the Python script, the following information is useful:

• the reproducibility rate you see on your side

• any insight you might have on the bug

• any workaround you have been able to find

If you would like to contribute to a project and send pull requests, take a look at the contributor guide.

Before you start this section, you must install Docker by following this official guide.

Building the image

Once you have access to this repository and the dev environment is installed on your host OS (via make setup_env once you followed the steps here), you should be able to launch the commands to build the dev Docker image with make docker_build.

Once you do that, you can get inside the Docker environment using the following command:

make docker_start

# or build and start at the same time
make docker_build_and_start

# or equivalently but shorter
make docker_bas

After you finish your work, you can leave Docker by using the exit command or by pressing CTRL + D.

There are three ways to contribute to Concrete ML:

• You can open issues to report bugs and typos and to suggest ideas.

• You can ask to become an official contributor by emailing hello@zama.ai. Only approved contributors can send pull requests (PR), so please make sure to get in touch before you do.

• You can also provide new tutorials or use-cases, showing what can be done with the library. The more examples we have, the better and clearer it is for the other users.

1. Creating a new branch

To create your branch, you have to use the issue ID somewhere in the branch name:

git checkout -b {feat|fix|refactor|test|benchmark|doc|style|chore}/short-description_$issue_id
git checkout -b short-description_$issue_id
git checkout -b $issue_id_short-description

For example:

git checkout -b feat/explicit-tlu_11
git checkout -b tracing_indexing_42
git checkout -b 42_tracing_indexing

2. Before committing

2.1 Conformance

Each commit to Concrete ML should conform to the standards of the project. You can let the development tools fix some issues automatically with the following command:

make conformance

Conformance can be checked using the following command:

make pcc

2.2 Testing

Your code must be well documented, containing tests and not breaking other tests:

make pytest

You need to make sure you get 100% code coverage. The make pytest command checks that by default and will fail with a coverage report at the end should some lines of your code not be executed during testing.

If your coverage is below 100%, you should write more tests and then create the pull request. If you ignore this warning and create the PR, GitHub actions will fail and your PR will not be merged.

There may be cases where covering your code is not possible (an exception that cannot be triggered in normal execution circumstances). In those cases, you may be allowed to disable coverage for some specific lines. This should be the exception rather than the rule, and reviewers will ask why some lines are not covered. If it appears they can be covered, then the PR won't be accepted in that state.

3. Committing

Concrete ML uses a consistent commit naming scheme, and you are expected to follow it as well (the CI will make sure you do). The accepted format can be printed to your terminal by running:

make show_scope

For example:

git commit -m "feat: implement bounds checking"
git commit -m "feat(debugging): add an helper function to draw intermediate representation"
git commit -m "fix(tracing): fix a bug that crashed PyTorch tracer"

Just a reminder that commit messages are checked in the conformance step and are rejected if they don't follow the rules. To learn more about conventional commits, check this page.

4. Rebasing

You should rebase on top of the main branch before you create your pull request. Merge commits are not allowed, so rebasing on main before pushing gives you the best chance of to avoid rewriting parts of your PR later if conflicts arise with other PRs being merged. After you commit changes to your new branch, you can use the following commands to rebase:

# fetch the list of active remote branches
git fetch --all --prune

# checkout to main
git checkout main

# pull the latest changes to main (--ff-only is there to prevent accidental commits to main)
git pull --ff-only

# checkout back to your branch
git checkout $YOUR_BRANCH

# rebase on top of main branch
git rebase main

# If there are conflicts during the rebase, resolve them
# and continue the rebase with the following command
git rebase --continue

# push the latest version of the local branch to remote
git push --force

You can learn more about rebasing here.

Concrete ML is a Python library, so Python should be installed to develop Concrete ML. v3.8 and v3.9 are the only supported versions. Concrete ML also uses Poetry and Make.

First of all, you need to git clone the project:

git clone https://github.com/zama-ai/concrete-ml

Some tests require files tracked by git-lfs to be downloaded. To do so, please follow the instructions on git-lfs website, then run git lfs pull.

Automatic installation

A simple way to have everything installed is to use the development Docker (see the Docker setup guide). On Linux and macOS, you have to run the script in ./script/make_utils/setup_os_deps.sh. Specify the --linux-install-python flag if you want to install python3.8 as well on apt-enabled Linux distributions. The script should install everything you need for Docker and bare OS development (you can first review the content of the file to check what it will do).

Manual installation

Python

To manually install Python, you can follow this guide (alternatively, you can google how to install Python 3.8 (or 3.9)).

Poetry

Poetry is used as the package manager. It drastically simplifies dependency and environment management. You can follow this official guide to install it.

make

The dev tools use make to launch various commands.

On Linux, you can install make from your distribution's preferred package manager.

On macOS, you can install a more recent version of make via brew:

# check for gmake
which gmake

# If you don't have it, it will error out, install gmake
brew install make

# recheck, now you should have gmake
which gmake

It is possible to install gmake as make. Check this StackOverflow post for more info.

On Windows, check this GitHub gist.

Cloning the repository

To get the source code of Concrete ML, clone the code repository using the link for your favorite communication protocol (ssh or https).

Setting up environment on your host OS

We are going to make use of virtual environments. This helps to keep the project isolated from other Python projects in the system. The following commands will create a new virtual environment under the project directory and install dependencies to it.

cd concrete-ml
make setup_env

Activating the environment

Finally, activate the newly created environment using the following command:

macOS or Linux

source .venv/bin/activate

Windows

source .venv/Scripts/activate

Setting up environment on Docker

Docker automatically creates and sources a venv in ~/dev_venv/

The venv persists thanks to volumes. It also creates a volume for ~/.cache to speedup later reinstallations. You can check which Docker volumes exist with:

docker volume ls

You can still run all make commands inside Docker (to update the venv, for example). Be mindful of the current venv being used (the name in parentheses at the beginning of your command prompt).

# Here we have dev_venv sourced
(dev_venv) dev_user@8e299b32283c:/src$ make setup_env

Leaving the environment

After your work is done, you can simply run the following command to leave the environment:

deactivate

Syncing environment with the latest changes

From time to time, new dependencies will be added to the project or the old ones will be removed. The command below will make sure the project has the proper environment, so run it regularly!

make sync_env

Troubleshooting your environment

in your OS

If you are having issues, consider using the dev Docker exclusively (unless you are working on OS-specific bug fixes or features).

Here are the steps you can take on your OS to try and fix issues:

# Try to install the env normally
make setup_env

# If you are still having issues, sync the environment
make sync_env

# If you are still having issues on your OS, delete the venv:
rm -rf .venv

# And re-run the env setup
make setup_env

At this point, you should consider using Docker as nobody will have the exact same setup as you. If, however, you need to develop on your OS directly, you can ask Zama for help.

in Docker

Here are the steps you can take in your Docker to try and fix issues:

# Try to install the env normally
make setup_env

# If you are still having issues, sync the environment
make sync_env

# If you are still having issues in Docker, delete the venv:
rm -rf ~/dev_venv/*

# Disconnect from Docker
exit

# And relaunch, the venv will be reinstalled
make docker_start

# If you are still out of luck, force a rebuild which will also delete the volumes
make docker_rebuild

# And start Docker, which will reinstall the venv
make docker_start

If the problem persists at this point, you should ask for help. We're here and ready to assist!