There are three ways to contribute to Concrete ML:
You can open issues to report bugs and typos and to suggest ideas.
You can become an official contributor but you need to sign our Contributor License Agreement (CLA) on your first contribution. Our CLA-bot will guide you through the process when you will open a Pull Request on Github.
You can also provide new tutorials or use-cases, showing what can be done with the library. The more examples we have, the better and clearer it is for the other users.
First, you need to properly set up the project by following the steps provided here.
To create your branch, you have to use the issue ID somewhere in the branch name:
For example:
Each commit to Concrete ML should conform to the standards of the project. You can let the development tools fix some issues automatically with the following command:
Conformance can be checked using the following command:
Your code must be well documented, containing tests and not breaking other tests:
You need to make sure you get 100% code coverage. The make pytest command checks that by default and will fail with a coverage report at the end should some lines of your code not be executed during testing.
If your coverage is below 100%, you should write more tests and then create the pull request. If you ignore this warning and create the PR, GitHub actions will fail and your PR will not be merged.
There may be cases where covering your code is not possible (an exception that cannot be triggered in normal execution circumstances). In those cases, you may be allowed to disable coverage for some specific lines. This should be the exception rather than the rule, and reviewers will ask why some lines are not covered. If it appears they can be covered, then the PR won't be accepted in that state.
Concrete ML uses a consistent commit naming scheme, and you are expected to follow it as well (the CI will make sure you do). The accepted format can be printed to your terminal by running:
For example:
Just a reminder that commit messages are checked in the conformance step and are rejected if they don't follow the rules. To learn more about conventional commits, check this page.
You should rebase on top of the main branch before you create your pull request. Merge commits are not allowed, so rebasing on main before pushing gives you the best chance of to avoid rewriting parts of your PR later if conflicts arise with other PRs being merged. After you commit changes to your new branch, you can use the following commands to rebase:
You can learn more about rebasing here.
Before you start this section, you must install Docker by following this official guide.
Once you have access to this repository and the dev environment is installed on your host OS (via make setup_env once you followed the steps here), you should be able to launch the commands to build the dev Docker image with make docker_build.
Once you do that, you can get inside the Docker environment using the following command:
After you finish your work, you can leave Docker by using the exit command or by pressing CTRL + D.
Documentation with GitBook is done mainly by pushing content on GitHub. GitBook then pulls the docs from the repository and publishes. In most cases, GitBook is just a mirror of what is available in GitHub.
There are, however, some use-cases where documentation can be modified directly in GitBook (and, then, push the modifications to GitHub), for example when the documentation is modified by a person outside of Zama. In this case, a GitHub branch is created, and a GitHub space is associated to it: modifications are done in this space and automatically pushed to the branch. Once the modifications have been completed, one can simply create a pull-request, to finally merge modifications on the main branch.
Documentation can alternatively be built using Sphinx:
The documentation contains both files written by hand by developers (the .md files) and files automatically created by parsing the source files.
Then to open it, go to docs/_build/html/index.html or use the follwing command:
To build and open the docs at the same time, use:
Concrete ML is a constant work-in-progress, and thus may contain bugs or suboptimal APIs.
Before opening an issue or asking for support, please read this documentation to understand common issues and limitations of Concrete ML. You can also check the outstanding issues on github.
Furthermore, undefined behavior may occur if the input-set, which is internally used by the compilation core to set bit-widths of some intermediate data, is not sufficiently representative of the future user inputs. With all the inputs in the input-set, it appears that intermediate data can be represented as an n-bit integer. But, for a particular computation, this same intermediate data needs additional bits to be represented. The FHE execution for this computation will result in an incorrect output, as typically occurs in integer overflows in classical programs.
If you didn't find an answer, you can ask a question on the Zama forum or in the FHE.org Discord.
When submitting an issue (here), ideally include as much information as possible. In addition to the Python script, the following information is useful:
the reproducibility rate you see on your side
any insight you might have on the bug
any workaround you have been able to find
If you would like to contribute to a project and send pull requests, take a look at the contributor guide.
Concrete ML is a Python library, so Python should be installed to develop Concrete ML. v3.8 and v3.9 are the only supported versions. Concrete ML also uses Poetry and Make.
First of all, you need to git clone the project:
Several files are tracked by git-lfs. While a few are required for running some tests, most of them are used for benchmarking and use case examples. By default, git clone downloads all LFS files, which can add up to several hundreds of MB to the directory. Is it however possible to disable such behavior by running the running the following command instead :
A simple way to have everything installed is to use the development Docker (see the Docker setup guide). On Linux and macOS, you have to run the script in ./script/make_utils/setup_os_deps.sh. Specify the --linux-install-python flag if you want to install python3.8 as well on apt-enabled Linux distributions. The script should install everything you need for Docker and bare OS development (you can first review the content of the file to check what it will do).
For Windows users, the setup_os_deps.sh script does not install dependencies because of how many different installation methods there are due to the lack of a single package manager.
The first step is to install Python (as some of the dev tools depend on it), then Poetry. In addition to installing Python, you are still going to need the following software available on path on Windows, as some of the basic dev tools depend on them:
Development on Windows only works with the Docker environment. Follow this link to setup the Docker environment.
To manually install Python, you can follow this guide (alternatively, you can google how to install Python 3.8 (or 3.9)).
Poetry is used as the package manager. It drastically simplifies dependency and environment management. You can follow this official guide to install it.
The dev tools use make to launch various commands.
On Linux, you can install make from your distribution's preferred package manager.
On macOS, you can install a more recent version of make via brew:
It is possible to install gmake as make. Check this StackOverflow post for more info.
On Windows, check this GitHub gist.
In the following sections, be sure to use the proper make tool for your system: make, gmake, or other.
To get the source code of Concrete ML, clone the code repository using the link for your favorite communication protocol (ssh or https).
We are going to make use of virtual environments. This helps to keep the project isolated from other Python projects in the system. The following commands will create a new virtual environment under the project directory and install dependencies to it.
The following command will not work on Windows if you don't have Poetry >= 1.2.
Finally, activate the newly created environment using the following command:
Docker automatically creates and sources a venv in ~/dev_venv/
The venv persists thanks to volumes. It also creates a volume for ~/.cache to speedup later reinstallations. You can check which Docker volumes exist with:
You can still run all make commands inside Docker (to update the venv, for example). Be mindful of the current venv being used (the name in parentheses at the beginning of your command prompt).
After your work is done, you can simply run the following command to leave the environment:
From time to time, new dependencies will be added to the project or the old ones will be removed. The command below will make sure the project has the proper environment, so run it regularly!
If you are having issues, consider using the dev Docker exclusively (unless you are working on OS-specific bug fixes or features).
Here are the steps you can take on your OS to try and fix issues:
At this point, you should consider using Docker as nobody will have the exact same setup as you. If, however, you need to develop on your OS directly, you can ask Zama for help.
Here are the steps you can take in your Docker to try and fix issues:
If the problem persists at this point, you should ask for help. We're here and ready to assist!