[readthedocs] Init readthedocs for Occlum

.readthedocs.yaml

@@ -0,0 +1,8 @@
+version: 2
+
+sphinx:
+  configuration: docs/readthedocs/docs/source/conf.py
+
+python:
+  install:
+    - requirements: docs/readthedocs/requirements.txt

docs/readthedocs/.gitignore

@@ -0,0 +1,156 @@
+
+# Created by https://www.toptal.com/developers/gitignore/api/python,jupyternotebooks
+# Edit at https://www.toptal.com/developers/gitignore?templates=python,jupyternotebooks
+
+### JupyterNotebooks ###
+# gitignore template for Jupyter Notebooks
+# website: http://jupyter.org/
+
+.ipynb_checkpoints
+*/.ipynb_checkpoints/*
+
+# IPython
+profile_default/
+ipython_config.py
+
+# Remove previous ipynb_checkpoints
+#   git rm -r .ipynb_checkpoints/
+
+### Python ###
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+
+# IPython
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# End of https://www.toptal.com/developers/gitignore/api/python,jupyternotebooks

docs/readthedocs/docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/readthedocs/docs/make.bat

@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=source
+set BUILDDIR=build
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.http://sphinx-doc.org/
+	exit /b 1
+)
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd

docs/readthedocs/docs/source/binaries_compatibility.md

@@ -0,0 +1,22 @@
+# Occlum-Compatible Executable Binaries
+
+The `hello_world` demo is based on [musl libc](https://www.musl-libc.org/) with recompiling. But Occlum actually can support both `musl libc` and `glibc` based executable binaries without recompiling if they meet below three principles.
+
+## No fork syscall
+
+By design, Occlum doesn't support `fork` syscall. If there is `fork` syscall in the application, users have to assess if the `fork` could be replaced by `vfork + exec` or `posix spawn`. If yes, code modification and recompiling is inevitable.
+
+## libc version compatibility
+
+No recompiling doesn't mean the original `libc` libraries can be directly used in Occlum. To run in Occlum TEE environment, customized `libc` libraries are provided in the Occlum development docker image.
+
+|    libc   |  Compatible Version in Occlum  | Path in Occlum Docker Image |
+| --------- | ------------------------------ | --------------------------- |
+| musl libc | <=1.1.24<br>(default version in Alpine:3.11) | /usr/local/occlum/x86_64-linux-musl/lib/ |
+|   glibc   | <=2.31<br>(default version in Ubuntu:20.04)  |         /opt/occlum/glibc/lib/           |
+
+Actually, the original `libc` libraries are to be replaced silently in `Occlum build` stage by `copy_bom` tool.
+
+## Compiled with PIE (Position-Independent-Executable)
+
+Current Ubuntu:20.04 and Alpine:3.11 enable `PIE` in default.

docs/readthedocs/docs/source/boot_flow.md

@@ -0,0 +1,27 @@
+# Boot Flow Overview
+
+Occlum has a unique **Occlum -> init ->application** boot flow, plus integrity check stage by stage. Below is the high-level overview.
+
+![Boot_flow](./images/boot_flow.png)
+
+## Measurements
+
+There are total three MACs which are all generated in building stage and verified in boot flow.
+
+* MAC of the init FS, it indicates the measurement of init file system which will be mounted in **init** stage.
+
+* MAC of the root FS, it indicates the measurement of application file system which will be mounted in **application** stage.
+
+* MAC of the configuration file. Both above MACs are filled into the configuration file. Thus the MAC of the configuration file reflects the differences of the two file systems to some extent.
+
+* **mrenclave**, it indicates the measurement of the Occlum LibOS.
+
+## Boot flow
+
+1. To user, the entry point is the command `occlum run`. The steps behind the command are PAL APIs `occlum_pal_init` and `occlum_pal_create_process`. It launches the Occlum LibOS kernel in the Enclave. The kernel then loads the configuration file `.Occlum_sys.json.protected`, doing integrity check with the MAC saved in `KSS ISV FamilyID`. If pass, it uses the settings in the configuration file to do memory/process initialization. Then, it tries mount the init filesystem. It does integrity check again with MAC in the `.Occlum_sys.json.protected`. If pass, the first user space process `init` got launched.
+
+2. There is a default minimal `init` provided. In this `init`, it calls the Occlum added syscall `SYS_MOUNT_FS` to mount the application file system. The syscall implementation in the Occlum kernel does integrity check again with MAC in the `.Occlum_sys.json.protected` to make sure expected application got launched.
+
+3. At this stage, the real user application got launched.
+
+Generally, all operation application required but not part of the application, such as remote attestation, could be put into **"init"**. This feature makes Occlum highly compatible to any remote attestation solution without involving application’s change.

docs/readthedocs/docs/source/build_and_install.md

@@ -0,0 +1,60 @@
+# Build and Install
+
+Generally, users don't need build the Occlum from source. They can directly use Occlum official docker image in Docker hub.
+```
+docker pull occlum/occlum:[version]-ubuntu20.04
+```
+
+## Build from Source
+
+To build Occlum from the latest source code, do the following steps in an Occlum Docker container (which can be prepared as shown in the last section):
+
+1. Download the latest source code of Occlum
+    ```
+    mkdir occlum && cd occlum
+    git clone https://github.com/occlum/occlum .
+    ```
+2. Prepare the submodules and tools required by Occlum.
+    ```
+    make submodule
+    ```
+3. Compile and test Occlum
+    ```
+    make
+
+    # test musl based binary
+    make test
+
+    # test glibc based binary
+    make test-glibc
+
+    # stress test
+    make test times=100
+    ```
+
+    For platforms that don't support SGX
+    ```
+    SGX_MODE=SIM make
+    SGX_MODE=SIM make test
+    ```
+4. Install Occlum
+    ```
+    make install
+    ```
+   which will install the `occlum` command-line tool and other files at `/opt/occlum`.
+
+If `release` build and install is required, just add **OCCLUM_RELEASE_BUILD=1** in front of every `make` command.   
+The Occlum Dockerfile can be found at [here](https://github.com/occlum/occlum/tree/master/tools/docker). Use it to build the container directly or read it to see the dependencies of Occlum.
+
+## How to Build and Run Release-Mode Enclaves?
+
+By default, the `occlum build` command builds and signs enclaves in debug mode. These SGX debug-mode enclaves are intended for development and testing purposes only. For production usage, the enclaves must be signed by a key acquired from Intel (a restriction that will be lifted in the future when Flexible Launch Control is ready) and run with SGX debug support disabled.
+
+Occlum has built-in support for both building and running enclaves in release mode.
+To do that, modify `Occlum.yaml` [metadata]-[debuggable] field to `false`. And then run the commands below:
+```
+$ occlum build --sign-key <path_to/your_key.pem>
+$ occlum run <prog_path> <prog_args>
+```
+
+Ultimately, whether an enclave is running in the release mode should be checked and judged by a trusted client through remotely attesting the enclave. See the remote attestation demo [here](demos/remote_attestation).

docs/readthedocs/docs/source/conf.py

@@ -0,0 +1,47 @@
+# Configuration file for the Sphinx documentation builder.
+
+# -- Project information
+
+# The suffix of source filenames.
+from recommonmark.parser import CommonMarkParser
+source_suffix = {'.rst': 'restructuredtext',
+                 '.txt': 'markdown',
+                 '.md': 'markdown',}
+
+
+project = 'Occlum'
+copyright = '2023, Occlum Contributors'
+author = 'Occlum Contributors'
+
+release = ''
+version = ''
+
+# The master toctree document.
+master_doc = 'index'
+
+# -- General configuration
+
+extensions = [
+    'sphinx.ext.duration',
+    'sphinx.ext.doctest',
+    'sphinx.ext.autodoc',
+    'sphinx.ext.autosummary',
+    'sphinx.ext.intersphinx',
+    'recommonmark',
+    'sphinx_markdown_tables'
+]
+
+intersphinx_mapping = {
+    'python': ('https://docs.python.org/3/', None),
+    'sphinx': ('https://www.sphinx-doc.org/en/master/', None),
+}
+intersphinx_disabled_domains = ['std']
+
+templates_path = ['_templates']
+
+# -- Options for HTML output
+
+html_theme = 'sphinx_rtd_theme'
+
+# -- Options for EPUB output
+epub_show_urls = 'footnote'

docs/readthedocs/docs/source/demos/demos.md

@@ -0,0 +1,3 @@
+# Demos
+
+This part introduces sample projects that demonstrate how Occlum can be used to build and run user applications. Full source code could be found on [github](https://github.com/occlum/occlum/tree/master/demos).

docs/readthedocs/docs/source/filesystem/encrypted_image.md

@@ -0,0 +1,9 @@
+# The Encrypted FS Image
+Occlum has supported using an encrypted FS image, which is encrypted by a user-provided key, to run apps inside the enclave. The confidentiality and integrity of user's files and libraries are both protected with it.
+
+## How to use
+To generate the encrypted FS image, user must give the `--image-key <key_path>` flag in the `occlum build` command (If the flag is not given, the secure FS image will be integrity protected only).
+
+The `<key_path>` refers to a file consisting of a 128-bit key and the user can generate it via the `occlum gen-image-key <key_path>` command.
+
+After generating the encrypted FS image, the [init](https://github.com/occlum/occlum/tree/master/tools/init) process is responsible for mounting the encrypted FS image as the rootfs for the user's Application. Usually the key should be acquired through RA or LA, please take the [init_ra](https://github.com/occlum/occlum/tree/master/demos/remote_attestation/init_ra_flow/init_ra) as an example to use this feature in real world.

docs/readthedocs/docs/source/filesystem/fs_overview.md

@@ -0,0 +1,148 @@
+# Occlum File System Overview
+
+Occlum supports various file systems: e.g., read-only integrity-protected SEFS, writable encrypted SEFS, UnionFS, Async-SFS, untrusted HostFS, RamFS, and other pseudo filesystems.
+
+Here is the default FS layout:
+
+```
+                  │"/"
+           ┌──────┴──────┐
+           │   UnionFS   │
+           │ ┌─────────┐ │
+           │ │ SEFS(RW)│ │
+           │ ├─────────┤ │
+           │ │ SEFS(RO)│ │
+           │ └─────────┘ │
+           │             │
+           └──────┬──────┘
+                  │
+    ┌────────┬────┴─────┬────────┐
+    │        │          │        │
+    │"/sfs"  │"/dev/shm"│"/proc" │"/dev"
+┌───┴─┐   ┌──┴──┐   ┌───┴──┐  ┌──┴──┐
+│A-SFS│   │RamFS│   │ProcFS│  │DevFS│
+└─────┘   └─────┘   └──────┘  └─────┘
+```
+
+## SEFS
+The SEFS is a filesystem based on the [Intel SGX Protected File (PFS)](https://www.intel.com/content/www/us/en/developer/articles/technical/overview-of-intel-protected-file-system-library-using-software-guard-extensions.html), it protects the integrity and confidentiality of disk I/O data from the host.
+
+Here is the hierarchy of SEFS:
+```
+  ┌──────────┐
+  │   SEFS   │
+  │   Inode  │
+  └────┬─────┘
+       │
+ ┌─────┴──────┐
+ │Rust SGX SDK│
+ │   SGXFile  │
+ └─────┬──────┘
+       │
+┌──────┴────────┐
+│ Intel SGX SDK │
+│ Protected File│
+└───────────────┘
+```
+
+There are two modes for SEFS:
+1. Integrity-only mode
+2. Encryption mode
+
+### Integrity-only mode
+We modified the Intel SGX PFS to add this mode. It only protects the integrity of FS, which will generate the deterministic hash for the same FS image. So it is convenient to implement the remote attestation for the enclave with the same FS image.
+
+The FS image (the `image` dir of occlum instance ) provided by the user via the `copy_bom` tool will be transformed into a SEFS image in this mode by default. For the use of `copy_bom`, please refer to [this](https://occlum.readthedocs.io/en/latest/tools/copy_bom.html).
+
+### Encryption mode
+The integrity and confidentiality of the FS are both protected in this mode. There are two key-generation policies for this mode: the autokey generation policy and the user-provided key policy.
+
+* The autokey generation policy
+
+In this policy mode, the user is not required to provide the key. The key is automatically derived from the MRSIGNER of the enclave, the ProdID of the enclave, and the hardware info. So the same owner of two enclaves can share the FS data on the same machine.
+This policy is the default one for the encryption mode of SEFS.
+
+* The user-provided policy
+
+In this policy mode, the key should be provided by the user, which means the enclave owner should manage the key. This policy is more flexible for the user to control the data for sharing or isolation. The [doc](https://occlum.readthedocs.io/en/latest/filesystem/encrypted_image.html) shows you how to use this policy mode.
+
+## UnionFS
+As you can tell, we use the UnionFS consisting of SEFS as the rootfs of LibOS. To attest to the integrity of the user-provided FS image while having the ability to write data when running apps, we introduce a filesystem called UnionFS to satisfy this requirement.
+
+UnionFS allows files and directories of separate file systems, known as branches,  to be transparently overlaid, forming a single coherent file system.
+
+We use two SEFSs to form the UnionFS. The lower layer is the read-only(RO) SEFS in integrity-only mode, and the upper is the writable(RW) SEFS in encryption mode. Generally speaking, the RO-SEFS is transformed by the `image` dir provided by the user while building the enclave, and the RW-SEFS is generated while the enclave is running.
+
+```
+┌─────────────┐
+│   UnionFS   │
+│ ┌─────────┐ │
+│ │ SEFS(RW)│ │
+│ ├─────────┤ │
+│ │ SEFS(RO)│ │
+│ └─────────┘ │
+│             │
+└─────────────┘
+```
+
+Here is the configuration of rootfs, the first item is the lower layer RO-SEFS and the second item is the upper layer RW-SEFS. As you can tell, the RO-SEFS is at `./build/mount/__ROOT` and the RW-SEFS is at `./run/mount/__ROOT`.
+```
+  - target: /
+    type: unionfs
+    options:
+      layers:
+        # The read-only layer which is generated in `occlum build`
+        - target: /
+          type: sefs
+          source: ./build/mount/__ROOT
+          options:
+            MAC: ''
+        # The read-write layer whose content is produced when running the LibOS
+        - target: /
+          type: sefs
+          source: ./run/mount/__ROOT
+```
+
+## Async-SFS
+The Async-SFS is an asynchronous filesystem, which uses Rust asynchronous programming skills, making it fast and concurrent. It is mounted at `/sfs` by default. To achieve the high-performanced security, it uses the JinDisk as the underlying data storage and sends async I/O requests to it.
+
+To accelerate block I/O, the page cache is introduced. It caches all the block I/O in the middle of Async-SFS and JinDisk. Thanks to the page cache and JinDisk, the result of the benchmark (e.g., FIO and Filebench) is significantly better than SEFS. If your App's performance is highly dependent on disk I/O, it is recommended to use Async-SFS.
+```
+┌───────────┐
+│           │
+│ Async-SFS │
+│           │
+└─────┬─────┘
+      │
+┌─────┴─────┐
+│           │
+│ Page Cache│
+│           │
+└─────┬─────┘
+      │
+┌─────┴─────┐
+│           │
+│  JinDisk  │
+│           │
+└───────────┘
+```
+
+Currently, there are some limitations of Async-SFS:
+1. The maximum size of the file is 4GB.
+2. The maximum size of FS is 16TB.
+
+## HostFS
+The HostFS is used for convenient data exchange between the LibOS and the host OS. It simply wraps the untrusted host OS file to implement the functionalities of FS. So the data is straightforwardly transferred between LibOS and host OS without any protection or validation.
+
+## RamFS and other pseudo filesystems
+The RamFS and other pseudo filesystems like ProcFS use the memory as the storage. So the data may lose if one terminates the enclave.
+
+Please remember to enlarge the `kernel_space_heap_size` of Occlum.yaml if your app depends on RamFS.
+
+## Q & A
+
+### How to decrypt and view the rootfs?
+One can use the `occlum mount` command to implements. Please refer to this [doc](https://occlum.readthedocs.io/en/latest/filesystem/mount.html#mount-command) for more information.
+
+### How to mount FS at runtime?
+Please refer to this [doc](https://occlum.readthedocs.io/en/latest/filesystem/mount.html#mount-and-unmount-filesystems-at-runtime).

docs/readthedocs/docs/source/filesystem/mount.md

@@ -0,0 +1,93 @@
+# Mount Support
+
+## Mount command
+
+The `occlum mount` command is designed to mount the secure FS image of the Occlum instance as a Linux FS at a specified path on Linux. This makes it easy to access and manipulate Occlum's secure FS for debug purpose.
+
+### Prerequisites
+
+The command depends on Linux's [Filesystem in Userspace (FUSE)](https://en.wikipedia.org/wiki/Filesystem_in_Userspace), which consists of two components: a kernel module and a userspace library. In most of Linux distributions, the FUSE kernel module is part of the Linux kernel by default, so it should not be a problem. The FUSE library can be installed via a package manager (e.g., `sudo apt-get install libfuse-dev` on Ubuntu). You do not need to install the package manually when using the Occlum Docker image as it has preinstalled the package.
+
+One common pitfall when using FUSE with Docker is about privilege. A Docker container, by default, does not have the privilege to use FUSE. To get the privilege, the Docker container must be started with the following flags:
+```
+--cap-add SYS_ADMIN --device /dev/fuse
+```
+or
+```
+--privileged
+```
+For more info about enabling FUSE for Docker, see [here](https://github.com/docker/for-linux/issues/321).
+
+### How to use
+
+To mount an Occlum's secure FS image successfully, three conditions must be satisfied:
+
+1. The secure FS image exists and is not being used (e.g., via the `occlum run` or `occlum mount` command). This condition ensures that the secure FS will not be broken due to the concurrent access of different Occlum commands.
+
+2. The three (optional) sign key, sign tool and image key arguments that are given to the `occlum mount` command must have the same values as those given to the `occlum build` command, which is how the image is created in the first place. This ensures that the secure FS can only be accessed by the owner of the enclave.
+
+3. If the image key is not given to the `occlum build` command, the `occlum mount` command must be run on the same machine as the `occlum run` command that runs the current Occlum instance and writes to the image. This condition is due to the fact that the automatically drived encryption key of the secure FS is bound to the machine, i.e., the MRSIGNER key policy.
+
+With the three conditions satisfied, the mount command is able to start a Linux FUSE FS server. Any I/O operations on the FUSE FS mounted at the specified path will be redirected by Linux kernel as I/O requests to the FUSE server. The FUSE server is backed by a special enclave, which can encrypt or decrypt the content of the secure FS image on demand.
+
+Please note that if the `autokey_policy` field of the configurations of FS is set in Occlum.yaml, the mount command will not work. This is because the MRENCLAVE is used as an input to generate the encryption key, and the mount tool cannot mimic it.
+
+The mount command **will not return** until the FUSE server is terminated in one of the two ways. The first one is to press ctrl+C. The second one is to use `umount` command. Both ways can terminate the server gracefully.
+
+Step 1: Create an empty directory to serve as the mount point
+```
+mkdir <path>
+```
+
+Step 2: Mount the secure FS at the newly created mount point
+```
+occlum mount [--sign-key <key_path>] [--sign-tool <tool_path>] [--image-key <key_path>] <path>
+```
+After mounting the secure FS successfully, you can access and manipulate the FS from the mount point as easy as regular Linux FS.
+
+
+## Mount and Unmount Filesystems at Runtime
+
+### Background
+Users can specify the mount configuration in the `Occlum.yaml` file, then the file systems are mounted during the libOS startup phase. While this design provides a safe and simple way to access files, it is not as convenient as traditional Host OS. Apps are not allowed to mount and unmount file systems at runtime.
+
+### How to mount filesystems at runtime?
+Apps running inside Occlum can mount some specific file systems via the [mount()](https://man7.org/linux/man-pages/man2/mount.2.html) system call. This makes it flexible to mount and access files at runtime.
+
+Currently, we only support to create a new mount with the trusted UnionFS consisting of SEFSs or the untrusted HostFS. The mount point is not allowed to be the root directory("/").
+
+#### 1. Mount trusted UnionFS consisting of SEFSs
+Example code:
+
+```
+mount("unionfs", "<target_dir>", "unionfs", 0/* mountflags is ignored */,
+      "lowerdir=<lower>,upperdir=<upper>,key=<128-bit-key>")
+```
+
+Mount options:
+
+- The `lowerdir=<lower>` is a mandatory field, which describes the directory path of the RO SEFS on Host OS.
+- The `upperdir=<upper>` is a mandatory field, which describes the directory path of the RW SEFS on Host OS.
+- The `key=<128-bit-key>` is an optional field, which describes the 128bit key used to encrypt or decrypt the FS. Here is an example of the key: `key=c7-32-b3-ed-44-df-ec-7b-25-2d-9a-32-38-8d-58-61`. If this field is not provided, it will use the automatic key derived from the enclave sealing key.
+
+#### 2. Mount untrusted HostFS
+Example code:
+
+```
+mount("hostfs", “<target_dir>”, "hostfs", 0/* mountflags is ignored */,
+      "dir=<host_dir>")
+```
+
+Mount options:
+
+- The `dir=<host_dir>` is a mandatory field, which describes the directory path on Host OS.
+
+### How to unmount filesystems at runtime?
+
+Apps running inside Occlum can unmount some specific file systems via the [umount()/umount2()](https://man7.org/linux/man-pages/man2/umount.2.html) system calls. Note that root directory("/") is not allowed to unmount.
+
+Example code:
+```
+umount("<target_dir>")
+```
+

docs/readthedocs/docs/source/how_to_debug.md

@@ -0,0 +1,7 @@
+# How to Debug?
+
+To debug an app running upon Occlum, one can harness Occlum's builtin support for GDB via `occlum gdb` command. More info can be found [here](https://github.com/occlum/occlum/tree/master/demos/gdb_support).
+
+Meanwhile, one can use `occlum mount` command to access and manipulate the secure filesystem for debug purpose.
+
+If the cause of a problem does not seem to be the app but Occlum itself, then one can take a glimpse into the inner workings of Occlum by checking out its log. Occlum's log level can be adjusted through `OCCLUM_LOG_LEVEL` environment variable. It has six levels: `off`, `error`, `warn`, `debug`, `info`, and `trace`. The default value is `off`, i.e., showing no log messages at all. The most verbose level is `trace`.

docs/readthedocs/docs/source/index.rst

@@ -0,0 +1,68 @@
+Welcome to Occlum's documentation!
+===================================
+
+**Occlum** is a memory-safe, multi-process library OS (LibOS) for Intel SGX.
+As a LibOS, it enables legacy applications to run on SGX with little or even no modifications of source code, thus protecting the confidentiality and integrity of user workloads transparently.
+
+.. image:: images/arch_overview.png
+   :alt: Occlum Arch Overview
+
+.. note::
+
+   This project is under active development.
+   There is no guarantee that the content here are all up to date.
+
+*****************
+Table of Contents
+*****************
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Introduction
+
+   quickstart.md
+   occlum_cmds.md
+   occlum_configuration.md
+   build_and_install.md
+   install_occlum_packages.md
+   binaries_compatibility.md
+   how_to_debug.md
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Tutorials
+
+   tutorials/gen_occlum_instance.md
+   tutorials/distributed_pytorch.md
+   tutorials/occlum_ppml.md
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Inside Occlum
+
+   boot_flow.md
+   filesystem/fs_overview.md
+   filesystem/mount.md
+   filesystem/encrypted_image.md
+   remote_attestation.md
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Demos and Examples
+
+   demos/demos.md
+   tests/benchmark.md
+   tests/tests.md
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Tools
+
+   tools/toolchains.md
+   tools/copy_bom.md
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Q&A
+
+   qa.md

docs/readthedocs/docs/source/install_occlum_packages.md

@@ -0,0 +1,79 @@
+# Install Occlum with Popular Package Managers
+
+Occlum can be easily installed with popular package managers like [APT](https://en.wikipedia.org/wiki/APT_(software)) and [RPM](https://en.wikipedia.org/wiki/RPM_Package_Manager). This document walks you through the steps to install Occlum on popular Linux distributions like Ubuntu and CentOS using package managers.
+
+## Prerequisite
+
+**Install enable_RDFSBASE Kernel Module**
+
+If the Kernel version is before `v5.9`, please follow [this README](https://github.com/occlum/enable_rdfsbase/blob/master/README.md) to install `enable_rdfsbase` kernel module.
+
+## Install Occlum with APT on Ubuntu 20.04
+
+1. Install Prerequisite
+```
+apt update
+DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends ca-certificates gnupg2 jq make gdb wget libfuse-dev libtool tzdata rsync
+```
+
+2. Install Intel® SGX Driver and Intel® SGX PSW
+
+Please follow [Intel SGX Installation Guide](https://download.01.org/intel-sgx/sgx-linux/2.13/docs/Intel_SGX_Installation_Guide_Linux_2.13_Open_Source.pdf) to install SGX driver and SGX PSW. SGX SDK is not required. Using PSW installer is recommanded.
+
+To install PSW, follow the guide to add Intel® SGX repository to APT source. And then run:
+```
+apt-get update
+apt-get install -y libsgx-dcap-ql libsgx-epid libsgx-urts libsgx-quote-ex libsgx-uae-service libsgx-dcap-quote-verify-dev
+```
+
+After installing PSW, please make sure `aesm` service is in `active (running)` state by checking:
+```
+service aesmd status
+```
+
+3. Install Occlum
+```
+echo 'deb [arch=amd64] https://occlum.io/occlum-package-repos/debian bionic main' | tee /etc/apt/sources.list.d/occlum.list
+wget -qO - https://occlum.io/occlum-package-repos/debian/public.key | apt-key add -
+apt-get update
+apt-get install -y occlum
+```
+
+### Occlum toolchains packages
+
+Besides, users can choose to install the toolchain installer based on the application's language. Currently, Occlum supports only `musl-gcc`, `glibc`. Users can install each one on demand.
+
+```
+apt install -y occlum-toolchains-gcc
+apt install -y occlum-toolchains-glibc
+```
+
+### Occlum Runtime package
+
+If users only expect to run the Occlum instance image, then `occlum-runtime` package is better choice for size reason.
+```
+apt install -y occlum-runtime
+```
+
+
+## Version Compatability Matrix
+
+When version is not specified, Occlum with the latest version will be installed. If a user would like to evaluate an older version, please make sure the corresponding Intel® SGX PSW is installed.
+
+The matrix below shows the version compatability since Occlum `0.16.0`. Please check before installing or upgrading.
+
+| Occlum Version  |  SGX PSW Version  | Tested under Ubuntu |
+| --------------- | ----------------- | ------------------- |
+|     0.16.0      |       2.11        |        18.04        |
+|     0.17.0      |       2.11        |        18.04        |
+|     0.18.1      |       2.11        |        18.04        |
+|     0.19.1      |       2.11        |        18.04        |
+|     0.20.0      |       2.11        |        18.04        |
+|     0.21.0      |       2.13        |        18.04        |
+|     0.23.1      |       2.13.3      |        18.04        |
+|     0.24.1      |       2.14        |        18.04        |
+|     0.27.3      |       2.15.1      |        20.04        |
+|     0.28.1      |       2.16        |        20.04        |
+|     0.29.0      |       2.17.1      |        20.04        |
+
+For more information about the packages, please checkout [here](../tools/installer/README.md).

docs/readthedocs/docs/source/occlum_cmds.md

@@ -0,0 +1,86 @@
+# User Commands
+
+## Work Flow Overview
+
+![Workflow Overview](./images/flow_overview.png)
+
+## General Commands
+
+Occlum provides easy user commands as below.
+
+```bash
+occlum new <path>
+```
+Create a new directory at <path> and initialize as the Occlum instance.
+
+```bash
+occlum init
+```
+Initialize a directory as the Occlum instance.
+
+```bash
+occlum build [--sign-key <key_path>] [--sign-tool <tool_path>] [--image-key <key_path>] [-f/--force]
+```
+Build and sign an Occlum SGX enclave (.so) and generate its associated secure FS image according to the user-provided image directory and Occlum.yaml config file.
+The whole building process is incremental: the building artifacts are built only
+when needed.
+To force rebuilding all artifacts, give the [-f/--force] flag.
+
+```bash
+occlum run [--cpus <num_of_cpus>] <program_name> <program_args>
+```
+Run the user program inside an SGX enclave.
+
+```bash
+occlum package [<package_name>.tar.gz]
+```
+Generate a minimal, self-contained package (.tar.gz) for the Occlum instance.
+The resulting package can then be copied to a deployment environment and unpacked
+as a runnable Occlum instance.
+
+All runtime dependencies required by the Occlum instance, except Intel SGX driver and Intel SGX PSW, are included in the package.
+
+If package_name is not specified, the directory name of Occlum instance will be used.
+In default only HW release mode package is supported. Debug or simulation mode package
+could be supported by adding "--debug" flag.
+
+```bash
+occlum gdb <program_name> <program_args>
+```
+Debug the program running inside an SGX enclave with GDB.
+
+```bash
+occlum mount [--sign-key <key_path>] [--sign-tool <tool_path>] [--image-key <key_path>] <path>
+```
+Mount the secure FS image of the Occlum instance as a Linux FS at an existing <path>.
+This makes it easy to access and manipulate Occlum's secure FS for debug purpose.
+
+```bash
+occlum gen-image-key <key_path>
+```
+Generate a file consists of a randomly generated 128-bit key for encryption of the FS image.
+
+## Container-like Commands
+
+Occlum has added several new experimental commands, which provide a more container-like experience to users, as shown below:
+
+```bash
+occlum start
+```
+Start an Occlum instance, completing all the initialization including LibOS boots, Init FS and application root FS mount. A background service is started to listen which application is going to be executed.
+
+```bash
+occlum exec [cmd1] [args1]
+```
+Actually start executing the application.
+
+```bash
+occlum exec [cmd2] [args2]
+occlum exec [cmd3] [args3]
+```
+If there are more executable application binaries in the Occlum instance entrypoint, users could start executing them in parallel.
+
+```bash
+occlum stop
+```
+Stop the Occlum instance including the background listening service.

docs/readthedocs/docs/source/occlum_configuration.md

@@ -0,0 +1,179 @@
+# Occlum Configuration
+
+Occlum can be configured easily via a configuration file named `Occlum.yaml`, which is generated by the `occlum init` command in the Occlum instance directory. The user can modify `Occlum.yaml` to configure Occlum. The template of `Occlum.yaml` is shown below.
+
+```yaml
+# All the settings below can only take effect after `occlum build`.
+resource_limits:
+  # The number of OS threads created for this instance.
+  num_of_cpus: 128
+  # The stack size of LibOS kernel.
+  kernel_space_stack_size: 16MB
+  # The heap size of LibOS kernel.
+  # With EDMM support, choose "init" size based on the expected initialization time. And increase the "max" field if memory insufficiency occurs.
+  # Without EDMM support, increase the "init" field if memory insufficiency occurs. And the "max" field is ignored.
+  kernel_space_heap_size:
+    # Reserved and committed during the initialization stage. The more, the longer it takes to initialize.
+    init: 32MB
+    # Only committed when necessary. Only valid with EDMM support.
+    max:  512MB
+  # The total size of enclave memory available to the user applications running in LibOS.
+  # With EDMM support, choose "init" size based on the expected initialization time. And increase the "max" field if memory insufficiency occurs.
+  # Without EDMM support, increase the "init" field if memory insufficiency occurs. And the "max" field is ignored.
+  user_space_size:
+    # Reserved and committed during the initialization stage. The more, the longer it takes to initialize.
+    init: 256MB
+    # Only committed when necessary. Only valid with EDMM support.
+    max:  64GB
+process:
+  # Default stack size for each process. If not sure, don't modify it.
+  default_stack_size: 4MB
+  # Default heap size for each process. If not sure, don't modify it.
+  default_heap_size: 32MB
+
+# Entry points
+# Specify all valid absolute <path> in `occlum run <path> <args>`.
+# This prevents outside attackers from executing arbitrary commands inside an
+# Occlum-powered enclave.
+entry_points:
+  - /bin
+
+# Environment variables
+#
+# This gives a list of environment variables for the "root" process started
+# by `occlum run` or `occlum exec` command.
+env:
+  # The default env vars given to each "root" LibOS process. As these env vars
+  # are specified in this config file, they are considered trusted.
+  default:
+    - OCCLUM=yes
+  # The untrusted env vars that are captured by Occlum from the host environment
+  # and passed to the "root" LibOS processes. These untrusted env vars can
+  # override the trusted, default envs specified above.
+  # For example, `OCCLUM=no occlum run xxx` can pass the env OCCLUM=no to the process
+  # running in LibOS with below settings.
+  #     env:
+  #       default:
+  #         - OCCLUM=yes
+  #       untrusted:
+  #         - OCCLUM
+  untrusted:
+    - EXAMPLE
+
+# Enclave metadata
+# If not sure, just keep them no change
+metadata:
+  # Enclave signature structure's ISVPRODID field
+  product_id: 0
+  # Enclave signature structure's ISVSVN field
+  version_number: 0
+  # Whether the enclave is debuggable through special SGX instructions.
+  # If set to false, no log could be output.
+  # For production enclave, it is IMPORTANT to set this value to false.
+  debuggable: true
+  # SGX Key Separation and Sharing feature support.
+  # Please always keep it as true.
+  enable_kss: true
+  # Enclave signature structure's ISVEXTPRODID field.
+  # It is separated as two 16 bytes strings.
+  ext_prod_id:
+    high: '0x0'
+    low: '0x0'
+  # Whether to turn on PKU feature in Occlum
+  # Occlum uses PKU for isolation between LibOS and userspace program,
+  # It is useful for developers to detect potential bugs.
+  #
+  # "pkru" = 0: PKU feature must be disabled
+  # "pkru" = 1: PKU feature must be enabled
+  # "pkru" = 2: PKU feature is enabled if the platform supports it
+  pkru: 0
+
+# Mount points and their file systems
+mount:
+  # RootFS mount point and file systems.
+  # Generally, just don't change it.
+  - target: /
+    type: unionfs
+    options:
+      layers:
+        # The read-only layer which is generated in `occlum build`
+        - target: /
+          type: sefs
+          source: ./build/mount/__ROOT
+          options:
+            MAC: ''
+        # The read-write layer whose content is produced when running the LibOS
+        - target: /
+          type: sefs
+          source: ./run/mount/__ROOT
+          # options:
+          # The SEFS just uses MRSIGNER to drived key by default, so the data can
+          # be shared between same enclave signer on the same machine.
+          # If you want more secure scheme, you can set the autokey_policy field
+          # to 1 to enforcing the key is derived by MRSIGNER|MRENCLAVE|ISVFAMILYID.
+          # So the data can only be accessed by the enclave itself.
+          #  autokey_policy: 1
+  #
+  # HostFS mount
+  # It provides a channel to exchange files between Host FS and LibOS FS.
+  # Just note, with this mount, the files written to the target in the LibOS
+  # is recognizable to users in the Host thus may introduce security issue.
+  #
+  # For example, below section mount the occlum_instance directory in the Host FS
+  # to the path /host in the LibOS FS.
+  # It is disabled in default. Uncomment it if you are very sure about it.
+  # - target: /host
+  #   type: hostfs
+  #   source: .
+  #
+  # Async FS mount
+  # - target: /sfs
+  #   type: async_sfs
+  #   source: ./run/async_sfs_image
+  #   options:
+  #     total_size: 4GB
+
+# Untrusted Unix domain socket
+# Besides the common Unix domain socket support that both ends are created in the same
+# Occlum instance, the Untrusted Unix domain socket can support cross-world connection by
+# mapping a libos socket address with the host socket address. In this way, a Unix domain
+# socket created in Occlum can communicate with a Unix domain socket in the host OS or in
+# another Occlum instance.
+# Uncomment and customize below if you are very sure about it.
+#
+# untrusted_unix_socks:
+# The libos path and the host path correspond to the same host sock file.
+# The host side should use the host path and libos side should use the libos path.
+#
+# Specify socket file in absolute path
+  # - host: /tmp/occlum/test.sock
+  #   libos: /root/test.sock
+# Or any files in the path
+  # - host: /tmp/root/
+  #   libos: /root/
+# Or the host path can be a relative path and is relative to the current Occlum instance dir.
+  # - host: ../test.sock
+  #   libos: /tmp/test.sock
+```
+
+## Runtime Resource Configuration for Occlum process
+
+Occlum has enabled per process resource configuration via [prlimit](https://man7.org/linux/man-pages//man2/prlimit.2.html) syscall and shell built-in command [ulimit](https://fishshell.com/docs/current/cmds/ulimit.html).
+
+```shell
+#! /usr/bin/bash
+ulimit -a
+
+# ulimit defined below will override configuration in Occlum.yaml
+ulimit -Ss 10240 # stack size 10M
+ulimit -Sd 40960 # heap size 40M
+ulimit -Sv 102400 # virtual memory size 100M (including heap, stack, mmap size)
+
+echo "ulimit result:"
+ulimit -a
+
+# Run applications with the new resource limits
+...
+```
+
+For more info, please check [demos/fish](https://github.com/occlum/occlum/tree/master/demos/fish).

docs/readthedocs/docs/source/qa.md

@@ -0,0 +1,47 @@
+# Q&A
+
+### Does an Occlum instance directory correspond to only one binary, or does an Occlum instance directory contain multiple binaries? If yes, do they all belong to the security zone?
+
+Occlum is a multiprocess LibOS, which means that a user could add multiple executables into one Occlum instance and all those applications are able to work together. Technically, all those applications are running in the same Enclave, so there are in the same security zone.
+
+### If there are two executables in an Occlum instance, is it possible to execute both of them by using occlum_pal_exec()?
+
+The short answer is yes. But the user could only run one of them by occlum_pal_exec() at a time. If you want to run both of them at the same time, one of the application could spawn the other, or you could prepare a script to launch them in the script one by one.
+
+### Is there a way to share memory between an Enclave with Occlum instance?
+
+No, Occlum does not support shared memory with other Occlum instances or enclaves.
+
+### How many CPU cores can be used inside Occlum/Enclave?
+
+The host OS manages the CPU resource, and doing the scheduling. So it is totally controlled by host OS that how many cpus are running the applications inside Occlum.
+
+### Can Occlum support running network related applications?
+
+Yes. Generally, the network related applications can run successfully in Occlum without modification. Just one note, besides the application itself, multiple files/directories may be required to be existed in Occlum image as well. For example,
+
+* **`hostname`, `hosts` and `resolv.conf` files**
+
+Generally, these files (in the host environment) are automatically parsed and transferred to Occlum LibOS for each `Occlum run` operation.
+
+* **DNS related files**
+
+Add below part in the bom file if required.
+```
+  - target: /opt/occlum/glibc/lib
+    copy:
+      - files:
+          - /opt/occlum/glibc/lib/libnss_files.so.2
+          - /opt/occlum/glibc/lib/libnss_dns.so.2
+          - /opt/occlum/glibc/lib/libresolv.so.2
+```
+
+* **CA related files**
+
+Add below part in the bom file if required.
+```bom.yaml
+  - target: /etc
+    copy:
+      - dirs:
+        - /etc/ssl
+```

docs/readthedocs/docs/source/quickstart.md

@@ -0,0 +1,102 @@
+# Quick Start
+
+## Environment 
+
+### Supported HW
+
+First, please make sure the baremetal or VM machine support SGX. Otherwise, users can only try SW simulation mode.
+
+From Occlum v1.0, only SGX2 or SGX1 with [FLC](https://www.intel.com/content/www/us/en/developer/articles/technical/an-update-on-3rd-party-attestation.html)(Flexible Launch Control) feature are supported. 
+
+Users can use command `cpuid` to detect if the HW satisfy Occlum v1.0 requirements.
+
+* SGX2, ```cpuid | grep SGX2```
+* FLC, ```cpuid | grep SGX_LC```
+
+### Prerequisites
+
+#### Kernel Version
+
+From Occlum v1.0, the `io_uring` is used to enhance the IO performance. The `io_uring` is first introduced in Linux kernel 5.1 and still fast growing. To make Occlum v1.0 works well with `io_uring` and also the SGX in-tree driver, Linux kernel 5.10+ is expected.
+
+For example, in the Ubuntu 20.04 OS (Occlum default development OS), users could update the kernel with below command to get all Occlum required kernel features.
+```
+$ sudo apt install --install-recommends linux-generic-hwe-20.04
+```
+
+#### Start the Occlum dev container
+To give Occlum a quick try, one can use the Occlum docker image by following below steps:
+
+Step 1 is to be done on the host OS (Linux).
+Step 2-3 are to be done on the guest OS running inside the Docker container.
+
+1. Run the Occlum docker container, which has Occlum and its demos preinstalled:
+    ```bash
+    # 1. Create softlinks on host
+    mkdir -p /dev/sgx
+    ln -sf ../sgx_enclave /dev/sgx/enclave
+    ln -sf ../sgx_provision /dev/sgx/provision
+
+    # 2. Create container in two methods:
+    # (1) With privileged mode
+    docker run -it --privileged -v /dev/sgx:/dev/sgx occlum/occlum:[version]-ubuntu20.04
+
+    # (1) With non-privileged mode
+    docker run -it --device /dev/sgx/enclave --device /dev/sgx/provision occlum/occlum:[version]-ubuntu20.04
+    ```
+
+2. (Optional) Try the sample code of Intel SGX SDK to make sure that SGX is working
+    ```
+    cd /opt/intel/sgxsdk/SampleCode/SampleEnclave && make && ./app
+    ```
+3. Check out Occlum's demos preinstalled at `/root/demos`. Or you can try to build and run your own SGX-protected applications using Occlum as shown in the demos.
+
+## Hello World
+
+If you were to write an SGX Hello World project using some SGX SDK, the project would consist of hundreds of lines of code. And to do that, you have to spend a great deal of time to learn the APIs, the programming model, and the build system of the SGX SDK.
+
+Thanks to Occlum, you can be freed from writing any extra SGX-aware code and only need to type some simple commands to protect your application with SGX transparently --- in four easy steps.
+
+**Step 1. Compile the user program with the Occlum toolchain (e.g., `occlum-gcc`)**
+```
+$ occlum-gcc -o hello_world hello_world.c
+$ ./hello_world
+Hello World
+```
+And Occlum can support `gcc` compile as well. The difference is that the binaries generated by `occlum-gcc` are musl-libc based, but are glibc based if compiled by `gcc`.
+
+Note that the Occlum toolchain is not cross-compiling in the traditional sense: the binaries built by the Occlum toolchain is also runnable on Linux. This property makes it convenient to compile, debug, and test user programs intended for Occlum.
+
+**Step 2. Initialize a directory as the Occlum instance via `occlum init` or `occlum new`**
+```
+$ mkdir occlum_instance && cd occlum_instance
+$ occlum init
+```
+or
+```
+$ occlum new occlum_instance
+```
+The `occlum init` command creates the compile-time and run-time state of Occlum in the current working directory. The `occlum new` command does basically the same thing but in a new instance directory. Each Occlum instance directory should be used for a single instance of an application; multiple applications or different instances of a single application should use different Occlum instances.
+
+**Step 3. Generate a secure Occlum FS image and Occlum SGX enclave via `occlum build`**
+```
+$ cp ../hello_world image/bin/
+$ occlum build
+```
+The content of the `image` directory is initialized by the `occlum init` command. The structure of the `image` directory mimics that of an ordinary UNIX FS, containing directories like `/bin`, `/lib`, `/root`, `/tmp`, etc. After copying the user program `hello_world` into `image/bin/`, the `image` directory is packaged by the `occlum build` command to generate a secure Occlum FS image as well as the Occlum SGX enclave.
+
+For platforms that don't support SGX, it is also possible to run Occlum in SGX simulation mode. To switch to the simulation mode, `occlum build` command must be given an extra argument or an environment variable as shown below:
+```
+$ occlum build --sgx-mode SIM
+```
+or
+```
+$ SGX_MODE=SIM occlum build
+```
+
+**Step 4. Run the user program inside an SGX enclave via `occlum run`**
+```
+$ occlum run /bin/hello_world
+Hello World!
+```
+The `occlum run` command starts up an Occlum SGX enclave, which, behind the scene, verifies and loads the associated Occlum FS image, spawns a new LibOS process to execute `/bin/hello_world`, and eventually prints the message.

docs/readthedocs/docs/source/remote_attestation.md

@@ -0,0 +1,70 @@
+# Remote Attestation
+
+Occlum provides wrapped library `libocclum_dcap` for `DCAP` remote attestion applications.
+This Occlum DCAP library is prebuilt as part of the Occlum toolchains in the [Occlum Docker images](https://hub.docker.com/r/occlum/occlum).
+
+The libraries are in the path `/opt/occlum/toolchains/dcap_lib`.
+```
+.
+|-- glibc
+|   |-- dcap_test
+|   |-- libocclum_dcap.a
+|   `-- libocclum_dcap.so
+|-- inc
+|   `-- occlum_dcap.h
+`-- musl
+    |-- dcap_test
+    |-- libocclum_dcap.a
+    `-- libocclum_dcap.so
+```
+
+Two versions (glibc and musl-libc), including static and dynamic libraries are provided to meet different scenarios. Unified header file `occlum_dcap.h` is provided as well in which defines the exported APIs for DCAP quote generation and verification.
+
+In short, applications can link to the prebuilt `libocclum_dcap.so` and use the APIs defined in `occlum_dcap.h` for their usage.
+
+For details how to use the library, please refer to the [demo](https://github.com/occlum/occlum/tree/master/demos/remote_attestation/dcap).
+
+The source code of the library is in the [path](https://github.com/occlum/occlum/tools/toolchains/dcap_lib/).
+
+Occlum also has a unique "Occlum -> init ->application" boot flow. Generally, all operation which is application required but not part of the application, such as remote attestation, could be put into `init` part. This feature makes Occlum highly compatible to any remote attestation solution without involving application’s change.
+
+![init_ra_flow](./images/ra_init.png)
+
+This design off load the remote attestation burden from application. Two RA solutions are provided for reference.
+
+## Init-RA solution
+
+It is based on a GRPC-RATLS implementation.
+Details please refer to the demo [init_ra_flow](https://github.com/occlum/occlum/tree/master/demos/remote_attestation/init_ra_flow).
+
+## Azure Attestation
+
+To support Azure Attestation, there are some demos provided. Users could choose each one to their actual applications. Details please refer to the demo [azure_attestation](https://github.com/occlum/occlum/tree/master/demos/remote_attestation/azure_attestation).
+
+## SGX KSS (Key Separation and Sharing feature) support
+
+Starting from SGX2, there is a new Key Separation and Sharing feature which provides more  flexibility. The new feature gives user a chance to fill in some meaningful information to the enclave either in the signing or running stage.
+
+* Signning stage:
+```
+ISVFAMILYID, 16 bytes
+ISVEXTPRODID, 16 bytes
+```
+* Running stage:
+```
+CONFIG ID, 64 bytes
+CONFIG SVN, 16 bits
+```
+Occlum can support both above by either modifying the fields in `Occlum.yaml` (for `Signning stage`) or using Occlum run arguments `"--config-id"` or `"--config-svn"` (for `Running stage`).
+
+* Note: `ISVFAMILYID` is reserved for saving the MAC of the configuration file.
+
+Details please refer to the [RFC](https://github.com/occlum/occlum/issues/589).
+
+
+## References
+
+- [DCAP Quick Install Guide](https://software.intel.com/content/www/us/en/develop/articles/intel-software-guard-extensions-data-center-attestation-primitives-quick-install-guide.html)
+
+- [Intel(R) Software Guard Extensions Data Center Attestation Primitives](https://github.com/intel/SGXDataCenterAttestationPrimitives)
+

docs/readthedocs/docs/source/tests/benchmark.md

@@ -0,0 +1,19 @@
+# Benchmark Demos
+
+This set of demos shows how commonly used benchmarking tools can be run inside SGX enclaves with Occlum.
+
+* [filebench](filebench/): A demo of [Filebench](https://github.com/filebench/filebench), a file system and storage benchmark tool.
+* [fio](fio/): A demo of [Flexible I/O Tester](https://github.com/axboe/fio).
+* [iperf2](iperf2/): A demo of [Iperf2](https://sourceforge.net/projects/iperf2/), a tool for measuring Internet bandwidth performance.
+* [iperf3](iperf3/): A demo of [Iperf3](https://github.com/esnet/iperf), a tool for measuring Internet bandwidth performance.
+* [sysbench](sysbench/): A demo of [Sysbench](https://github.com/akopytov/sysbench), a scriptable multi-threaded benchmark tool for Linux.
+
+## Benchmarks Data
+
+There is a enabled [benchmarks CI](https://github.com/occlum/occlum/blob/1.0.0-preview/.github/workflows/benchmarks.yml) for continuous benchmarking. It utilizes the [github-action-benchmark](https://github.com/benchmark-action/github-action-benchmark) to provide a chart view for visualized historical benchmarks data on the GitHub pages.
+
+The CI runs periodically. For example, **sysbench** has the historical benchmarks chart as below.
+
+![sysbench_chart](../images/benchmark.png)
+
+[**History Data**](https://occlum.io/occlum/dev/benchmarks/)

docs/readthedocs/docs/source/tests/tests.md

@@ -0,0 +1,31 @@
+# Tests for Occlum
+
+## Unit Tests
+
+There is a set of unit tests in the source [**test**](https://github.com/occlum/occlum/tree/master/test). It includes almost all the syscall (Occlum supported) test cases. Every PR will run this unit tests to make sure of no failures introduced on basic functions.
+
+Users could run the unit test manually as well.
+
+```
+// run all the unit test cases for musl-libc
+# make test
+
+// run all the unit test cases for glibc
+# make test-glibc
+
+// run only specified test case, timerfd for example
+# TESTS=timerfd make test
+
+// run test cases for 100 times
+# make test times=100
+
+// run test without rebuilding Occlum, using binaries installed already
+# OCCLUM_BIN_PATH=/opt/occlum/build/bin make test
+```
+
+## Gvisor Tests
+<To be added>
+
+## LTP Tests
+
+Linux [LTP](https://github.com/linux-test-project/ltp) is the most popular test suite for Linux. Occlum could also apply the LTP to verify the stability and compatibility to Linux app. For detail, please refer to [**linux-ltp**](https://github.com/occlum/occlum/tree/master/demos/linux-ltp).

docs/readthedocs/docs/source/tools/copy_bom.md

@@ -0,0 +1,57 @@
+# Copy Bom
+The `copy_bom` tool is designed to copy files described in a bom file to a given dest root directory.
+
+## Bom file
+Bom file is used to describe which files should be copied to the root directory(usually, the image directory). A bom file contains all files, directories that should be copied to the root directory. We also can define symbolic links and directories that should be created in a bom file. The meanings of each entry in the bom file can be seen in [RFC: bom file](https://github.com/occlum/occlum/issues/565). Also, a bom example with all optional entries can be seen in `example.yaml`.
+
+## copy_bom
+### overview
+`copy_bom` is the tool designed to create directories and symbolic links, copy all files and directories defined in a bom file to the root directory. Internally, `copy_bom` will use `rsync` to do the real file operations. `copy_bom` will copy each file and directory incrementally, i.e., only changed parts will be copied. The permission bits and modification times will be reserved. This is done by the `-a` option of `rsync`. `copy_bom` will not ensure the whole image directory as described in bom file (sync behavior) because it will not try to delete old files. To pursue a sync behavior, one can delete the old image directory and copy files again.
+
+### dependencies
+`copy_bom` will analyze all dependencies(shared objects) of each ELF file. `copy_bom` will analyze dependencies for each user-defined file in `files` entry as well as files in user-defined directory in `dirs` entry. For user-defined elf file, it will report error and abort the program if we can't find the dependent shared objects. For files in user-defined directories, we will report warning if autodep fails. We analyze dependencies via the dynamic loader defined in the `.interp` section in elf files and automatically copy dependencies to the root directory. If there's no `.interp` section for an elf file, `copy_bom` will try to infer the loader if all other elf files have the same loader. Currently, `copy_bom` only copy dependencies with absolute paths. We support only one dependency pattern in the result of dynamic loader.
+- name => path   e.g., `libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6`  
+All dependencies will be copied to the corresponding directory in root directory. For example, if root directory is `image`, then the dependency `/lib64/ld-linux-x86-64.so.2` will be copied to `image/lib64/ld-linux-x86-64.so.2`. An entry named `autodep` with value `false` can be added to each file to avoid finding and copying dependencies automatically.
+
+### log
+`copy_bom` uses the same log setting as `occlum`. One can set `OCCLUM_LOG_LEVEL=trace` to see all logs printed by `copy_bom`. To only view real file operations, `OCCLUM_LOG_LEVEL=info` is a proper level.
+
+### prepare and install
+* **Prepare**
+
+Since `copy_bom` relies on `rsync` to copy files. We need to install `rsync` at first. On ubuntu, this can be done by `apt install rsync -y`.
+* **Install**.
+
+`copy_bom` is part of the occlum tools which could be found on /opt/occlum/build/bin. The occlum tools are preinstalled in occlum development docker image or installed by `apt install -y occlum`.
+
+### basic usage
+`copy_bom [FLAGS] [OPTIONS] --file <bom-file> --root <root-dir>`
+- bom-file: The bom file which describes files we want to copy.
+- root-dir: The destination root directory we want to copy files to. Usually the `image` directory for occlum.
+
+### options
+- dry run mode: pass an option `--dry-run` to `copy_bom` will enable dry run mode. Dry run mode will output all file operations in log but does not do real operations. It is useful to check whether `copy_bom` performs as expected before doing real operations.
+
+### flags
+- -i, --include-dir: This flag is used to indicate which directory to find included bom files. This flag can be set multiple times. If the `include-dir` is set as a relative path, it is a path relative to the current path where you run the `copy_bom` command. 
+- -h, --help: print help message
+
+### About the `occlum_elf_loader.config` file
+
+This file is put in `/etc/template`. This file is used to define where to find occlum-specific loaders and occlum-specific libraries. If you want to find libraries in different paths, you should modify this config file.  
+
+The file content looks like as below:
+```
+/opt/occlum/glibc/lib/ld-linux-x86-64.so.2 /usr/lib/x86_64-linux-gnu:/lib/x86_64-linux-gnu 
+/lib/ld-musl-x86_64.so.1 /opt/occlum/toolchains/gcc/x86_64-linux-musl/lib
+```
+Each line in this file represents a loader. Since occlum supports both musl and glibc loader, there are two loaders in the config file now.  
+Each line contains two parts, which is separated with a space. The first part is the path of occlum-specific loader. This loader is used to analyze dependencies of elf file.  
+The second part in the line indicates where to find shared libraries. All paths should be separated by colons. The loader will first try to find libraries in the loader path, then will try to find libraries in user-provided path. This is done by set the `LD_LIBRARY_PATH` environmental variables. The order of paths matters, since we will find libraries in the order of given path.
+
+## known limitations
+
+- The use of wildcard(like *) in files or directories is not supported. It may result in `copy_bom` behaving incorrectly.
+- If we create symbolic link in bom file, it will always delete the old link and create a new one. It will change the modification time of the symbolic link.
+- Environmental variables pointing to an empty value may fail to resolve.
+

docs/readthedocs/docs/source/tools/toolchains.md

@@ -0,0 +1,114 @@
+# Builtin Toolchains
+
+Occlum provides multiple builtin toolchains or libraries in the Occlum development docker image to ease the porting or developing effort for users. Part of them are toolchains used to recompile or repackage the applications to make them runnable in Occlum TEE environment. The others are some frequently-used auxiliary libraries which can be directly used or linked by applications running in Occlum.
+
+All the build scripts could be found on [github](https://github.com/occlum/occlum/tree/master/tools/toolchains).
+
+## Toolchains
+
+Generally, Occlum supports both musl-libc and glibc based toolchains. It is up to the users to descide which one to be chosen.
+
+* Less memory footprint, perhaps musl-libc based.
+* More compatible to existed applications, maybe glibc based.
+
+### glibc
+
+To support running glibc based application in Occlum, a customized [glibc](https://github.com/occlum/glibc) is provided in the Occlum development docker image.
+
+|    libc   |  Compatible Version in Occlum  | Path in Occlum Docker Image |
+| --------- | ------------------------------ | --------------------------- |
+|   glibc   | <=2.31<br>(default version in Ubuntu:20.04)  |  /opt/occlum/glibc/lib/ |
+
+The main changes on top of general glibc are as below:
+
+* Redirect syscalls into Occlum
+* Support posix_spawn syscall for Occlum
+* Modify vdso to calling syscalls in Occlum
+
+To users, all the glibc libraries to be used in Occlum need to be replaced by the ones in ```/opt/occlum/glibc/lib/```.
+
+Thus, Occlum can support `gcc` compilation with **PIE** enabled, which make it compatible to popular compile systems.
+
+### golang
+ 
+To support compiling and running Golang in Occlum LibOS, a customized [go](https://github.com/occlum/go) is provided in the Occlum development docker image, path `/opt/occlum/toolchains/golang`. Every Golang to be executed in Occlum needs to be recompiled by **occlum-go** (a wrapper of go).
+
+Currently Occlum supports two versions of Golang, 1.16 and 1.18 (default one in Occlum development docker image) which are both linked to musl-libc in default. But users can easily configure the **occlum-go** to use **gcc** like below.
+```bash
+# CC=gcc occlum-go build
+```
+
+### java
+
+There are three JAVA versions provided in the Occlum development docker image, path `/opt/occlum/toolchains/jvm`.
+
+|    JDK   |  libc  | Path in Occlum Docker Image |
+| --------- | ------ | --------------------------- |
+| openjdk8 |  musl-libc |  /opt/occlum/toolchains/jvm/java-1.8-openjdk |
+| openjdk11 |  musl-libc |  /opt/occlum/toolchains/jvm/java-11-openjdk |
+| dragonwell-jdk11 |  musl-libc |  /opt/occlum/toolchains/jvm/java-11-alibaba-dragonwell |
+
+Both the unmodified openjdk 8/11 in the table are directly imported from Alpine:3.11.
+
+The dragonwell-jdk11 for enclave is a musl-based JDK version compatible with the Alpine Linux and Occlum, and it's an open source project, see [Alibaba Dragonwell](https://github.com/alibaba/dragonwell11/tree/dragonwell-for-enclave).
+
+By default Occlum uses Dragonwell JDK11 as the default JDK. Thus `occlum-java` and `occlum-javac` (provided in the Docker image path **/opt/occlum/toolchains/jvm/bin/**) use Dragonwell JDK11. 
+
+Besides the musl-libc based JDK (less memory footprint), glibc based JDK are also supported. Users are free to change to other JDK version by setting the `JAVA_HOME` to point to the installation directory of OpenJDK and copying it into Occlum instance.
+
+### musl-gcc
+
+To support compiling and running musl-libc based application in Occlum, a customized [musl](https://github.com/occlum/musl) is provided in the Occlum development docker image.
+
+|    libc   |  Compatible Version in Occlum  | Path in Occlum Docker Image |
+| --------- | ------------------------------ | --------------------------- |
+| musl libc | <=1.1.24<br>(default version in Alpine:3.11) | /usr/local/occlum/x86_64-linux-musl/lib/ |
+
+To users, all the musl-libc libraries to be used in Occlum need to be replaced by the ones in ```/usr/local/occlum/x86_64-linux-musl/lib/```.
+
+Moreover, wrapped **occlum-gcc**, **occlum-g++** and **occlum-ld** are provided as well to do the recompiling if necessary. Any applications generated by these wrapped tools, are expected to run successfully in Occlum.
+
+### rust
+
+Occlum supports general glibc based rust tools such as **cargo** and **rustc**.
+
+Wrapped **occlum-cargo** and **occlum-rustc** are also provided to do musl-libc based rust compilation, which can be found on the path `/opt/occlum/toolchains/rust/bin/` in the Occlum development docker image.
+
+## Auxiliary Libraries
+
+Besides toolchains, several auxiliary libraries to be used in Occlum are provided as well to ease the development effort.
+
+### bash
+
+To support running bash shell script in Occlum, a customized [bash](https://github.com/occlum/bash) is provided in the Occlum development docker image. Both musl-libc and glibc versions are provided in the path `/opt/occlum/toolchains/bash`. Users can use it directly for application in Occlum, details could refer to demo [bash](https://github.com/occlum/occlum/tree/master/demos/bash).
+
+### busybox
+
+To support running general CLI commands in Occlum, a prebuilt busybox is provided in the Occlum development docker image. Both musl-libc and glibc versions are provided in the path `/opt/occlum/toolchains/busybox`. Users can use it directly for application in Occlum, details could refer to demo [bash](https://github.com/occlum/occlum/tree/master/demos/bash).
+
+### DCAP library
+
+Occlum provides wrapped library `libocclum_dcap` for `DCAP` remote attestion applications.
+Both musl-libc and glibc versions are provided in the path `/opt/occlum/toolchains/dcap_lib`
+
+```
+.
+|-- glibc
+|   |-- dcap_test
+|   |-- libocclum_dcap.a
+|   `-- libocclum_dcap.so
+|-- inc
+|   `-- occlum_dcap.h
+`-- musl
+    |-- dcap_test
+    |-- libocclum_dcap.a
+    `-- libocclum_dcap.so
+```
+
+Two versions (glibc and musl-libc), including static and dynamic libraries are provided to meet different scenarios. Unified header file `occlum_dcap.h` is provided as well in which defines the exported APIs for DCAP quote generation and verification.
+
+In short, applications can link to the prebuilt `libocclum_dcap.so` and use the APIs defined in `occlum_dcap.h` for their usage.
+
+For details how to use the library, please refer to the [demo](https://github.com/occlum/occlum/tree/master/demos/remote_attestation/dcap).
+
+The source code of the library is in the [path](https://github.com/occlum/occlum/tools/toolchains/dcap_lib/).

docs/readthedocs/docs/source/tutorials/distributed_pytorch.md

@@ -0,0 +1,3 @@
+# Distributed PyTorch
+
+To be Added

docs/readthedocs/docs/source/tutorials/gen_occlum_instance.md

@@ -0,0 +1,101 @@
+# Insight of Occlum Instance Generation
+
+For every application to be running in Occlum (TEE env), all the running required files, libraries and binaries have to be put into Occlum file system. Here is the tree view of one Occlum instance.
+```
+./occlum_instance/
+|-- Occlum.yaml
+|-- build
+|-- image       // Occlum root file system
+|   |-- bin
+|   |-- dev
+|   |-- etc
+|   |-- host
+|   |-- lib
+|   |-- lib64
+|   |-- opt
+|   |-- proc
+|   |-- root
+|   |-- sfs
+|   |-- sys
+|   `-- tmp
+|-- initfs      // Occlum init file system
+|   |-- bin
+|   |-- dev
+|   |-- etc
+|   |-- lib
+|   `-- proc
+`-- run
+```
+
+## File System from Occlum Perspective
+
+Let's clarify some definitions users usually get confused.
+* **host file system**
+
+Host means the environment where users run Occlum build. Usually it is the Occlum official docker image. In this environment, users build and prepare all files to be running in Occlum.
+
+* **Occlum init file system**
+
+Occlum has a unique **Occlum -> init ->application** boot flow, please check [boot_flow](https://occlum.readthedocs.io/en/latest/boot_flow.html) for detail. So the Occlum init file system is what the Occlum **init** process sees. In develop stage, it is in the path `occlum_instance/initfs`. Generally, a `occlum new` generates a default init file system, users don't need modify this part unless you know exactly what you are doing.
+
+* **Occlum root file system**
+
+It is the file system the real application sees. And it is also the place users need put all the running required files, libraries and binaries. In develop stage, it is in the path `occlum_instance/image`.
+
+In summary, to generate Occlum instance, one important step is to copy application running required files from **host file system** to **Occlum root file system**.
+
+Next, it is an example of using **copy_bom** to ease the Occlum root file system creation.
+
+## Redis in Occlum
+
+There is a redis demo in [github](https://github.com/occlum/occlum/tree/master/demos/redis) which is built from source. Actually, users could use the OS installed redis binary directly to generate a runnable Occlum Redis instance by following steps.
+
+* Install the redis by **apt**.
+```
+apt install redis-server
+```
+
+* Create a **redis.yaml** to assist generate redis Occlum instance.
+```redis.yaml
+includes:
+  - base.yaml
+targets:
+  - target: /bin
+    copy:
+      - files:
+        - /usr/bin/redis-server
+```
+
+* Generate and build redis Occlum instance
+```
+occlum new redis_instance
+cd redis_instance
+copy_bom -f ../redis.yaml --root image --include-dir /opt/occlum/etc/template
+occlum build
+```
+
+*  Run the redis server in Occlum
+```
+occlum run /bin/redis-server
+```
+
+The whole flow is like below.
+![redis](../images/redis_build.png)
+
+Very easy and straightforward, right?
+
+Next let's explore the magic of **copy_bom**.
+
+## **copy_bom** Case Study
+
+The `copy_bom` tool is designed to copy files described in a bom file to a given dest root directory. For details users could refer to [page](https://occlum.readthedocs.io/en/latest/tools/copy_bom.html).
+
+The most important and useful function of copy_bom is the automatic dependencies finding and copy. For the redis case, there are so many dependent libraries the `redis-server` required for running.
+
+![LDD for redis-server](../images/redis_dependencies.png)
+
+All the dependent libraries above have to be copied to **occlum_instance**. But the **redis.yaml** showed above just has `redis-server`. How come it is running well in Occlum? 
+
+That is because the `copy_bom` would detect binaries or libraries defined in the yaml file, find all the dependencies and copy them to the corresponding path in **occlum_instance/image**. For this case, all required libraries would be in place after `copy_bom` operation.
+
+![Occlum redis tree](../images/redis_tree.png)

docs/readthedocs/docs/source/tutorials/occlum_ppml.md

@@ -0,0 +1,17 @@
+# Secure Spark Data Analytics using BigDL PPML and Occlum
+
+Protecting privacy and confidentiality is critical for large-scale data analysis and machine learning. Occlum collaborates with BigDL [PPML](https://bigdl.readthedocs.io/en/latest/doc/PPML/Overview/intro.html), provides a Trusted Cluster Environment for secure Big Data & AI applications, even on untrusted cloud environment. The solution ensures end-to-end security enabled for the entire distributed Spark workflows.
+
+## Overall Architecture
+
+![Architecture](../images/occlum_ppml.png)
+
+## PPML Occlum Spark on Azure
+
+Please check the [link](https://bigdl.readthedocs.io/en/latest/doc/PPML/Overview/azure_ppml.html#) for all the details.
+
+Also, there is a introduction document published in [Azure Confidential Computing Blog](https://techcommunity.microsoft.com/t5/azure-confidential-computing/bigdl-privacy-preserving-machine-learning-with-occlum-oss-on/ba-p/3658667), demonstrates the PPML Occlum solution using a sample analytics application built for the NYTaxi dataset.
+
+## PPML Occlum Spark on Aliyun (Chinese version)
+
+It is a detail how-to and technical insight about deploying the PPML Occlum Spark solution on Aliyun. Please check the [link](https://bigdl.readthedocs.io/en/latest/doc/PPML/Overview/ali_ecs_occlum_cn.html) for all the details.

docs/readthedocs/pyproject.toml

@@ -0,0 +1,8 @@
+[build-system]
+requires = ["flit_core >=3.2,<4"]
+build-backend = "flit_core.buildapi"
+
+[project]
+name = "Occlum"
+authors = [{name = "Qi Zheng", email = "huaiqing.zq@antgroup.com"}]
+dynamic = ["version", "description"]

docs/readthedocs/requirements.txt

@@ -0,0 +1,12 @@
+breathe<4.13.0
+toml>=0.10
+
+pillow
+mock==1.0.1
+alabaster>=0.7,<0.8,!=0.7.5
+commonmark==0.9.1
+recommonmark==0.5.0
+sphinx
+sphinx-rtd-theme
+readthedocs-sphinx-ext<2.2
+sphinx_markdown_tables