Add config guide for EDMM
This commit is contained in:
Hui, Chunyang
volcano
parent
587f4debb3
commit
af2643f9e0
127
docs/edmm/edmm_config_guide.md
Normal file
127
docs/edmm/edmm_config_guide.md
Normal file
@ -0,0 +1,127 @@
|
|||||||
|
# OCCLUM EDMM Configuration Guide
|
||||||
|
|
||||||
|
## EDMM Environment Requirements
|
||||||
|
1. Hardware Platform: SGX-2 support is required.
|
||||||
|
2. OS: Linux Kernel v6.0 or above is needed. And v6.2 is primarily used for development and testing.
|
||||||
|
|
||||||
|
## EDMM Enabling
|
||||||
|
EDMM feature is not enabled by default. To enable it, set `ENABLE_EDMM` during the Occlum build phase like below:
|
||||||
|
```bash
|
||||||
|
ENABLE_EDMM=Y occlum build
|
||||||
|
occlum run /xxx/xxx
|
||||||
|
```
|
||||||
|
If not enabled, Occlum will be built as running on the non-EDMM platform by default.
|
||||||
|
|
||||||
|
## Occlum.json Configuration
|
||||||
|
|
||||||
|
### Design Philosophy
|
||||||
|
A single Occlum.json file can be built for both EDMM and non-EDMM platforms while maximizing the EDMM functionality on the EDMM platform.
|
||||||
|
|
||||||
|
### EDMM Principle
|
||||||
|
To leverage EDMM, the `Enclave.xml` file offers two types of configuration options, namely `Init` and `Max`, for memory-related settings including TCS, heap, and stack (some may include a third option, `Min`). These options enable the Enclave to load a smaller memory footprint during the initialization phase, thereby enhancing the startup speed of the Enclave application. The remaining memory is loaded progressively at runtime in an on-demand manner. The `Enclave.xml` file is generated from the `Occlum.json` file using Occlum's internal configuration tool and serves as the definitive configuration file for the Enclave. To fully harness the capabilities of EDMM, additional configuration options are provided in the `Occlum.json` file.
|
||||||
|
|
||||||
|
The configuration strategy is as follows: to achieve a faster Occlum startup speed, configure a smaller value for the `Init` memory. In cases of insufficient memory availability, the size of the `Max` memory should be increased. The `Init` memory is loaded during Occlum's startup irrespective of its actual usage, while the `Max` memory determines the maximum memory requirement of the application and is dynamically loaded during runtime to avoid memory wastage.
|
||||||
|
|
||||||
|
### Occlum.json EDMM Configuration Introduction
|
||||||
|
In general, three optional fields have been added to the existing Occlum.json configuration: `kernel_space_heap_max_size`, `user_space_max_size`, and `init_num_of_threads`. An example of the memory section configuration in `Occlum.json` when all the EDMM-related configurations are enabled:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"resource_limits": {
|
||||||
|
"kernel_space_stack_size": "1MB", // (Legacy, Required)
|
||||||
|
"kernel_space_heap_size": "4MB", // (Legacy, Required)
|
||||||
|
"kernel_space_heap_max_size": "40MB", // !!! (Newly-Introduced，Optional)
|
||||||
|
"user_space_size": "1MB", // (Legacy, Required)
|
||||||
|
"user_space_max_size": "600MB", // !!! (Newly-Introduced，Optional)
|
||||||
|
"init_num_of_threads": 2, // !!! (Newly-Introduced，Optional)
|
||||||
|
"max_num_of_threads": 64 // (Legacy, Required)
|
||||||
|
},
|
||||||
|
"process": {
|
||||||
|
"default_stack_size": "4MB", // (Legacy, Required)
|
||||||
|
"default_heap_size": "8MB", // (Legacy, Required)
|
||||||
|
"default_mmap_size": "100MB" // (Legacy, Required, but inoperative)
|
||||||
|
},
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**`occlum build` will fail if any of the `Required` field is not provided.**
|
||||||
|
|
||||||
|
### Detail Explanation
|
||||||
|
|
||||||
|
[]()
|
||||||
|
[]()
|
||||||
|
[-lightgrey?style=for-the-badge)]()
|
||||||
|
|
||||||
|
|
||||||
|
#### Kernel Stack
|
||||||
|
|
||||||
|
[]() **corresponds to the Occlum kernel space stack memory and maintain consistency with the existing configuration method**
|
||||||
|
|
||||||
|
- corresponds to StackMaxSize, StackMinSize of `Enclave.xml` file
|
||||||
|
- Due to the fact that this stack is only intended for Occlum kernel threads, the memory requirement is not substantial. Typically, it ranges from 1 to 4MB, and in the majority of cases, no modifications are necessary. Consequently, no additional configuration options are provided
|
||||||
|
- The recommended configuration approach is to keep it consistent with the previous settings without any modifications
|
||||||
|
|
||||||
|
|
||||||
|
#### Kernel Heap
|
||||||
|
|
||||||
|
[]() []() **corresponds to the Occlum kernel space heap memory**
|
||||||
|
|
||||||
|
- If only []() is configured
|
||||||
|
- On the EDMM platform, the default [-lightgrey?style=flat)]() provided by the Occlum configuration tool serves as the maximum value for the heap allocated on-demand, while []() represents the initial kernel heap size during LibOS initialization
|
||||||
|
- On non-EDMM platforms, []() remains consistent with its previous usage, corresponding to the static size of the kernel space heap
|
||||||
|
|
||||||
|
- If []() is newly added
|
||||||
|
- On the EDMM platform, the maximum value provided by []() is compared with the default [-lightgrey?style=flat)]() from the Occlum configuration tool to determine the maximum value for the kernel heap. Meanwhile, []() represents the initial kernel heap size during LibOS initialization
|
||||||
|
- On non-EDMM platforms, []() is used as the static size of the kernel space heap, and []() **no longer takes effect**
|
||||||
|
|
||||||
|
- Recommended configuration approach:
|
||||||
|
- On the EDMM platform: Add []() configuration field and can increase the amount a little bit compared to the original [](). The []() can be reduced appropriately based on the desired startup time. In theory, a smaller []() configuration leads to faster LibOS startup speed.
|
||||||
|
- On non-EDMM platforms: Keep the configuration consistent with the previous settings, i.e., no modifications to [](). Alternatively, adding []() configuration field can provide future scalability.
|
||||||
|
|
||||||
|
|
||||||
|
#### User Space
|
||||||
|
|
||||||
|
[]() []() **corresponds to Occlum's user space memory**
|
||||||
|
|
||||||
|
- If only []() is configured
|
||||||
|
- On the EDMM platform, the default [-lightgray?style=flat)]() provided by the Occlum configuration tool serves as the maximum value for the user space allocated on-demand, while []() represents the initial user space size during LibOS initialization
|
||||||
|
- On non-EDMM platforms, []() remains consistent with its previous usage, corresponding to the static size of the user space
|
||||||
|
|
||||||
|
- If []() is newly added
|
||||||
|
- On the EDMM platform, the maximum value provided by []() is compared with the default [-lightgray?style=flat)]() from the Occlum configuration tool to determine the maximum value for the user space. Meanwhile, []() represents the initial user space size during LibOS initialization
|
||||||
|
- On non-EDMM platforms, []() is used as the static size of the user space, and []() **no longer takes effect**
|
||||||
|
|
||||||
|
- Recommended configuration approach:
|
||||||
|
- On the EDMM platform: since we only pay for what we use, user space memory becomes more affordable. []() can be configured to be relatively large, such as multiplying the previous value by `2`, to prevent the application OOM (out of memory) issues. The original []() can be reduced appropriately based on the desired startup time. In theory, a smaller []() configuration leads to faster LibOS startup speed
|
||||||
|
- On non-EDMM platforms: Keep the configuration consistent with the previous settings, i.e., no modifications to [](). Alternatively, consider adding []() configuration for future scalability
|
||||||
|
|
||||||
|
|
||||||
|
#### TCS Number
|
||||||
|
|
||||||
|
[]() []() **corresponds to the total amount of threads used by LibOS kernel and user space**
|
||||||
|
|
||||||
|
- Unlike the configuration of kernel heap and user space, the configuration for the TCS number introduces a separate configuration for the `Init` value, while retaining the original configuration for the `Max` number of threads
|
||||||
|
|
||||||
|
- If only []() is configured
|
||||||
|
- On the EDMM platform, the Occlum configuration tool provides a default value of [-lightgray?style=flat)]() for the number of `Init` threads, while []() is compared with the default value of [-lightgray?style=flat)]() from the Occlum configuration tool to determine the maximum number of threads
|
||||||
|
- On non-EDMM platforms, []() remains consistent with its previous usage, corresponding to the static number of threads
|
||||||
|
|
||||||
|
- If []() is newly added
|
||||||
|
- On the EDMM platform, the minimum value of []() compared with the default value of [-lightgray?style=flat)]() provided by the Occlum configuration tool is used as the number of `Init` threads during initialization, while []() is compared with the default value of [-lightgray?style=flat)]() from the Occlum configuration tool to determine the maximum number of threads
|
||||||
|
- On non-EDMM platforms, []() does not take effect, and only []() is effective, remaining consistent with its previous usage, corresponding to the static number of threads
|
||||||
|
|
||||||
|
- Recommended configuration approach
|
||||||
|
- On the EDMM platform: Add the []() configuration field and set a smaller value based on the desired startup time. In theory, a smaller []() configuration leads to faster LibOS startup speed. []() can remain the same as the previous configuration
|
||||||
|
- On non-EDMM platforms: Keep the configuration consistent with the previous settings, i.e., no modifications to []() .
|
||||||
|
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
1. When running Occlum on the EDMM platform, it will automatically make maximum use of the EDMM functionality. In the current implementation, even if the user manually configures []() or [](), it may still exceed this limit and use the default system-provided [](). Therefore, limiting the `Max` setting cannot effectively restrict the maximum physical memory usage of the application. However, these physical memory will not be wasted if they are not used, since the memory is allocated on-demand.
|
||||||
|
|
||||||
|
2. If compatibility needs to be considered, i.e., an `Occlum.json` file needs to run on both EDMM and non-EDMM platforms, it is advisable not to set the max-related values (including []() []() []() ) too large. On the EDMM platform, due to memory being committed on demand, setting a large `Max` value does not affect normal program execution or cause wastage. However, on non-EDMM platforms, where memory is fully committed, a large `Max` value may result in slower LibOS startup speed and could even cause Enclave loading failures.
|
||||||
|
|
||||||
|
3. The environment variable `SGX_MODE` takes precedence over `ENABLE_EDMM`. This means EDMM cannot be enabled in the SGX Simulation mode. If both `ENABLE_EDMM=Y` and `SGX_MODE=SIM` are configured, only the `SGX_MODE` environment variable takes effect.
|
||||||
|
|
||||||
|
4. Users should try to avoid conflicts between the EDMM configuration during `occlum build` and the actual execution environment during `occlum run`. If `occlum build` and `occlum run` are performed in different environments:
|
||||||
|
- If EDMM is not enabled during `occlum build` but the environment supports EDMM during `occlum run`: Occlum will not enable EDMM capabilities and will run with the non-EDMM configuration
|
||||||
|
- If EDMM is enabled during `occlum build` but the environment does not support EDMM during `occlum run`: The Enclave can still start, but since the memory usage of the configuration file is targeted for the EDMM environment, undefined errors may occur in the non-EDMM environment. This includes explicit errors or panics due to insufficient memory
|
||||||
126
docs/edmm/edmm_config_guide_zh_CN.md
Normal file
126
docs/edmm/edmm_config_guide_zh_CN.md
Normal file
@ -0,0 +1,126 @@
|
|||||||
|
# OCCLUM EDMM 配置介绍
|
||||||
|
|
||||||
|
## EDMM环境要求
|
||||||
|
1. 硬件平台：需要支持SGX-2
|
||||||
|
2. OS: 需要Linux Kernel v6.0及以上版本，目前的开发测试主要使用的是v6.2
|
||||||
|
|
||||||
|
## EDMM启用方法
|
||||||
|
EDMM功能默认不启用。如需开启，在Occlum build阶段通过设置`ENABLE_EDMM`启用：
|
||||||
|
```bash
|
||||||
|
ENABLE_EDMM=Y occlum build
|
||||||
|
occlum run /xxx/xxx
|
||||||
|
```
|
||||||
|
|
||||||
|
如果未启用，则默认按照非EDMM平台进行build以及配置。
|
||||||
|
|
||||||
|
## Occlum.json配置
|
||||||
|
|
||||||
|
### 配置设计原则
|
||||||
|
一份Occlum.json文件可以同时运行在EDMM以及非EDMM平台。并且在EDMM平台能够自动地最大限度地使用EDMM的功能。
|
||||||
|
|
||||||
|
### EDMM原理
|
||||||
|
在支持EDMM后，`Enclave.xml`对于包括TCS, heap, stack等内存相关配置都提供类似`Init`和`Max`的两个配置项（有的还有第三个配置项 Min）。它的意义就是可以使得Enclave在启动阶段能够加载比较少的内存，来提高Enclave应用的启动速度。而把其他内存的加载时间平摊在运行时。`Occlum.json`由Occlum内部的配置工具转换成`Enclave.xml`并作为Enclave的最终配置。为了充分利用EDMM能力，因此也补充了相关的配置项。
|
||||||
|
|
||||||
|
总体配置思路大概是：如果希望Occlum启动速度比较快，就配置较少的`Init`内存。而当内存不足时，应该增加`Max`内存大小。`Init`内存作为Occlum启动时加载的内存，不管有没有真正用到都会被加载。`Max`内存决定了应用需要的最大内存，即使不用也不会浪费，在运行时进行按需加载。
|
||||||
|
|
||||||
|
### Occlum.json EDMM相关配置示例
|
||||||
|
总体是在原有`Occlum.json`基础上增加了3个选填字段，包括`kernel_space_heap_max_size`, `user_space_max_size`, 以及`init_num_of_threads`。打开全部EDMM相关配置后的`Occlum.json`内存部分配置如下：
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"resource_limits": {
|
||||||
|
"kernel_space_stack_size": "1MB", // (已有，必填)
|
||||||
|
"kernel_space_heap_size": "4MB", // (已有，必填)
|
||||||
|
"kernel_space_heap_max_size": "40MB", // !!! (新增，选填)
|
||||||
|
"user_space_size": "1MB", // (已有，必填)
|
||||||
|
"user_space_max_size": "600MB", // !!! (新增，选填)
|
||||||
|
"init_num_of_threads": 2, // !!! (新增，选填)
|
||||||
|
"max_num_of_threads": 64 // (已有，必填)
|
||||||
|
},
|
||||||
|
"process": {
|
||||||
|
"default_stack_size": "4MB", // (已有，必填)
|
||||||
|
"default_heap_size": "8MB", // (已有，必填)
|
||||||
|
"default_mmap_size": "100MB" // (已有，必填，但不生效)
|
||||||
|
},
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**如果没有提供必填项，则occlum build会失败。**
|
||||||
|
|
||||||
|
### 详细解释
|
||||||
|
|
||||||
|
[]()
|
||||||
|
[]()
|
||||||
|
[]()
|
||||||
|
|
||||||
|
|
||||||
|
#### Kernel Stack
|
||||||
|
|
||||||
|
[]() **对应Occlum kernel space stack内存, 与原有配置方式保持一致**
|
||||||
|
|
||||||
|
- 对应`Enclave.xml`文件的StackMaxSize, StackMinSize (大小相同)
|
||||||
|
- 由于该stack只为Occlum kernel thread提供, 因此内存需求并不会很大。一般都在1~4M，并且绝大多数时候都不需要改动。因此不再提供其他配置项
|
||||||
|
- 推荐配置方式：与之前配置一致，不做修改
|
||||||
|
|
||||||
|
|
||||||
|
#### Kernel Heap
|
||||||
|
|
||||||
|
[]() []() **对应Occlum kernel space heap内存**
|
||||||
|
|
||||||
|
- 如果只配置了 []()
|
||||||
|
- 在EDMM平台，Occlum配置工具会提供缺省的 [-lightgrey?style=flat)]() 作为按需分配的heap最大值，而 []() 作为LibOS初始化时的kernel heap大小
|
||||||
|
- 在非EDMM平台，[]() 和之前的用法保持一致，对应静态的kernel space heap大小
|
||||||
|
|
||||||
|
- 如果增加了 []() 配置
|
||||||
|
- 在EDMM平台，由 []() 对比Occlum配置工具提供缺省的 [-lightgrey?style=flat)]() 的最大值作为kernel heap的最大值，而[]()作为LibOS初始化时的kernel heap大小
|
||||||
|
- 在非EDMM平台，使用 []() 作为静态的kernel space heap大小，[]() **不再生效**
|
||||||
|
|
||||||
|
- 推荐配置方式
|
||||||
|
- EDMM平台：增加 []() 配置，可以相比之前的 []() 增加一些。原 []() 可以根据期望的启动时间，适当减小。理论上 []() 配置的越小，LibOS启动速度越快
|
||||||
|
- 非EDMM平台：与之前配置一样，不做修改，即保持 []() 与之前一致。或者增加 []() 配置来提供未来的拓展性
|
||||||
|
|
||||||
|
|
||||||
|
#### User Space
|
||||||
|
|
||||||
|
[]() []() **对应Occlum user space 大小**
|
||||||
|
|
||||||
|
- 如果只配置了 []()
|
||||||
|
- 在EDMM平台，Occlum配置工具会提供缺省的 [-lightgray?style=flat)]() 作为按需分配的user space最大值，而[]() 作为LibOS初始化时的user space大小
|
||||||
|
- 在非EDMM平台，[]() 和之前的用法保持一致，对应静态的user space大小
|
||||||
|
|
||||||
|
- 如果增加了[]() 配置
|
||||||
|
- 在EDMM平台，由[]()对比Occlum配置工具提供缺省的 [-lightgray?style=flat)]() 的最大值作为user space的最大值，而 []() 作为LibOS初始化时的user space大小
|
||||||
|
- 在非EDMM平台，使用 []() 作为静态的user space大小，[]() **不再生效**
|
||||||
|
|
||||||
|
- 推荐配置方式
|
||||||
|
- EDMM平台：user space的内存变得廉价，所以 []() 可以配置的比较大，比如在之前的 []() 上乘以2倍，以杜绝应用OOM。原 []() 可以根据期望的启动时间，适当减小。理论上 []() 配置的越小，LibOS启动速度越快
|
||||||
|
- 非EDMM平台：与之前配置一样，不做修改，即保持 []() 与之前一致。也可以考虑增加 []() 配置来提供未来的拓展性
|
||||||
|
|
||||||
|
#### TCS Number
|
||||||
|
|
||||||
|
[]() []() **对应LibOS kernel线程和user线程数量的总和**
|
||||||
|
|
||||||
|
- 和kernel heap以及user space大小配置不同的是，对于线程数量的配置是增加了对`Init`数量的配置，而保留了原有的`Max`数量的配置
|
||||||
|
|
||||||
|
- 如果只配置了 []()
|
||||||
|
- 在EDMM平台，Occlum配置工具会提供缺省的 [-lightgray?style=flat)]() 作为`Init`数量的线程 ，而由 []() 对比Occlum配置工具提供缺省的 [-lightgray?style=flat)]() 的最大值作为线程数量的最大值
|
||||||
|
- 在非EDMM平台，[]() 和之前的用法保持一致，对应静态的线程数量
|
||||||
|
|
||||||
|
- 如果增加了 []() 配置
|
||||||
|
- 在EDMM平台，由 []() 对比Occlum配置工具提供的缺省 [-lightgray?style=flat)]() 的最小值作为初始化的线程数量，而 []() 对比Occlum配置工具提供缺省的 [-lightgray?style=flat)]() 的最大值作为线程数量的最大值
|
||||||
|
- 在非EDMM平台，[]() 不生效，只有 []() 生效，与之前用法保持一致，对应静态的线程数量
|
||||||
|
|
||||||
|
- 推荐配置方式
|
||||||
|
- EDMM平台：增加 []() 配置，可以根据期望的启动时间设定一个较小的值。理论上 []() 配置的越小，LibOS的启动速度越快。[]() 可以保持和之前的配置一样
|
||||||
|
- 非EDMM平台：与之前配置一样，不做修改，即保持 []() 与之前一致
|
||||||
|
|
||||||
|
|
||||||
|
## 需要注意
|
||||||
|
1. 运行在EDMM平台会自动地最大限度使用EDMM功能。在目前的实现中，即使用户手动配置了 []() 或者 []() , 也可能会超过这个限制而使用系统提供的 []()。因此无法通过限制`max_size`来限制应用的最大物理内存用量。由于是按需分配的，这些内存如果没有用到的话，是不会占用物理内存的。
|
||||||
|
|
||||||
|
2. 如果需要考虑兼容性，即一份`Occlum.json`需要同时运行在EDMM和非EDMM平台，则请勿将`Max`相关值（包括[]() []() []()）设置的过大。在EDMM平台，由于内存按需提交，过大的`Max`值不影响程序正常运行，也不会造成浪费。但是在非EDMM平台，由于内存全量提交，`Max`值配置的比较大会导致LibOS启动速度慢，如果过大甚至会导致加载Enclave失败。
|
||||||
|
|
||||||
|
3. 环境变量`SGX_MODE`优先级高于`ENABLE_EDMM`。即在Simulation模式下无法启用EDMM。 如果同时配置了`ENABLE_EDMM=Y SGX_MODE=SIM`则只有`SGX_MODE`环境变量生效。
|
||||||
|
|
||||||
|
4. 用户应尽力避免`occlum build`时对EDMM的配置与`occlum run`时实际的EDMM配置冲突。如果`occlum build`和`occlum run`在不同环境：
|
||||||
|
- `occlum build`时不启用EDMM, `occlum run`时环境支持EDMM: Occlum不会启用EDMM能力，依然按照非EDMM配置运行
|
||||||
|
- `occlum build`时启用EDMM, `occlum run`时环境不支持EDMM: Enclave可以启动，但是由于配置文件的内存用量是按照EDMM环境进行配置的，在非EDMM环境会出现未定义错误。包括可能由于内存不足报错退出或发生panic等
|
||||||
Loading…
Reference in New Issue
Block a user