1. Five new ioctl commands of /dev/sgx are added for occlum
applications to securely get and verify DCAP quote;
2. Not all the functions of the intel DCAP package are open to
developers to simplify the DCAP usage;
3. The test may only run on the platform with DCAP driver installed;
4. A macro OCCLUM_DISABLE_DCAP is used to separate the DCAP code from
the other code.
5. Skip DCAP test when DCAP driver is not detected or in simulation mode
Before this commit, the epoll implementation works by simply delegating to the
host OS through OCall. One major problem with this implementation is
that it can only handle files that are backed by a file of the host OS
(e.g., sockets), but not those are are mainly implemented by the LibOS
(e.g., pipes). Therefore, a new epoll implementation that can handle all
kinds of files is needed.
This commit completely rewrites the epoll implementation by leveraging
the new event subsystem. Now the new epoll can handle all file types:
1. Host files, e.g., sockets, eventfd;
2. LibOS files, e.g., pipes;
3. Hybrid files, e.g., epoll files.
For a new file type to support epoll, it only neends to implement no
more than four methods of the File trait:
* poll (required for all file types);
* notifier (required for all file files);
* host_fd (only required for host files);
* recv_host_events (only required for host files).
An event can be anything ranging from the exit of a process (interesting
to `wait4`) to the arrival of a blocked signal (interesting to
`sigwaitinfo`), from the completion of a file operation (interesting to
`epoll`) to the change of a file status (interesting to `inotify`).
To meet the event-related demands from various subsystems, this event
subsystem is designed to provide a set of general-purpose primitives:
* `Waiter`, `Waker`, and `WaiterQueue` are primitives to put threads
to sleep and later wake them up.
* `Event`, `Observer`, and `Notifier` are primitives to handle and
broadcast events.
* `WaiterQueueObserver` implements the common pattern of waking up
threads once some interesting events happen.
Socket-related ocalls, e.g, sendto, sendmsg and write, may cause SIGPIPE
in host. Since the ocall is called by libos, this kind of signal should
be handled in libos. We ignore SIGPIPE in host and raise the same signal
in libos if the return value of the above ocalls is EPIPE. In this way
the signal is handled by libos.
This commit mainly accomplish two things:
1. Use makefile to manage dependencies for `occlum build`, which can save lots of time
2. Take dirs `build`, `run` outside from `.occlum`. Remove env var "OCCLUM_INSTANCE_DIR"
The new interrupt subsystem breaks the simulation mode in two ways:
1. The signal 64 is not handled by Intel SGX SDK in simulation mode. A
handled real-time signal crashes the process.
2. The newly-enabled test case exit_group depends on interrupts. But
enclave interrupts, like enclave exceptions, are not supported in
simulation mode.
This commit ensures signal 64 is ignored by default and exit_group test
case is not enabled in simulation mode.
Before this commit, events like signals and exit_group are handled by
LibOS threads in a cooperative fashion: if the user code executed by a
LibOS thread does not invoke system calls (e.g., a busy loop), then the LibOS
won't have any opportunity to take control and handle events.
With the help from the POSIX signal-based interrupt mechanism of
Occlum's version of Intel SGX SDK, the LibOS can now interrupt the
execution of arbitrary user code in a LibOS thread by sending real-time
POSIX signals (the signal number is 64) to it. These signals are sent by
a helper thread spawn by Occlum PAL. The helper thread periodically
enters into the enclave to check if there are any LibOS threads with
pending events. If any, the helper thread broadcast POSIX signals to
them. When interrupted by a signal, the receiver LibOS thread may be in
one of the two previously problematic states in terms of event handling:
1. Executing non-cooperative user code (e.g., a busy loop). In this
case, the signal will trigger an interrupt handler inside the enclave,
which can then enter the LibOS kernel to deal with any pending events.
2. Executing an OCall that invokes blocking system calls (e.g., futex,
nanosleep, or blocking I/O). In this case, the signal will interrupt the
blocking system call so that the OCall can return back to the enclave.
Thanks to the new interrupt subsystem, some event-based system calls
are made robust. One such example is exit_group. We can now guarantee
that exit_group can force any thread in a process to exit.
On lightweight Linux distribution, like alpine, getpwuid()
returns NULL, and errno is ENOENT, this patch fix crash
caused by this situation.
Signed-off-by: Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
Add "untrusted" sections for environment variables defined in Occlum.json. Environment
variable defined in "default" will be shown in libos directly. Environment variable
defined in "untrusted" can be passed from occlum run or PAL layer and can override
the value in "default" and thus is considered "untrusted".
Before this commit, the three ECalls of the LibOS enclave do not give
the exact reason on error. In this commit, we modify the enclave entry code
to return the errno and list all possible values of errno in Enclave.edl.
In this commit, we add eight signal-related syscalls
* kill
* tkill
* tgkill
* rt_sigaction
* rt_sigreturn
* rt_sigprocmask
* rt_sigpending
* exit_group
We implement the following major features for signals:
* Generate, mask, and deliver signals
* Support user-defined signal handlers
* Support nested invocation of signal handlers
* Support passing arguments: signum, sigaction, and ucontext
* Support both process-directed and thread-directed signals
* Capture hardware exceptions and convert them to signals
* Deliver fatal signals (like SIGKILL) to kill processes gracefully
But we still have gaps, including but not limited to the points below:
* Convert #PF (page fault) and #GP (general protection) exceptions to signals
* Force delivery of signals via interrupt
* Support simulation mode
This commits improves both readability and correctness of the scheduling-related
system calls. In terms of readability, it extracts all scheduling-related code
ouf of the process/ directory and put it in a sched/ directory. In terms
of correctness, the new scheduling subsystem introduces CpuSet and SchedAgent
types to maintain and manipulate CPU scheduler settings in a secure and robust way.
Now one can specify the log level of the LibOS by setting `OCCLUM_LOG_LEVEL`
environment variable. The possible values are "off", "error", "warn",
"info", and "trace".
However, for the sake of security, the log level of a release enclave
(DisableDebug = 1 in Enclave.xml) is always "off" (i.e., no log) regardless of
the log level specified by the untrusted environment.
It is slow to allocate big buffers using SGX SDK's malloc. Even worse, it
consumes a large amount of precious trusted memory inside enclaves. This
commit avoids using trusted buffers and allocates untrusted buffers for
sendmsg/recvmsg directly via OCall, thus improving the performance of
sendmsg/recvmsg. Note that this optimization does not affect the security of
network data as it has to be sent/received via OCalls.
Before this commit, using custom C types in ECalls/OCalls defined in Occlum's
EDL is cumbersme. Now this issue is resolved by providing `occlum_edl_types.h`
header file. There are two versions of this file: one is under
`src/libos/include/edl/` for LibOS, the other is under
`src/pal/include/edl/` for PAL. So now to define a new custom C type, just
edit the two versions of `occlum_edl_types.h` to define the type.
SGX SDK's sgx_init_quote may return SGX_ERROR_BUSY, which is previously not
handled. The implementation of ioctl for /dev/sgx is now fixed to handle this
error.
By providing Occlum PAL as a shared library, it is now possible to embed and
use Occlum in an user-controled process (instead of an Occlum-controlled one).
The APIs of Occlum PAL can be found in `src/pal/include/occlum_pal_api.h`. The
Occlum PAL library, namely `libocclum-pal.so`, can be found in `.occlum/build/lib`.
To use the library, check out the source code of `occlum-run` (under
`src/run`), which can be seen as a sample code for using the Occlum PAL
library.
1. Add a separate net/ directory for the network subsystem;
2. Move some existing socket code to net/;
3. Implement sendmsg/recvmsg with OCalls;
4. Extend client/server test cases.
BACKGROUND
The exit_group syscall, which is implicitly called by libc after the main function
returns, kills all threads in a thread group, even if these threads are
running, sleeping, or waiting on a futex.
PROBLEM
In normal use cases, exit_group does nothing since a well-written program
should terminate all threads before the main function returns. But when this is
not the case, exit_group can clean up the mess.
Currently, Occlum does not implement exit_group. And the Occlum PAL process
waits for all tasks (i.e., SGX threads) to finish before exiting. So without
exit_group implemented, some tasks may be still running if after the main task
exits. And this causes the Occlum PAL process to wait---forever.
WORKAROUND
To implement a real exit_group, we need signals to kill threads. But we do not
have signals, yet. So we come up with a workaround: instead of waiting all
tasks to finish in PAL, we just wait for the main task. As soon as the main
task exits, the PAL process terminates, killing the remaining tasks.
* 'occlum init' does not copy signing key file any more.
* 'occlum build' supports to set signing key and signing tool in args.
* 'occlum run' supports to run enclave in sgx release mode.
1. All generated, build files are now in a separate build directory;
2. The CLI tool supports three sub-commands: init, build, and run;
3. Refactor tests to use the new tool.
There are two types of stacks: the kernel ones and the user ones. The kernel
stacks are used by Occlum and managed by Intel SGX SDK itself, while the user
stacks are used by the threads created and managed by Occlum. These user stacks
are transparent to Intel SGX SDK so far.
The problem is that Intel SGX SDK needs to be aware of the user stacks.
SGX exception handlers will check whether the rsp value---when the exception
happened---is within the stack of the current SGX thread. If the check fails,
the registered exception handler will not be triggered. But when exceptions are
triggered by the threads running upon Occlum, the rsp value points to the user
stacks, which Intel SGX SDK are completely unware of. So the check always
fails.
Therefore, we extend Intel SGX SDK with two new APIs:
int sgx_enable_user_stack(size_t stack_base, size_t stack_limit);
void sgx_disable_user_stack(void);
And this commit uses the two APIs to inform Intel SGX SDK about the
Occlum-managed stacks. And the rsp checks in SGX exception handlers will
check whether rsp is within the user stacks.
