Container security

Gremlin containers run as root

When Gremlin runs within a container, gremlin processes run as root. This is because there are several file resources that Gremlin must mount from the host which are owned by root on the host machine:

• /var/lib/gremlin and /var/log/gremlin: These are mounted from the host into the Gremlin agent container and sidecar experiments. By default, container runtimes will set the ownership of these resources to root:root
• Container runtime sockets (e.g. /var/run/docker.sock), which by default will be owned by root
• /proc/sysrq-trigger and /sys/fs/cgroup for attack capabilities, which by default will be owned by root

Mitigating the privileges of root

Though Gremlin runs as root within the container in which it runs, Gremlin is restricted by the specific capabilities it requests, despite running as root. Here are some other mitigations system administrators can make to reduce the impact of Gremlin's usage of root:

• While Docker's default seccomp profile is already fairly restrictive, you can remove any undesired syscalls by overriding this default, or provide a more restrictive seccomp profile to Gremlin containers.
• Restrict Gremlin's resource access on the host using AppArmor and the Gremlin Agent AppArmor Profile.
• Avoid running Gremlin within the network or process namespace of the host. For Docker, avoid docker run --net=host --pid=host. Note that this will disable Gremlin's ability to run host-level Network and Process Killer experiments.
• Consider an extra layer of safety by enabling SELinux to harden all containers running on your systems

Alternatives

If you do not wish to run Gremlin as a root user, even with the restrictions described above, consider installing Gremlin directly onto the host where it runs under a dedicated Linux user: gremlin.

Docker (Linux)

User namespace isolation

Gremlin currently uses the host's file system to store temporary log and state information about experiments. When running Docker with user namespace remapping (userns-remap), Gremlin needs to assume the user namespace of the host. This applies for both the gremlin daemon container as well as when running gremlin experiment-container. Note that by assuming the user namespace of the host, we are creating an exception to backspace isolation for the Docker containers running Gremlin.

For running the Gremlin daemon in a container

1docker run -d \
2    --userns-remap=host \
3    -e GREMLIN_BYPASS_USERNS_REMAP=1 \
4    -v /var/lib/gremlin:/var/lib/gremlin \
5    -v /var/log/gremlin:/var/log/gremlin \
6    gremlin/gremlin daemon

For running the Gremlin daemon on the host

1echo "GREMLIN_BYPASS_USERNS_REMAP=1" | sudo tee -a /etc/default/gremlind
2sudo systemctl restart gremlind

For running a Gremlin experiment from the command line

1export GREMLIN_BYPASS_USERNS_REMAP=1
2gremlin attack-container 38dbd9016529 cpu

SELinux and Gremlin in Containers

Gremlin performs some actions that are not allowed by the default SELinux process label for containers (container_t):

• Install and manipulate files on the host: /var/lib/gremlin, /var/log/gremlin
• Load kernel modules for manipulating network transactions during network experiments, such as net_sch
• Communicate with the container runtime socket (e.g. /var/run/docker.sock) to launch containers that carry out experiments
• Read files in /proc

Bypass container_t restrictions

It is possible to alleviate these restrictions on container_t by installing the following policy. However, this grants the privileges required by Gremlin to all other containers on your system that use container_t.

If you wish to run Gremlin with the container_t process label, and bypass its restrictions, supply the following type enforcement rules into a new SELinux policy:

1# WARNING: This policy adds capabilities to all containers run under the default type: container_t
2# Gremlin needs access to /var/log/gremlin
3allow container_t container_log_t:dir { read write create getattr setattr unlink link add_name remove_name rmdir open };
4allow container_t container_log_t:file { read write create getattr setattr append unlink link open };
5allow container_t var_log_t:dir { write add_name };
6
7# Gremlin needs access to /var/run/docker.sock
8allow container_t container_runtime_t:unix_stream_socket connectto;
9
10# Gremlin needs access to /var/lib/gremlin
11allow container_t container_var_lib_t:dir { read write create getattr setattr unlink link add_name remove_name rmdir open };
12allow container_t container_var_lib_t:file { read write create getattr setattr append unlink link open };
13
14# Gremlin needs to load the kernel modules: net_sch
15allow container_t kernel_t:system module_request;