Cloud Foundry Security: Do Containers Contain?

New Cloud Foundry users are often worried about how secure containers are. Many ask if it is possible to leave container boundaries or, e.g., for one application to take up all CPU resources, etc. In this post, I will explain what is under the hood of CF containers to answer these questions.

Understanding CF containers

Cloud Foundry containers are basically Linux containers. Note that they are not the same as LXC, which were used initially, but were later replaced with another implementation (discussed below) due to their limitations. Linux containers are an OS-level virtualization environment for running multiple isolated Linux systems (containers) on a single Linux control host. Containers are based on namespaces and cgroups—both of which are Linux kernel features.

A namespace wraps a global system resource in an abstraction that makes it appear to the processes within the namespace as if they had their own isolated instance of the global resource. Changes to the global resource are visible to processes that are members of the namespace, but are invisible to other processes. One of the use cases for namespaces is to implement containers.

Cgroups (abbreviated from control groups) is a Linux kernel feature that limits, accounts for, and isolates resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes.

Cloud Foundry’s current container implementation is called Warden. Warden manages isolated, ephemeral, and resource controlled environments. The project’s primary goal is to provide a simple API for managing isolated environments—or containers—that can be limited in terms of CPU usage, memory usage, disk usage, and network access.

New-generation Cloud Foundry uses Garden—an implementation of Warden in Go, which features backends not only for Linux-based apps, but also for the .NET stack and Docker images.

Read more in this post: Cloud Foundry Containers: Warden, Docker, and Garden.

How processes are isolated

Warden creates isolated environments, using namespaces, which make it is possible to have multiple “nested” process trees—each with an entirely independent set of processes. This can ensure that processes belonging to one process tree cannot inspect or kill—in fact, cannot even know about the existence of—processes in other sibling or parent process trees.

Network namespaces allow each of these processes to see an entirely different set of networking interfaces. Even the loopback interface is different for each network namespace. The following diagram is an example of relations between parent and child namespaces.

Garden, the runtime of the “new-generation” Cloud Foundry, goes even further and implements the Linux user namespaces, so getting privileges inside a container doesn’t mean getting the same privileges on the host.

How mature are these features? Namespaces were introduced in Linux kernel more than eight years ago and cgroups were first released in 2007. CoreOS, Docker, Hadoop, and other projects use cgroups as their basis.

Can one app take all resources?

Resource control is handled by Control Groups (cgroups). Every container is placed in its own control group, where it is limited in the amount of memory requested for the container and a proportional amount of memory. I say proportional for simplicity, the actual algorithm is smarter and a bit more sophisticated.

Every container within CF gets a private root file system created by stacking a read-only and a read-write file system together. This is implemented with aufs on Ubuntu versions 10.04 to 11.10 and overlayfs on Ubuntu 12.04.

The read-only file system contains the minimal set of Ubuntu packages and Warden-specific modifications common to all containers. The read-write file system stores files, overriding container-specific settings, when necessary. Because all writes are applied to the read-write file system, containers can share the same read-only base file system.

The read-write file system is created by formatting a large sparse file. Since the size of this file is fixed, the file system that it contains cannot grow beyond this initial size. As a result, each container only gets access the allocated amount of resources, no more and no less.

Can CF containers be trusted?

Let’s sum it up. Cloud Foundry uses containers to ease deployment and management of apps and their dependencies, but the isolation that containers provide also results in increased security, because it reduces the degree to which applications can interact with each other. Containers are based on namespaces and cgroups—core Linux kernel features that have been around for years and serve as the basis for many popular projects. So, we believe that you can trust CF containers as much as you can trust the Linux kernel itself.

Further reading

• Cloud Foundry Security Overview
• Cloud Foundry Containers: Warden, Docker, and Garden
• Cloud Foundry’s Garden: Back Ends, Container Security, and Debugging