Rule-Driven Automation on Kubernetes with Autopilot Monitoring
Common Kubernetes security issues
When running applications on Kubernetes, there are some common security problems that can be frequently missed. One of these mistakes involves running pods on a host volume. When this happens, data is left behind in the host machine even after the pod is terminated, leaving it vulnerable. Another issue is identifying whether or not a rogue container located on the host is accessing volumes attached to the host. These are just a few of the problems that can cause data vulnerability in Kubernetes.
While there are some security precautions available in Kubernetes, such as enabling role-based access control (RBAC) and encrypting persistent volumes, there are still limitations. In RBAC’s case, it can’t manage access to components that are not under the control of Kubernetes. Additionally, there are a few more loopholes that are difficult to secure:
- A pod may not successfully terminate holding a reference to the volume causing software failure.
- Left-over host mounts can be vulnerable and accessed by pods or malicious containers.
- Rogue containers bound to the host can access all the attached and mounted persistent volumes.
At a Kubernetes meetup in Santa Clara, Gou Rao and Aditya Dani of Portworx discussed how some of these security loopholes can be automatically patched using an open-source solution called Autopilot.
Gao Rou, CTO and co-founder of Portworx“When I’m accessing my e-mail at Gmail, I really don’t know what server it’s running on. I’m just going to assume that things are safe. The same thing applies here. When I’m running my database container in a 100-node Kubernetes cluster, I don’t know which machine it’s running on. I shouldn’t have to care whether that data is going to be accessed by somebody else or if it’s safe.” —Gou Rao, Portworx
What is Autopilot?
Autopilot is a rule-based analytical engine, which uses a monitor-and-react model. The solution takes its input from various metrics, logs, and tracers of the stateful applications like Postgres, Cassandra, and Redis. From this input, Autopilot can perform certain actions complying with a condition set. Both input rules and outcome actions are based on well-defined Kubernetes CustomResourceDefinitions (CRD).
Autopilot workflow (Image credit)“It’s basically a policy-based engine that’s looking out for the administrator’s back. If an application’s running rogue, if there’s a security violation, or if there’s a performance issue, its job is to step in and either raise a red flag or take action on it.” —Gou Rao, Portworx
Some of the actions Autopilot can perform include:
- Automatic persistent volume updates and relocation
- Automatic scaling of a volume by increasing or decreasing input/output operations per second
According to the Portworx team:
- Performance of an application and its containers at the required levels is ensured via monitoring.
- High availability is achieved through redundancy.
- Pod scaling and application-level rebalancing are supported out-of-the-box.
Aditya Dani, software developer at Portworx“You define an application-level policy, which is given as input to the Autopilot inference engine. The other input are metrics, logs, and tracers. It also talks to Kubernetes, then it does correlations based on the input and the timelines. Based on the conditions that have been defined in the policy, it’s going to perform an action. The action can be specific to an application or it can be a generic action.” —Aditya Dani, Portworx
Detecting breadcrumbs with Autopilot
To get a better understanding of Autopilot, the Portworx team referred to a common security problem in Kubernetes, which is detecting breadcrumbs (the data an application leaves on a node) and stopping rogue containers which try to access it.
Detecting breadcrumbs with Autopilot (Image credit)In their example, the speakers utilized and monitored cAdvisor, which provides metrics for resource usage and performance characteristics of running containers.
An example of a metric from cAdvisor (Image credit)In order to use the container_fs_read_bytes_total metric from cAdvisor, a Postgres volume security policy is defined. Under this policy, containers which aren’t part of the /kubepods Kubernetes cgroup are stopped if they try to access the breadcrumbs.
An example of a storage policy CRD (Image credit)“This is extensive. You can define your own actions and your own conditions. You can also define your own application-level specific policies.” —Aditya Dani, Portworx
Through the use of Kubernetes CRDs, the input and output policies of Autopilot can be heavily customized to meet different security and storage requirements. While the example presented here is about volume security, Autopilot can also be used to monitor and automate applications, as well as volume health. More examples and Autopilot’s development can be tracked in the project’s GitHub repo.
Want details? Watch the video!
These are the slides presented.
Further reading
- Tips for Implementing Shift-Left Security on Kubernetes
- Improving Security for Kubernetes Deployments at Scale
- Integrating Calico and Istio to Secure Zero-Trust Networks on Kubernetes
About the experts
