Tips for Implementing Shift-Left Security on Kubernetes

by Carlo GutierrezFebruary 26, 2019

Learn how to achieve better security in Kubernetes across the networks, build pipelines, hosts, container runtimes, admission control, etc.

Table of Contents

Security on both sides

In the past, when the monolithic architecture was commonplace, software development teams tended to stick to the waterfall model in their work. While this approach was simple—as it enabled teams to develop software gradually per level—it also meant security and testing were postponed until the very end of development. Often, this would lead to unexpected bugs and problems, which could delay deployment.

Now, with the microservices architecture becoming the new standard and with more and more teams adopting agile methods, security and testing are being moved earlier in the software development life cycle. This approach, called shift-left testing, resolves the issues that were frequent when security and testing were delayed.

At a recent Kubernetes meetup in Santa Clara, Gunjan Patel of Palo Alto Networks provided recommendations to enhance security in Kubernetes on both left and right.

“Security should not be an afterthought, especially in the microservices world. It has to go hand-in-hand with development.” —Gunjan Patel, Palo Alto Networks

Gunjan Patel at the meetup in Santa Clara

Security in Kubernetes

Kubernetes provides security across different stages of development. Security measures enabled by default include logging, OS hardening, and least privilege. Still, other security measures are quite easy to implement:

role-based access control (RBAC)
image scanning
network policies

There are also security efforts that require a bit more work before they can be enabled:

secure computing mode (seccomp)
security-enhanced Linux (SELinux)
pod security policies
kernel exploits
container escapes
application policies

Breakdown of Kubernetes security concepts (Image credit)

Although it takes some work to get all of these security measures up and running, having them all provides Kubernetes with security on the left and the right.

“Ultimately, it’s not just about shifting left. Shifting left is really important, but is should be shifting left and right.” —Gunjan Patel, Palo Alto Networks

Security best practices

Gunjan provided some security tips that would cover both left and right. While his pieces of advice work for Kubernetes, they can also be considered platform-agnostic. He listed the following guidelines for improving security across the board.

Securing the build pipeline:

verify the source of your images
use official base images
lock down access to the image registry
scan container image layers for common vulnerabilities and exposures
automatically tag vulnerable images from running in certain clusters

Improving network security:

secure the services exposed to the Internet using a firewall
use network policies to lock down the L3/4 networks
create granular L7 policies using a service mesh, such as Istio
employ mutual TLS (mTLS) to authenticate containerized workloads

Shift left is about early detection and prevention of bugs and vulnerabilities (Image credit)

Securing the host:

run secure operating systems, such as container-optimized OS
exploit seccomp to restrict system calls that containers are allowed to make
utilize SELinux to further isolate containers

In securing container runtimes, Gunjan recommended making sure the security configuration spans across container runtimes, especially if the environment has multiple container runtimes in the cluster.

He also advised implementing admission controls by using policies to restrict which containers can run, such as non-privileged containers, containers that don’t need write access to the file system, and containers that don’t need certain system calls.

Since there are multiple cloud providers that support Kubernetes, Gunjan emphasized the need to identify and understand the version of orchestrator and container runtime components your cloud provider is running by default. If some components have been modified from their open-source versions, you should also track them.

Container infrastructure security (Image credit)

Finally, he provided additional guidelines for securing the orchestrator:

manage orchestrator API access using RBAC and network policies
be aware of running third-party plugins and what access they have
maintain access to the orchestrator control plane APIs from third-party plugins using RBAC and service accounts

The orchestrator configuration can also be secured by making sure it is always validated by a security system. Changes to the configuration should also go through a change and review process.

“The thing that is pushing us more and more towards left and right is the cloud. You’re not managing everything in the stack, and you shouldn’t.” —Gunjan Patel, Palo Alto Networks

As previously mentioned, Kubernetes provides certain security measures by default. With Gunjan’s suggestions, Kubernetes users can further improve the security of their clusters from both the left and the right.

Want details? Watch the video!

Table of contents

What is shift left? (5’08”)
How has containers changed software development? (9’30”)
Security measures in Kubernetes (15’20”)
Security best practices for Kubernetes (17’44”)
How can Istio help? (27’16”)
Demo of using Istio (28’45”)
Questions and answers (33’38”)

These are the slides from the session.

About the expert

Gunjan Patel is Cloud Architect in the Developer Relations team at Palo Alto Networks. Prior to that, he served as a developer at Tigera, working on container networking and security projects—such as Project Calico, flannel, and CNI. Before that, Gunjan worked at Ciena and Cisco. He is part of the core team at GoBridge—an organization that helps underrepresented groups in the Go community. Gunjan is also the organizer of the Go Remote Meetup group.