Misconfigurations Make Up 59% of Kubernetes Security Incidents

The state of Kubernetes security

Containerization has significantly changed the software life cycle as containers provide portability, agility, scalability, and the ability to use a microservices architecture. However, containers, being distributed, make it harder to identify security risks, such as vulnerabilities and misconfigurations. Organizations will typically deploy an increasing amount of containers over time, so keeping track and managing all of them is another challenge.

To help organizations manage containers at a large scale, container orchestration solutions like Kubernetes have become a necessity. However, using a platform such as Kubernetes adds a new layer that needs to be secured. Without prior experience or expertise, organizations may be leaving their Kubernetes deployments vulnerable.

Areas that need to be secured (Image credit)

Recent reports emphasize security as one of the main concerns for Kubernetes deployments. We look into those reports from a security perspective and cover various Kubernetes best practices to help organizations protect their deployments.

What are the security concerns?

According to a report by Red Hat about the state of Kubernetes security, 94% of respondents experienced a security incident in the last 12 months. The report compiled survey results from over 500 DevOps, engineering, and security professionals, providing insight on where security can be improved when using containers and Kubernetes.

Leading causes of security incidents (Image credit)

While many organizations use Kubernetes to facilitate the deployment and management of containers, configuring the container orchestration system, especially at scale, can get complicated. This is evidenced by nearly half of respondents being concerned over exposures due to misconfigurations in their Kubernetes environments. Common configuration mistakes include:

• accidentally exposing internal services
• running containers with too many privileges
• errors when defining a pod’s security context or when implementing role-based access control

Nearly half are concerned over misconfigurations (Image credit)

With most organizations being worried about misconfigurations, it is no surprise that they are also concerned about the runtime life cycle phase of containers. Typically, security issues at runtime are caused by misconfigurations during the build or deploy phases. Additionally, flaws in security are likely to be exposed and felt once an application is up and running in production.

Most vulnerabilities are felt at runtime (Image credit)

With nearly half of respondents running containers in hybrid environments, organizations need security that runs consistently regardless of where workloads are deployed.

Container deployment strategies (Image credit)

The report also highlighted problems regarding the container adoption strategies implemented by companies. Over a quarter of respondents are worried that their organization’s container adoption strategy does not invest enough in security.

Container strategy concerns (Image credit)

On a positive note, plenty of organizations are moving on the right track and are advancing their container and Kubernetes security strategies. More than half of respondents have either a basic or intermediate plan in place, and 11% already have an advanced plan for security. This means that more and more companies are coming up with mature security strategies, which also help push Kubernetes adoption. (In a recent blog post, we covered how Kubernetes adoption grew by 67% in 2021.)

Breakdown of container security strategies (Image credit)

Security requires a collective effort

As more organizations adopt the practice of shifting left, where security is factored in at the earliest stages in the development life cycle, 15% of the report respondents consider developers to be responsible for security. 27% of respondents say that the burden of security should lie with DevOps.

Besides developers and DevOps, there is a near even distribution among other roles. This indicates that there is no one-size-fits-all strategy when it comes to Kubernetes security, and organizations should have tools that can facilitate collaboration between different teams.

Roles responsible for Kubernetes security (Image credit)

Building on the idea of shifting left, more companies are building development, security, and operations (DevSecOps) teams. DevSecOps involves automating the integration of security at every step of the software development life cycle.

In the report, only 26% are still keeping DevOps and security roles separate, and nearly half of respondents indicate having some kind of DevSecOps initiative. This is a good indicator that most organizations are prioritizing security early in the development life cycle.

A quarter of organizations are integrating and automating security (Image credit)

Kubernetes security best practices

As previously mentioned, Kubernetes helps organizations to manage and deploy multiple containers. However, configuring and securing Kubernetes deployments at scale can get complicated.

To help companies secure their deployments, there are several security guidelines provided in the official documentation for Kubernetes. (In one of our previous blog posts, we shared a checklist for keeping deployments secure.) Additionally, the United States’ National Security Agency (NSA) and Cybersecurity and Infrastructure Security Agency (CISA) released a report detailing how to protect Kubernetes clusters. All of these security best practices are applicable regardless of a Kubernetes distribution.

Scan container and pods for vulnerabilities and misconfigurations. Performing image scanning is an ideal method to prevent running software from becoming vulnerable. One way to implement image scanning is by using an admission controller, a Kubernetes-native feature that can intercept and process requests to the Kubernetes API prior to persistence of the object, but after the request is authenticated and authorized. Before an image is deployed to a cluster, a custom webhook can be implemented. The admission controller can then block the image from being deployed if it does not meet security policies that are defined in the webhook configuration.

Container build workflow with image scanning (Image credit)

Follow the principle of least privilege. Run containers and pods with the least privilege possible. This will reduce the impact of any potential attacks. Implementing the principle of least privilege can be done in multiple ways. One method is to build container images to run as a non-root user. Another option is to configure RBAC policies to specify allowed actions for users, groups, and service accounts in a namespace.

Isolate networks. Use network separation tools, such as host and network firewalls and Kubernetes network policies to control the amount of damage a breach can cause. Isolate namespaces by setting a default network policy that denies all ingress and egress, and only permits connections that are explicitly required.

Enable logging. Use log auditing so administrators can monitor activity and receive alerts. A service mesh, such as Istio, can be implemented to help collect logs from each service, helping developers diagnose problems of microservices.

Service mesh integrates logging with network security (Image credit)

Bridge the gap between DevOps and security. Most organizations with either have their DevOps or security teams running security platforms. To reduce friction, introduce security tooling that can facilitate quick interaction between both teams.

Run benchmark security tests. Even when following security best practices, it is possible to overlook a few factors. To be certain, use benchmarks, such as those published by the Center for Internet Security (CIS) to test cluster security.

Periodical configuration review. Regularly check all Kubernetes settings and use vulnerability scans to help identify risks and to ensure that security patches are applied.

Typically, Kubernetes deployments are functional out-of-the-box, but are not necessarily secure. Following the recommendations above can help organizations, especially those running multiple clusters, create a standardized deployment process that is both secure and consistent.

Want to learn more about the current state of Kubernetes adoption? Read our recent article that examines how containerization has grown by 2022, covering recommendations for facilitating the cloud-native journey.

Further reading

• Machine Learning Constitutes 65% of Kubernetes Workloads
• Ensuring Security Across Kubernetes Deployments
• Tips for Implementing Shift-Left Security on Kubernetes