Improving Security for Kubernetes Deployments at Scale
Security lags behind speed and scalability
Since its initial launch in June of 2014, Kubernetes, a container orchestration system, has gradually risen in popularity over the years. One of the reasons for this growth in adoption is the increased rate of application development and ease of infrastructure scaling it provides.
Another factor that has contributed to the rise of Kubernetes is the ability to deploy Kubernetes in the cloud. While it can still be deployed on premises, cloud-based installations enable organizations to forego the costs associated with having and maintaining their own infrastructure.
The problem with cloud-based Kubernetes deployments, however, is that these require a different approach when it comes to security.
At a recent Kubernetes meetup in Silicon Valley, Dan Hubbard, Chief Product Officer at Lacework, revealed some of the vulnerabilities with Kubernetes workloads in the cloud and also shared some recommendations to improve security.
“Kubernetes, like a lot of these new technologies, have this speed and scale, as well as secure and safe balancing act. Speed and scale win.” —Dan Hubbard, Lacework
Dan Hubbard at the meetup in Silicon Valley (Photo credit: Altoros)
How bad is it?
According to the research conducted by Lacework in June 2018, over 21,000 container orchestration and API management systems were discoverable on the Internet. While being visible on the Internet isn’t always a vulnerability, it is certainly an unnecessary security risk as this becomes a “window of opportunity” that malicious attackers can take advantage of.
Kubernetes accounted for 76% of the discovered systems (Image credit)Lacework conducted their research using Shodan—an engine that encapsulates SSL data mining, web crawling, and other functionality to identify potentially vulnerable systems.
“With simple tools and a relatively simple query, I can find all kinds of vulnerable things.”
—Dan Hubbard, Lacework
While Kubernetes does include built-in security features, such as SSL and default authentication, the Lacework team encountered glaring issues in some of the Kubernetes systems they found, including:
- open dashboards in the middle of configuration
- open dashboards that required no authentication
- open dashboards that could be brute-forced
- information disclosure on nodes with invalid certificates
In cases where the dashboard was completely exposed, malicious attackers can gain full remote administration of the corresponding systems. When this happens, attackers have full access to perform such operations as adding applications, modifying infrastructure, changing credentials, extracting data, etc.
Some examples of the exposed dashboards (Image credit)The Lacework team also found 38 instances of healthz, a Kubernetes container health check service, which required no authentication whatsoever.
“We found roughly 21,000 systems open on the Internet. That doesn’t mean 21,000 are insecure. Many of them could have full hardware tokens built-in or other forms of security. Of those, we found a little over 300 with no username or password.” —Dan Hubbard, Lacework
During Lacework’s search, Dan made it clear that they did not attempt to access any of the systems that were open. Instead, they reached out to the nodes that were potentially exposing critical data.
How can security be improved?
After exploring the vulnerabilities uncovered, Lacework came up with some general recommendations on improving security:
- Regardless of the network policy in place, use multi-factor authentication for all access
- Apply strict controls to network access, especially for UI and API ports
- Use SSL for all servers and use valid certificates with proper expiration and enforcement policies
- Deploy VPNs or reverse proxy to enable connection to sensitive servers
Dan also shared a few security tips specific for Kubernetes:
- Build a pod security policy by preventing pods from running as root, as well as from accessing host ports and certain volume types
- Configure your Kubernetes pods to run read-only file systems
- Restrict privilege escalation in Kubernetes with role-based access control
Example pod security policy (Image credit)For organizations without a security team, Dan suggested the use of existing products and services, such as Lacework’s Polygraph—which constitutes a set of behavioral maps—in order to manage container security. Polygraph is a security platform for cloud deployments, which is capable of capturing relevant activities, organizing data logically, establishing a baseline of container behavior, identifying deviations in container behavior, and delivering actionable insights.
“We really believe that security needs to change with modern deployment methods. It’s not about bringing an appliance into the cloud, virtualizing something, and pushing it out, or taking a normal piece of software that’s designed for on premises and putting it in the cloud. It’s about building and designing purpose-fit technologies for your public cloud.”
—Dan Hubbard, Lacework
Lacework Polygraph dashboard (Image credit)With so many companies looking to speed up and scale their development life cycle, it’s not a surprise to see the increased adoption rate of container orchestration systems like Kubernetes. That said, security shouldn’t be an afterthought and should always be part of the process.
Want details? Watch the video!
Related slides
Further reading
- Configuring UAA to Provide a Single Entry Point for Kubernetes and Cloud Foundry
- A Multitude of Kubernetes Deployment Tools: Kubespray, kops, and kubeadm
- Kubernetes Cluster Ops: Options for Configuration Management
About the expert
