Using Logsearch for Hybrid Cloud Foundry and Kubernetes Deployments
The need for a unified logging tool
Logging is the first step in troubleshooting any kind of software deployment. In Cloud Foundry, the default means of retrieving logs is using the cf logs command. This way, however, there is no user interface, which makes logging quite inconvenient.
In addition, the retention period of Cloud Foundry logs is set to a mere 24 hours. While this can be extended with manual configuration, it is not the most convenient method for managing logs.
When a log retention period expires, developers can no longer access any log data unless they configured the system to archive such data in some external storage. Sometimes, retrieving data from an external data warehouse involves a lot of redundant manipulations.
Retrieving logs using a grep command (Image credit)With hybrid environments gaining popularity and developers exploring the combination of Cloud Foundry and Kubernetes deployments, troubleshooting becomes even more challenging as developers now have to keep track and search through logs in each environment. At KubeCon 2018, Viktoryia Kotava and Andrei Krasnitski of Altoros demonstrated how to simplify logging in a hybrid environment.
What’s Logsearch?
Logsearch is an open-source project for building and operating log analysis clusters in the cloud. Using the Elasticsearch, Logstash, and Kibana (ELK) stack, Logsearch provides the following features:
- Unified log management. All logs produced by the deployed environment are available in a single place.
- Highly available centralized storage. Elasticsearch uses the concept of shards, where each index is broken down into them. Each shard can have one or more replicas. All components support horizontal and vertical scaling.
- Parsing rules. Precise parsing rules for Logstash transform raw log data into understandable and readable format from any structured/unstructured data source. Filtration is based on already parsed logs.
- Security and multitenancy. Logsearch is secured with the Cloud Foundry identity management service, User Account and Authentication (UAA), which is also available for Kubernetes deployments. Syslog traffic can be encrypted and secured with TLS/SSL certificates.
“Logsearch has a native integration with Cloud Foundry UAA. You may also use Cloud Foundry UAA for a Kubernetes installation. If you enable UAA, those permissions you configure for role-based or attribute-based access control systems will work for Logsearch, too.” —Viktoryia Kotava
Viktoryia Kotava and Andrei Krasnitski at KubeCon 2018 (Courtesy of Michael Maximilien)The encryption of the syslog traffic ensures that messages are encrypted in the transport layer. Any communication requires the syslog sender to authenticate the syslog receiver, and vise versa. This mutual authentication prevents man-in-the-middle attacks.
What logs can you monitor?
In Cloud Foundry and Kubernetes environments, Logsearch can help to collect the following logs:
- Platform logs from all the Cloud Foundry components, Kubernetes pods and nodes, as well as the underlying infrastructure.
- Third-party services logs from Cloud Foundry, Pivotal CF, and related data services.
- Application logs collected using firehose-to-syslog in Cloud Foundry or the Fluentd data collector in Kubernetes.
Viktoryia Kotava and Andrei Krasnitski at KubeCon 2018 (Courtesy of Michael Maximilien)In addition, Logsearch provides out-of-the-box dashboards for applications and platform logs. Automated backups can also be configured to work with any Amazon S3 compatible storage system.
How it works
Andrei provided some insights into how log management works in Cloud Foundry and Kubernetes/Kubo—exemplified on sample architectural schemes. The architectural solution for aggregating platform logs in Cloud Foundry includes such components as NATS, UAA, Cloud Controller, etc. For Cloud Foundry, one uses the Logsearch BOSH release, which encapsulates RSYSLOG responsible for efficient forwarding of log messages.
Logstash serves as a syslog server (receiving log messages via the syslog protocol), a persistent queue, and a log messages parser. To make log management even more convenient, Logsearch provides a set of custom parsing rules for Logstash. Elasticsearch plays the role of a distributed and highly available storage, while Kibana delivers a web interface.
Managing system logs on Cloud Foundry (Image credit)The architectural solution for managing application logs on Cloud Foundry will include almost the same components, except for firehose-to-syslog replacing RSYSLOG to ensure log pulling.
Managing application logs on Cloud Foundry (Image credit)The team at Altoros decided to use its knowledge about log management in Cloud Foundry, as well as a codebase developed in three years, and re-apply it to Kubernetes. As a result, we’ve come up with the following implementation:
- To aggregate container logs, Fluentd agents are installed on top of Kubernetes pods using BOSH runtime config.
- Fluentd ships log data straight to Elasticsearch, excluding Logstach.
- The ELK stack is deployed as a BOSH release.
In addition to container logs, this architectural approach allows for aggregating infrastructure-related logs. The approach is valid for both Kubernetes and Kubo—a tool for launching Kubernetes environment managed by BOSH.
A sample architecture for managing container logs on Kubernetes (Image credit)To install Logsearch for Kubernetes, you may consider two of the following options:
- If you a novice to Logsearch or need a separate deployment for Kubo or Pivotal Container Service, it’s recommended to install Logsearch from scratch.
- If you are already using Logsearch with an open-source Cloud Foundry or Pivotal CF, you may bind your Kubernetes environment with the existing Logsearch installation using this extension.
Two options for installing Logsearch on top of Kubernetes (Image credit)As hybrid deployments garner more interest, unified tools, such as Logsearch, will become increasingly useful. At the moment, Logsearch is available as a BOSH release for open-source Cloud Foundry and as a tile for Pivotal CF. For Kubernetes and Kubo deployments, you may use the extension designed by Altoros.
Related slides
Further reading
- Configuring UAA to Provide a Single Entry Point for Kubernetes and Cloud Foundry
- Using Centralized Logs and Metrics to Monitor Cloud Foundry Across All Levels
- Troubleshooting .NET Apps on Cloud Foundry: Logs, Metrics, and Remote Debugging
About the experts
