Cloud Tools: Efficient System Administration with Splunk

by Sergey Bushik and Katherine VasilegaMarch 10, 2012

This blog post overviews Splunk Log Aggregation—an effective tool for monitoring and logging.

Table of Contents

Pros, cons, and recommended use

Splunk Log Aggregation is a tool used for logging, monitoring, analytics, and reporting that enables system administrators to index, search, alert, and report on performance data from applications, servers, and network devices. The solution provides unique visibility across your entire IT infrastructure from a single location in real time. Splunk stores crawled data in a searchable repository and can generate graphs, SQL reports, and alerts. It is intended to assist system administrators in identifying patterns and the diagnosis of a problem. Log files can be correlated across systems and software components, which can help administrators uncover the cause of system failures.

Pros

Splunk possesses a customizable search engine, which is intuitive to administer, and the ability to train its parsing engine.
Nowadays, there is an enormous amount of live data from your IT systems that system administrators and employees have to monitor every day. The process is time-consuming and poses a real challenge. To overcome this, Splunk provides system administrators with a single interface to search, report, and alert on all IT data across the entire IT infrastructure. Splunk can be applied to any unstructured data and used to analyze events, such as web analytics, telecom call records, earthquake data, etc. Search results can be assembled into larger and more familiar concepts or groups, such as “IP Addresses” or “Failed Transactions.” Furthermore, Splunk can process data from any source, including files, directories, ports, and scripts on local or remote machines, and keep track of changes.
Another powerful feature of Splunk’s data crawler is the ability to define and extract custom fields from the event data. Splunk’s search application has its own search language, similar to Google’s, which allows users to search by expression and AND-OR logic, as well as exclude terms, etc. In addition to that, the Splunk search language implements the “pipe” concept, allowing users to apply operations, such as sorting, filtering, and grouping, to intermediate search results and pass them to the next operation. The Splunk Report building tool can accept search engine output and create graphs of different types: bar, pie, XY axis plots, etc.

Cons

However, there are certain challenges that users faced during the implementation of real-world cloud deployment projects. They included database clustering, cloud provider coupling, as well as the potential cost escalation.
There are also limitations on types of source data that logging/monitoring tools (including Splunk) can process and the necessity to carefully check your logging software configuration. Although a regular PC should be enough to get started, there should be mentioned other requirements for deploying monitoring tools in the cloud effectively, such as a high-performance dedicated server for each of the tool’s components. In some cases, licenses issued for personal usage can possess certain limitations, making them inadequate for getting the necessary level of visibility, compared with offerings for large organizations.

Recommended use

Splunk is a recommended solution for making log search much simpler and easier to execute, particularly when your IT infrastructure produces a huge amount of similar log data. Having a saved record of common problems can be of much help, as it enables users to identify the nature of a problem in a faster manner, which could be very important in real-time applications.