Introduction to Splunk

Splunk is a software platform that collects and stores all this machine data in one place. It makes it as easy to search through and investigate that data as using Google. Basically, it's Google for log files! Beyond troubleshooting, you can use this search capability to build reports and dashboards to monitor performance, reliability, or other metrics across a whole collection of related servers and devices, and even create alerts to warn you by text or email when something is going wrong.

It's also used to detect security threats, and since you have all this data in one place, you can do event correlation across devices and apply machine learning to it for the purposes of anomaly detection, user behavior analytics, and even predictive analytics to identify potential problems before they happen.

Splunk components

One of the first and most important things you need to learn about Splunk in order to work with it effectively is what the functional components are and how they work together. We will take a look at 3 main components of Splunk.

•         Indexer

•         Search head

•         Universal forwarder

Universal forwarders, indexers, and search heads constitute the majority of Splunk functionality

The universal forwarder (UF) is a free small-footprint version of Splunk Enterprise that is installed on each application, web, or other type of server (which may be running various flavors of Linux or Windows operating systems) to collect data from specified log files and forward this data to Splunk for indexing (storage). In a large Splunk deployment, you may have hundreds or thousands of forwarders that consume and forward data for indexing.

An indexer is the Splunk component that creates and manages indexes, which is where machine data is stored. Indexers perform two main functions: parsing and storing data, which has been received from forwarders or other data sources into indexes, and searching and returning the indexed data in response to search requests.

A search head is an instance of Splunk Enterprise that handles search management functions. This includes providing a web-based user interface called Splunk Web, from which users issue search requests in what is called Search Processing Language (SPL). Search requests initiated by a user (or a report or dashboard) are sent to one or more indexers to locate and return the requested data; the search head then formats the returned data for presentation to the user.

Lab . Installing and Configuring Splunk(on Linux)

You can get Splunk Enterprise for Linux on the Splunk website, starting at this URL: https://www.splunk.com/en_us/download/splunk-enterprise.html.

Create a free account with Splunk from this page, or log in if you already have one. On the Choose Your Download page, click the tab for the operating system (Linux, in this case), and select one of the packaging options—.deb, .tgz, or .rpm.

Clicking the Download button next to .rpm will start a download process, but it also reveals a link you can click to download the rpm using the command line (wget) – we'll use this option. Clicking the link opens a message box where you can copy the wget command :

wget -O splunk-7.1.1-8f0ead9ec3db-linux-2.6-x86_64.rpm 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64andplatform=linuxandversion=7.1.1andproduct=splunkandfilename=splunk-7.1.1-8f0ead9ec3db-linux-2.6-x86_64.rpmandwget=true'

After verifying that rpm downloaded successfully, install it:

rpm -i splunk-7.1.1-8f0ead9ec3db-linux-2.6-x86_64.rpm

Before we start Splunk for the first time, we'll need to verify and modify a few Linux settings.

ulimits

Splunk has some specific requirements regarding the maximum file size, number of open files, user processes, and data segment sizes on the Linux platform—known as ulimits—that you will want to verify. From a Terminal command, type these commands (as root) and verify that the numbers meet or exceed the values given:

ulimit -f unlimited

ulimit -u 16000 

ulimit -n 64000

ulimit -d 1073741824 

Transparent huge pages

Some Linux distributions typically have a transparent huge pages feature that is enabled by default; this causes some fairly significant performance losses with Splunk Enterprise , so Splunk suggests this feature be disabled.

GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 console=tty0 net.ifnames=0 crashkernel=auto transparent_hugepage=never"

We are now ready to start Splunk for the first time! You will want to run Splunk Enterprise as a Splunk user by using the: su - splunk command.

Then, navigate to the /opt/splunk/bin directory with the: cd /opt/splunk/bin command and type ./splunk start --accept-license.

when it is finished, it will tell you where the Splunk Web interface is. Enter the given string into a new browser window and you will be presented with the Splunk login screen:

By default, Splunk does not start when the server is rebooted; you will typically want it to do so.

/opt/splunk/bin/splunk enable boot-start -user splunk

Now we need to collect logs from remote servers. For collecting logs you can read the next blog post "Getting data into Splunk".