Getting Data into Splunk

In the first blog post(Introduction to Splunk), I walked you through the process of getting Splunk installed on your Linux systems. Now I will talk about how you can collect logs from remote servers. You can get data into your Splunk platform instance in a number of ways. The best way depends on the location and volume of data, your infrastructure and security needs. In this blog post, you will learn how to use a universal forwarder in order to collect logs from remote servers. The Universal Forwarder is the best mechanism for collecting logs from servers and end-user systems.

Installing Splunk universal forwarder

Universal Forwarders provide reliable, secure data collection from remote sources and forward that data into Splunk software for indexing and consolidation. They can scale to tens of thousands of remote systems, collecting terabytes of data.

You can download the Splunk Enterprise universal forwarder from this link: https://www.splunk.com/en_us/download/universal-forwarder.html.

For Linux, when installing Splunk Enterprise, you can download and save the rpm , but you can also copy the wget command to install the package from your server by clicking on Download via Command Line :

wget -O splunkforwarder-7.1.2-a0c72a66db66-linux-2.6-x86_64.rpm 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.1.2&product=universalforwarder&filename=splunkforwarder-7.1.2-a0c72a66db66-linux-2.6-x86_64.rpm&wget=true'

Then run the below command.

rpm -i splunkforwarder-7.1.2-a0c72a66db66-linux-2.6-x86_64.rpm

Starting the universal forwarder

sudo su - splunk

cd $SPLUNK_HOME/bin

./splunk start --accept-license

On Linux, however, you will need to configure Splunk to auto-start upon reboot, by executing the following command

./splunk enable boot-start -user splunk

Configuring outputs.conf

We now need to tell the forwarder which index to send their data to, you can do this from the command line:

./splunk add forward-server [indexer  IP]:9997 -auth [username]:[pass]

Creating Index 

The index is the repository for Splunk Enterprise data. Splunk Enterprise transforms incoming data into events, which it stores in indexes. As Splunk Enterprise processes incoming data, it adds the data to indexes. Splunk Enterprise ships with several indexes, and you can create additional indexes as needed.

In Splunk Enterprise Web, navigate to Settings > Indexes and click New.

To create a new index, enter:

A name for the index. User-defined index names must consist of only numbers, lowercase letters, underscores, and hyphens. They cannot begin with an underscore or hyphen. 

Configuring inputs.conf

Now, we will configure inputs.conf file to monitor log files and send the entries to Splunk for indexing. In this example case, we have installed a forwarder on an Apache web server running on a Linux host; the web server log files reside in the /var/log/httpd directory

To monitor these logs, we need to edit the inputs.conf file that was created in the /opt/splunkforwarder/etc/system/local directory when the Splunk forwarder was installed, with the following contents:

[default]

host = test_server

[monitor:///var/log/httpd/access_log]

index = web_logs_access

sourcetype = access_combined

ignoreOlderThan = 30d

 That is all you need to do. Now go to the Splunk Enterprise Web and Search for "index=* source="/var/log/httpd/access_log" " and you can observe the incoming logs.