Patrick Bareiss

In this blog post, I will explain how to monitor a Linux Server with Splunk. We will cover different logging/monitoring options for Linux Server using Splunk Enterprise. This tutorial assumes that you have already installed Splunk as described in this blog post. We will monitor the logs of the Linux Server running Splunk.

In the first step, we will download Splunk Add-On for Unix and Linux from splunkbase:

https://splunkbase.splunk.com/app/833/

We login to our Splunk instance and click on Manage Apps:

Then, we click on Install app from file:

We choose the downloaded .tgz file by clicking on Durchsuchen/Choose and then click on Upload:

In the next steps, we will configure the Splunk Add-On for Unix and Linux. We connect over terminal to our Splunk server and navigate to the Splunk Add-On for Unix and Linux folder as splunk user:

cd /opt/splunk/etc/apps/Splunk_TA_nix

Normally, you should never edit the files in an apps default folder. Instead, you should create a local folder and make your changes there:

mkdir local

We will copy the inputs.conf configuration in the new folder:

cp default/inputs.conf local/

The first part of the inputs.conf configuration file uses different bash scripts to collect information about the Linux server. You can enable it by changing the disabled value to 0 and adding the index value:

# Copyright (C) 2018 Splunk Inc. All Rights Reserved.
[script://./bin/vmstat.sh]
index = unix
interval = 60
sourcetype = vmstat
source = vmstat
disabled = 0

[script://./bin/iostat.sh]
index = unix
interval = 60
sourcetype = iostat
source = iostat
disabled = 0

[script://./bin/ps.sh]
index = unix
interval = 30
sourcetype = ps
source = ps
disabled = 0

[script://./bin/top.sh]
index = unix
interval = 60
sourcetype = top
source = top
disabled = 0

...

In my opininon, the files /var/log/messages, /var/log/secure and /var/log/audit/audit.log are worth to collect. As we installed Splunk as splunk user, which is a non-root user, we have to perform some changes in order to be able to read these log files.

In order to read the /var/log/auditd/audit.log, we will change in /etc/audit/auditd.conf the log_group to splunk:

#
# This file controls the configuration of the audit daemon
#

local_events = yes
write_logs = yes
log_file = /var/log/audit/audit.log
log_group = splunk
log_format = RAW
flush = INCREMENTAL_ASYNC
...

After that, we restart the auditd daemon with the following command:

system auditd restart

Unfortunately, the rlog.sh script, which is responsible for reading the /var/log/audit/audit.log file, is not working for me. Therefore, I changed the rlog.sh under /opt/splunk/etc/apps/Splunk_TA_nix/bin/ to the following:

#!/bin/sh
# Copyright (C) 2018 Splunk Inc. All Rights Reserved.
#
#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.
#
# credit for improvement to http://splunk-base.splunk.com/answers/41391/rlogsh-using-too-much-cpu
. `dirname $0`/common.sh

SEEK_FILE=$SPLUNK_HOME/var/run/splunk/unix_audit_seekfile
AUDIT_FILE=/var/log/audit/audit.log

#if [ "x$KERNEL" = "xLinux" ] ; then
#    assertInvokerIsSuperuser
#    assertHaveCommand service
#    assertHaveCommandGivenPath /sbin/ausearch
    if [ -n "`service auditd status`" -a "$?" -eq 0 ] ; then
            if [ -e $SEEK_FILE ] ; then
                SEEK=`head -1 $SEEK_FILE`
            else
                SEEK=0
                echo "0" > $SEEK_FILE
            fi
            FILE_LINES=`wc -l $AUDIT_FILE  | cut -d " " -f 1`
            if [ $FILE_LINES -lt $SEEK ] ; then
                # audit file has wrapped
                SEEK=0
            fi
            awk -v START=$SEEK -v OUTPUT=$SEEK_FILE 'NR>START { print } END { print NR > OUTPUT }' $AUDIT_FILE | tee $TEE_DEST | /sbin/ausearch -i 2>/dev/null | grep -v "^----"
    fi
#elif [ "x$KERNEL" = "xSunOS" ] ; then
#    :
#elif [ "x$KERNEL" = "xDarwin" ] ; then
#    :
#elif [ "x$KERNEL" = "xHP-UX" ] ; then
#	:
#elif [ "x$KERNEL" = "xFreeBSD" ] ; then
#	:
#fi

In order to be able to read /var/log/messages and /var/log/secure, we will run the following command as root user:

setfacl -m g:splunk:r /var/log/messages
setfacl -m g:splunk:r /var/log/secure

Unfortunatley, this will not persist a logrotate. Therefore, we will create the following ACL configuration under /etc/logrotate.d/Splunk_ACLs as root user:

{
    postrotate
        /usr/bin/setfacl -m g:splunk:r /var/log/messages
        /usr/bin/setfacl -m g:splunk:r /var/log/secure
    endscript
}

Now, we are ready to configure the inputs.conf configuration under /opt/splunk/etc/apps/Splunk_TA_nix/local and add the following lines:

[monitor:///var/log]
index = unix
whitelist=(messages|secure)
disabled = 0

We restart Splunk to enable the new configuration:

/opt/splunk/bin/splunk restart

We can see the logs in Splunk and do some Splunk magic with it

Thank you for reading.