Logs are an important part of security and server administration. They document all kinds of events that take place on a server. Having to dig through multiple servers for logs can be a tedious process. A Security Information and Event Management (SIEM) application can make this a lot easier for us.

Splunk is an analytics-driven SIEM that meets a wide range of security use cases including compliance, application security, incident management, advanced threat detection and real-time monitoring. We can use Splunk to parse through logs from all machines in our home lab. To begin, we need to configure an Rsyslog server to accept the logs and also Rsyslog clients to send logs to our Rsyslog server. I’m going to be using my Dell T40 server running Ubuntu 20.04 LTS.

Configuring the Rsyslog Server

apt-get install rsyslog -y

Once installed, we’ll need to setup the Rsyslog server with the configuration file:

vim /etc/rsyslog.conf

To use both TCP and UDP connections, you’ll need to uncomment the following lines:

$ModLoad imudp
$UDPServerRun 514
$ModLoad imtcp
$InputTCPServerRun 514

Now we can define the specific subnet that we’ll accept logs from. My home network uses 192.168.1.0/24 so I’ll be using:

$AllowedSender TCP, 127.0.0.1, 192.168.0.0/24
$AllowedSender UDP, 127.0.0.1, 192.168.0.0/24

Now we can tell the Rsyslog server how to store incoming logs. Add the following before the GLOBAL DIRECTIVES section:

$template remote-incoming-logs, "/var/log/%HOSTNAME%/%PROGRAMNAME%.log" 
*.* ?remote-incoming-logs

Now we can save the configuration file and check for syntax errors with the following command:

rsyslogd -f /etc/rsyslog.conf -N1

Lastly, restart Rsyslog to have the service use those settings:

systemctl restart rsyslog

Configuring the Rsyslog Client

Our Rsyslog server is now configured. Now we’ll want to setup our Rsyslog clients on all of the devices in our home lab. I’ve already set this up on most of my devices, but I just added a Raspberry Pi running Pi Hole to my home lab so I’ll be setting up the Rsyslog client on this Raspberry Pi.

vim /etc/rsyslog.conf

Now we’ll want to add the following to the end of the file:

##Enable sending of logs over UDP add the following line:

*.* @192.168.1.X:514


##Enable sending of logs over TCP add the following line:

*.* @@192.168.1.X:514

##Set disk queue when rsyslog server will be down:

$ActionQueueFileName queue
$ActionQueueMaxDiskSpace 1g
$ActionQueueSaveOnShutdown on
$ActionQueueType LinkedList
$ActionResumeRetryCount -1

You’ll want to ensure that the 192.168.1.X IP address is the IP address of your Rsyslog server. Once that’s setup, save the file and restart Rsyslog on the client:

sudo service rsyslog restart

The Rsyslog client is now configured. We can now navigate to the /var/logs/ directory on the Rsyslog server to find the clients logs within a directory with the clients hostname.

Adding the logs to Splunk

Now that the client logs are being sent to the Rsyslog server, we can create a free account at splunk.com. Once a free account is created and Splunk is downloaded onto the server, we can navigate to the Splunk login page to login using a browser:

http://192.168.1.X:8000/

To add our client logs to Splunk, we’ll first click on Settings at the top of the home page. From the drop-down menue, we’ll need to select Data inputs. Now we’ll click on +Add New to the right of Files & Directories. From here, we’ll click on Browse, then navigate to /var/log and select the hostname of our client. Now we’ll click on Next and ensure that the Host Field Value is the hostname of our client. We can now click on Review > Submit.

Our client logs are now added to Splunk on the Rsyslog server. We can search for logs from our client in Splunk with:

source="/var/log/HOST/*" host="HOST"

Splunk can help us identify threats by showing us patterns within our logs. For example, searching for errors can point out IP addresses that are repeatedly attempting to connect to our server. We can also use specific searches to find the countries that the attackers are connecting from:

host="HOST"  IP="*" | iplocation IP | stats count by Country

This will output a list of countries that are attempting to access your server. You can then block the IP address of the users or block the country using IPTables. The point is that Splunk is an extremely useful tool that can help you confidently monitor your logs.