Splunk is a platform for aggregating, indexing, searching & analyzing logs and other machine generated data. At my day job, we use it extensively to aggregate application logs hosted on different servers. Using Splunk queries, we could query those log data to debug any production issues as well as to build different charts and dashboards for both engineering and business reporting.
Even though I was fairly comfortable with writing Splunk queries to build dashboards utilizing the log data, I’ve never actually configured servers to forward logs to Splunk. I wanted to try and understand what goes into configuring the Splunk universal forwarder. This article is simply a documentation of the steps I took to configure Splunk universal forwarder to forward application logs from a EC2 server and a dockerized app deployed on ECS Fargate.
Splunk forwarders send data from data sources to Splunk cloud for indexing which makes it easier for searching, querying and building dashboards. To push logs to Splunk cloud, we’d need to use one of Splunk forwarders. Splunk provides different type of forwarders,
heavy forwarder and
While Splunk provides something called
HTTP Event Collector allowing us to push data from servers to Splunk cloud over HTTP/HTTPs, I opted to use the
universal forwarder. Since HTTP Event Collector relies on HTTP connections, it might impact the application performance.
Unlike HTTP Event Collector,
Universal Forwarder are lightweight agents which are installed as a package on the host machine which periodically monitors the log file in the background and pushes them to the Splunk Cloud.
You could use following steps to configure the Splunk Universal Forwarder on a EC2 machine or any other VPS servers.
SSH into the EC2 server
splunkforwarder package for EC2 OS from Splunk Downloads web page e.g. for Debian based ubuntu
wget -O splunkforwarder-9.0.1-82c987350fde-linux-2.6-amd64.deb "https://download.splunk.com/products/universalforwarder/releases/9.0.1/linux/splunkforwarder-9.0.1-82c987350fde-linux-2.6-amd64.deb"
Install the package using
sudo dpkg -i splunk-forwarder.deb This typically install the
Start Splunk by accepting license
sudo /opt/splunkforwarder/bin/splunk start --acept-license. You’ll be asked for a local
password , Keep note of it, as we’ll need it later.
sudo /opt/splunkforwarder/bin/splunk restart
Download the Splunk forwarder credentials file from the Splunk universal forwarder documentation link https://yourdomain.splunkcloud.com/en-US/app/splunkclouduf/setupuf
Copy Splunk credentials file (downloaded in step6) from your local machine to the EC2 server using Secure Copy Protocol
scp , Use following command on your local machine.
scp /path/to/file/splunkclouduf.spl ubuntu@IP:/server/pathIf you have
pem file, use
scp -i <pem file> file_to_copy_local_path ec2-user@ip:location_of_server
Install Splunk credentials using
sudo /opt/splunkforwarder/bin/splunk install app ./splunkclouduf.spl . Use
password setup in step 4
Add your Splunk Cloud forward server
sudo /opt/splunkforwarder/bin/splunk add forward-server yourdomain.splunkcloud.com:9997
Monitor log files on certain index
sudo /opt/splunkforwarder/bin/splunk add monitor ./my-app/log/production.log -index my-app-prod-indexHere,
my-app-prod-index is an index which is manually created in Splunk. The default index is
For dockerized applications, Splunk provides a docker image for
universal forwarder. We could potentially run this as a sidecar container alongside the app container or probably use
docker-compose and add it as a dependency to our application service.
However, for simplicity, I opted not to use the docker image provided by Splunk. I manually download the Splunk package and run Splunk commands when building the Docker image and when running the container.
We can download & install the Splunk universal forwarder package, expose the required ports, accept Splunk license and setup the Splunk admin user when creating a docker image.
We’ll have to configure forward server, add monitors and start Splunk forwarder when we run the docker container. We can use docker
entrypoint to run Splunk commands when starting the container.
# Dockerfile # Splunk Universal Forwarder RUN wget https://download.splunk.com/products/universalforwarder/releases/126.96.36.199/linux/splunkforwarder-188.8.131.52-9e907cedecb1-linux-2.6-amd64.deb \ && apt-get install -f ./splunkforwarder-184.108.40.206-9e907cedecb1-linux-2.6-amd64.deb \ && rm -f splunkforwarder-220.127.116.11-9e907cedecb1-linux-2.6-amd64.deb # PORTS for Splunk Universal Forwarder EXPOSE 9997 EXPOSE 8000 # Accept Splunk license and setup admin user RUN deploy/splunk/auth.sh ENTRYPOINT ["./deploy/entrypoint.sh"]
In the above
Dockerfile, we’ve downloaded and installed the Splunk universal forwarder package. Additionally, we are running commands in
/splunk/auth.sh. This fill will run commands to accept Splunk’s license and setup admin user for running Splunk commands on the container. Since Splunk will ask to manually type the username and password, we are using the
expect package to automate the process. These credentials can be anything, but we need to keep note of it as we require it in next stages.
#!/usr/bin/expect set timeout -1 spawn /opt/splunkforwarder/bin/splunk start --accept-license expect "Please enter an administrator username: " send -- "user\r" expect "Please enter a new password: " send -- "password\r" expect "Please confirm new password: " send -- "password\r" expect eof
entrypoint.sh will run commands to start Splunk, add a forward server and monitor logs. Note that
splunkclouduf.spl is a file downloaded from https://yourdomain.splunkcloud.com/en-US/app/splunkclouduf/setupuf page and the user name and password should be the same used in the previous step in
#!/bin/bash /opt/splunkforwarder/bin/splunk start /opt/splunkforwarder/bin/splunk install app /path/to/app/splunkclouduf.spl -auth user:password /opt/splunkforwarder/bin/splunk add forward-server yourdomain.splunkcloud.com:9997 /opt/splunkforwarder/bin/splunk add monitor /path/to/app/log/ -index app-prod-index /opt/splunkforwarder/bin/splunk restart
Basically we are following the same commands that we used to configure the Splunk universal forwarder in EC2, however we are separating and defining them in
[entrypoint.sh](http://entrypoint.sh) instead on manually SSH-ing into the server and configuring them.
sudo /opt/splunkforwarder/bin/splunk status
sudo /opt/splunkforwarder/bin/splunk list forward-server
sudo /opt/splunkforwarder/bin/splunk list monitor
sudo /opt/splunkforwarder/bin/splunk restart