Configuring internal SSL communication using Self-Signed Certificates

Introduction

To ensure encrypted communication, it is necessary to create your own certification authority, individual certificates and keys. We can use the $SPLUNK_HOME/bin/splunk executable to create it, which walks us through the process itself, or the $SPLUNK_HOME/bin/genRootCA.sh and $SPLUNK_HOME/bin/genSignedServerCert.sh set of scripts, which are called by the command. The process is quite lengthy and you need to watch $SPLUNK_HOME/var/log/splunk/splunkd.log in case of problems.

Legend

Abbreviated names of frequently used files, directories and paths :

./splunk == $SPLUNK_HOME/bin/splunk
splunkd.log == $SPLUNK_HOME/var/log/splunk/splunkd.log
server.conf == $SPLUNK_HOME/etc/system/local/server.conf
inputs.conf == $SPLUNK_HOME/etc/system/local/inputs.conf
outputs.conf == $SPLUNK_HOME/etc/system/local/outputs.conf
web.conf == $SPLUNK_HOME/etc/system/local/web.conf
auth/ == $SPLUNK_HOME/etc/auth

Creating a system variable

We often use the $SPLUNK_HOME variable which contains the path to the SPLUNK home folder. To simplify the work, we can create a system variable that applies to all users. In our case, for the /opt/splunkforwarder home folder, the command to create it looks like this :

echo 'SPLUNK_HOME="/opt/splunkforwarder"' >> /etc/environment && source /etc/environment

Generating CA certificates and keys 1

We recommend creating a separate directory in auth/ to store the newly created files. In our case, we have created the CA_SSL directory:

mkdir $SPLUNK_HOME/etc/auth/CA_SSL

We start by creating a private key for our CA :

$SPLUNK_HOME/bin/splunk cmd openssl genrsa -aes256 -out $SPLUNK_HOME/etc/auth/CA_SSL/privateKeyCA.key 2048

After the command is executed, the console prompts us for the authority password. We will enter a unique password and store it safely, as it will be used when our authority signs each new certificate.

We create a request to sign the certificate we want to sign with our key :

$SPLUNK_HOME/bin/splunk cmd openssl req -new -key $SPLUNK_HOME/etc/auth/CA_SSL/privateKeyCA.key -out $SPLUNK_HOME/etc/auth/CA_SSL/signingRequestCA.csr

After the command is executed, the console prompts us for the authority password. Enter the password we used to create our key. The key is then decrypted and used to sign the request - meaning that we signed the certificate request ourselves.

After password verification, OpenSSL prompts us for the certificate parameters:

Country name - preferably in two-character ISO format (EN,GB,DE,SK, etc.)

Name of the country or region - in word or code

Name of the location - in words the name of the town/village/village

Name of organisation - in word or in phrase name of company/organisation

Name of department - in word or in phrase

Fully Qualified Device Name [FQDN] - Important enter the domain name of the CA, if you want to create a wildcard certificate enter * (for example *.mydomain.com)

Email address - email address for contacting administrators, it will be published in the certificate

Sign the created request using the key we created earlier :

$SPLUNK_HOME/bin/splunk cmd openssl x509 -req -in $SPLUNK_HOME/etc/auth/CA_SSL/signingRequestCA.csr -sha512 -signkey $SPLUNK_HOME/etc/auth/CA_SSL/privateKeyCA.key -CAcreateserial -out $SPLUNK_HOME/etc/auth/CA_SSL/signedCertificateCA.pem -days 1095

You will be prompted to enter the authority key password.
We can modify the entered parameters based on the OpenSSL documentation, which can be found on their website.
We can modify the validity of the certificate using the -days switch which specifies the number of days the certificate is valid. After expiration, it is necessary to recreate a new certificate, or disable certificate validity checking on systems that use it.

Generate certificates and keys for individual devices 1

After creating our own CA, we can proceed to create certificates for individual servers/devices in our architecture. In this step, we will create a certificate for our main instance of SPLUNK, i.e. the indexer with the web interface. We can replicate this step on each device, or we can create all the necessary certificates on this instance and then send them to the individual devices using scp for example.

We can modify the individual steps to create a wildcard certificate that we can use for all endpoint devices in our infrastructure, but if the certificate is broken or expires, all devices will be disconnected.

We recommend creating a separate directory in auth/ to store the newly created files. In our case, we have created the SPLUNK_SSL directory :

mkdir $SPLUNK_HOME/etc/auth/SPLUNK_SSL

We start by creating a private key for our server :

$SPLUNK_HOME/bin/splunk cmd openssl genrsa -aes256 -out $SPLUNK_HOME/etc/auth/SPLUNK_SSL/privateKeyHost.key 2048

We will be prompted to create a password for the server key.

We will create a certificate signing request that we want to be signed by a certificate authority :

$SPLUNK_HOME/bin/splunk cmd openssl req -new -key $SPLUNK_HOME/etc/auth/SPLUNK_SSL/privateKeyHost.key -out $SPLUNK_HOME/etc/auth/SPLUNK_SSL/signingRequestHost.csr

You will be prompted to enter the password for the server key.
We will also be prompted to enter the information we entered when we created our CA. We follow the previous instructions, but make sure that the [FQDN] is unique, or that it belongs to a subdomain of the certificate (such as splunk.mydomain.com), or that it is a wildcard.

Finally, we use the created request, the CA certificate and its key to create the resulting certificate :

$SPLUNK_HOME/bin/splunk cmd openssl x509 -req -in $SPLUNK_HOME/etc/auth/SPLUNK_SSL/signingRequestHost.csr -SHA256 -CA $SPLUNK_HOME/etc/auth/CA_SSL/signedCertificateCA.pem -CAkey $SPLUNK_HOME/etc/auth/CA_SSL/privateKeyCA.key -CAcreateserial -out $SPLUNK_HOME/etc/auth/SPLUNK_SSL/signedCertificateHost.pem -days 1095

We will be prompted to enter the password for the CA key.

Merging the resulting files into a format suitable for SPLUNK 2

SPLUNK requires the files it needs, to be concatenated into one in a specific order. Specifically, the following order :

Signed certificate of the device
Certificate key of the device
Signed certificate of the certification authority

Specifically, the resulting file should have the following format :

We can easily create this file with the cat command. In this case we should replace $SPLUNK_HOME with the path, since our account may not have the variable set and we are not running the command via ./splunk (in our case /opt/splunk) :

cat /opt/splunk/etc/auth/SPLUNK_SSL/signedCertificateHost.pem /opt/splunk/etc/auth/SPLUNK_SSL/privateKeyHost.key /opt/splunk/etc/auth/CA_SSL/signedCertificateCA.pem > /opt/splunk/etc/auth/SPLUNK_SSL/mergedCertificate.pem

Configuring the indexer - log receiver 3 4 5

To set up certificates and SSL, you need to edit the server.conf and inputs.conf files. We can also use the certificates we created for the web interface by editing the web.conf file, thus unifying the whole configuration.

The server.conf file exists in the base installation, but needs to be edited. Do not modify the original content that is not listed, edit or add only the entries listed below to each stanza :

[general]
serverName = splunk.myserver.com # Insert the FQDN of the device from the certificate here

[sslConfig]
sslPassword = PLAINTEXT_KEY_PASS # Here we insert the password from the SSL key in a readable format, SPLUNK will encrypt it after reboot
enableSplunkdSSL = true # TRUE will enable SSL, FALSE will disable
serverCert = $SPLUNK_HOME/etc/auth/SPLUNK_SSL/mergedCertificate.pem # Here we insert the path to the merged certificate
cliVerifyServerName = false # Checks if the server name matches the certificate, does not work for self-signed certificates so must be set to FALSE
requireClientCert = false # Requires a certificate from the client, but causes problems with SPLUNK CLI so it is recommended to leave it to FALSE

Stanza is a specific configuration block that is located under categories. So everything under [general] is from the same category up to the next stanza.

The inputs.conf file is not present in a fresh installation of SPLUNK and needs to be created. The contents should be as follows :

[SSL]
serverCert = $SPLUNK_HOME/etc/auth/SPLUNK_SSL/mergedCertificate.pem # Insert the path to the merged certificate here
sslPassword = PLAINTEXT_KEY_PASS # Here we insert the password from the SSL key in a readable format, SPLUNK will encrypt it after reboot
sslVersions = tls1.2 # List of SSL versions we want to use
useSSLCompression = true # Use compression when transferring data

[tcp-ssl:9997]
disabled = false # Enable listening on a given port

The [tcp-ssl:9997] stanza defines an alternate name for the port. The port can be renamed and even changed at will, but the format must remain the same.

To use these certificates also for the web interface, just edit the web.conf file :

[settings]
enableSplunkWebSSL = true # Enable SSL on the web interface
privKeyPath = $SPLUNK_HOME/etc/auth/SPLUNK_SSL/privateKeyHost.key # Path to the private key
serverCert = $SPLUNK_HOME/etc/auth/SPLUNK_SSL/signedCertificateHost.pem # Path to the server certificate
sslPassword = PLAINTEXT_KEY_PASS # Insert the password from the SSL key here in a readable format, SPLUNK will encrypt it after reboot
sslRootCAPath = $SPLUNK_HOME/etc/auth/CA_SSL/signedCertificateCA.pem # Path to the authority certificate
httpport = 443 # Port on which the web interface is provided | Default value : 8000

Configuring the forwarder - log sender 3 4

For the forwarder it is also necessary to edit the server.conf file :

[general]
serverName = forwarder.myserver.com # Insert the FQDN of the device from the certificate here

[sslConfig]
sslPassword = PLAINTEXT_KEY_PASS # Here we insert the password from the SSL key in a readable format, SPLUNK will encrypt it after reboot
serverCert = $SPLUNK_HOME/etc/auth/FORWARDER_SSL/mergedCertificate.pem # Insert the path to the merged certificate here
cliVerifyServerName = false # Verify the server name against the certificate, doesn't work for self-signed certificates so must be set to FALSE
requireClientCert = false # Requires a certificate from the client, but causes problems with SPLUNK CLI so it is recommended to leave it to FALSE

Then create the outputs.conf file :

[tcpout]
defaultGroup = ssl-logging # Default group name

[tcpout:ssl-logging]
server = splunk.mojserver.en:9997 # Server and port where we want to send logs

[tcpout-server://splunk.okmavkacka.sk:9997]
useSSL = true # Enable SSL
clientCert = $SPLUNK_HOME/etc/auth/FORWARDER_SSL/mergedCertificate.pem # Path to the merged certificate
sslPassword = PLAINTEXT_KEY_PASS # Insert the password from the SSL key here in a readable format, SPLUNK will encrypt it after reboot
useACK = true # If TRUE, the forwarder verifies that the files it sent arrived at the server, if not it sends them again and waits for an ACK packet from the server
disabled = false # Enable output on the port
useClientSSLCompression = true # Enable compression in SSL communication
sslVersions = tls1.2 # List of enabled SSL versions

tutorials.kormanik.sk