Home > Compute > Unlock Splunk Universal Forwarder service

Unlock Splunk Universal Forwarder service

After restarting a server, I occasionally notice that the splunkforwarder service doesn’t gracefully shut down prior to the reboot.  This makes a geek a little bitter, and when the security peeps get the “no check-in in the past 24hrs’ alert, they aren’t all that enthused either.

Fortunately, a GPO, some PowerShell, and the task scheduler help me out of this bind.  Read on to find out how!

When the Splunk Universal Forwarder starts, it creates a file called conf-mutator.pid in the .\var\run\splunk folder of the install directory. It is a text file and contains the PID of the splunkd process. If the service is stopped gracefully, this file is deleted at service shutdown. If, however, the service stops less-than-gracefully (read: the splunkd process crashes), the conf-mutator.pid file remains. When the service tries to start again, whether after a server restart or trying to start it manually, if conf-mutator.pid exists, the service start fails.

Using group policy – only because I haven’t implemented DSC in production… yet! – I created a file and a scheduled task preference. The file preference creates the file c:\automation\Unlock-SplunkUniversalForwarder.ps1.  The schedule task policy sets up a task to run, once per hour as the system account, the Unlock-SplunkUniversalForwarder.ps1 script.

To make sure the service is can start after a failure, the script checks for an ungraceful service stop, attempts to restart the service, and sends one of two email notifications (depending on the outcome of the service start attempt).

Understanding that it may, on occasion, be necessary to intentionally stop the service, the script only attempts to start the splunkforwarder service if it finds conf-mutator.pid. If the service was gracefully stopped, the script leaves it alone.

Here’s the script:

###########################################################################
#
# NAME: Unlock Splunk Universal Forwarder
#
# AUTHOR:  Jimmy Hester (@geekjimmy)
#
# COMMENT: 
#
# FILENAME: Unlock-SplunkUniversalForwarder.ps1
#
# VERSION HISTORY:
# 1.0 5/7/2014 - Initial release
# 1.1 5/8/2014 - Parameterized & anonymized for the interwebs
#
###########################################################################

param (
    [string]$lockfile = "$env:programfiles\SplunkUniversalForwarder\var\run\splunk\conf-mutator.pid",
    [string]$SendFrom = "sender@foo.com",
    [string]$SendTo = "recipient@foo.com",
    [string]$SMTPServer = "smtp.foo.com" 
)

function Send-ErrorEmail () {
    Send-MailMessage -From $SendFrom -To $SendTo -Subject "$env:computername Splunk Problem" `
    -Body "Splunk Forwarder on $env:computername is stopped and failed to restart. Please investigate" -SmtpServer $SMTPServer
}
   
function Send-NotificationEmail () {
    Send-MailMessage -From $SendFrom -To $SendTo -Subject "Splunk Restarted: $env:computername" `
    -Body "Splunk Forwarder on $env:computername was stopped but was successfully restarted." -SmtpServer $SMTPServer
}

function Send-NotificationMessage () {
    if ( $splunkerror.exception -ne $null ) {
        Send-ErrorEmail
    }
    else {
        Send-NotificationEmail
    }
}

if ((Get-Service splunkforwarder).Status -ne 'Running') {
    if (Test-Path -Path $lockfile) {
        Remove-Item $lockfile -Force
        Start-Service splunkforwarder -ErrorVariable splunkerror
        Send-NotificationMessage
    }
}

Advertisements
Categories: Compute Tags: , ,
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: