Creating a Job Log Drain Using Splunk

This tutorial describes how to create a job log drain using Splunk.

Steps

  1. Configure your cluster to use the splunk-indexer.

    See Configuring Splunk Logging for details on adding this component to your cluster.

    When you configure your cluster with the splunk-indexer component, the splunk-indexer has a process named "syslog-ng" running on it that is listening on port 1514. This process is configured to write out the logs to a location where Splunk will ingest them. You can then search for the job in Splunk using the job name or UUID.

  2. Add a log drain to Splunk for a job.

    The command syntax is as follows:

     apc drain add syslog://<splunk-indexer-private-ip-address>:<port> --app app-name
    

    Where the IP address is the private IP of the splunk-indexer host and the port is 1514.

    For example:

         apc drain add syslog://10.0.0.58:1514 --app redmine
         ╭───────────────────────────────────────────╮
         │            Drain Add Settings             │
         ├─────────────────┬─────────────────────────┤
         │      Drain URL: │ syslog://10.0.0.58:1514 │
         │       App name: │ redmine                 │
         │ Max entry size: │ 2048                    │
         ╰─────────────────┴─────────────────────────╯
    
         Is this correct? [Y/n]: 
         Attaching drain "syslog://10.0.0.58:1514" to app "redmine"... done
         Success!
    

    If you receive the system error "Failed to forward logs: failed to connect after 5 tries…connection timed out" in the job logs for the app configured with a Splunk log drain , make sure that TCP port 1514 on the Splunk Indexer host is open.

  3. Access your Splunk interface and search for the job to verify successful integration with Splunk.