Configuring Splunk Logging

If you are a Splunk customer, you can integrate your Apcera cluster with Splunk to collect and search component logs. You can also use Splunk as a log drain for collecting job logs. For additional Splunk integration options, contact Apcera Support.

Configuring Splunk logging

With the following Splunk configuration, component logs from all server nodes in the cluster will be automatically forwarded to the splunk-indexer host. A syslog server is also installed on the splunk-indexer host that can be used as a log drain target for jobs in the cluster.

  1. Include the splunk-indexer.tf module with your Terraform files that allocates 500GB disk with purpose splunk-indexer.

  2. Add a host to your cluster.conf file with suitable_tags: [ "splunk-indexer" ].

     machines: {
       splunk: {
         hosts: [ '10.0.2.7' ]
         suitable_tags: [ "splunk-indexer" ]
       }
     }
    
  3. Add the mount:

     chef: {
       "continuum": {
         "mounts": {
           "splunk-indexer": {
             "device": "/dev/xvdp"
           }
         },          # mounts
       }
     }
    
  4. Include the machine count:

     components: {
               splunk-indexer: 1
               ...
     }
    
  5. Specify the following block in cluster.conf to configure a Splunk indexer server inside your cluster.

     chef: {
       "continuum": {
         "splunk": {
           "users": {
             "admin": { "password": "EXAMPLE_PASSWORD" }
           }
           # The IP address of a splunk license server to connect to
           "master": "1.2.3.4",
           "ssl": {
             # Provide a SSL certificate for use by Splunk
             "enable": true,
             "certs":
               {
                 "server_names": [ "splunk-search.domain.tld"],
                 "certificate_chain": (-----BEGIN CERTIFICATE-----
                   CERTIFICATE
                   -----END CERTIFICATE-----
                   -----BEGIN CERTIFICATE-----
                   CERTIFICATE
                   -----END CERTIFICATE-----
                 )
                 "private_key": (-----BEGIN RSA PRIVATE KEY-----
                   CERTIFICATE
                   -----END RSA PRIVATE KEY-----
                   )
             }       # splunk -> ssl -> certs
           }         # splunk -> ssl
         }           # splunk
       },            # continuum
     }
    
  6. To search Splunk for the cluster or job logs related to a specific job you can use the job UUID.

    To get the job UUID, run the following APC command:

     apc job list -l
    

    Or you can search using the FQN of a job by creating a search macro to execute the following search of the logs for a job named /sandbox/user::jobname (assuming a logdrain to splunk was added to the job):

     index=apcera-job-log | eval job=source | rex field=job mode=sed "s/^.*\/([^\/]*)/'\1'/" | join job [search sourcetype="continuum-cluster-monitor" stats message "job::/sandbox/user::jobname"| rex field=_raw "setting to (?<stats_message>.*)" | spath input=stats_message output=instances Instances{} | mvexpand instances | spath input=instances | dedup JobUUID, JobFQN | search JobFQN="job::$fqn$" | eval job="'".JobUUID."'" | fields job] 
    

    For example:

    screenshot

    And here is a search of all component logs for logs related to a specific job:

     [search sourcetype="continuum-cluster-monitor" stats message "job::/sandbox/user::jobname"| rex field=_raw "setting to (?<stats_message>.*)" | spath input=stats_message output=instances Instances{} | mvexpand instances | spath input=instances | dedup JobUUID, JobFQN | search JobFQN="job::/sandbox/user::jobname" | eval job="'".JobUUID."'" | fields job]