Configuring Cluster Monitoring

To monitor the jobs in your cluster (your code), you use the Web Console and the APC client, and perhaps third-party tools you use to instrument your applications, such as AppDynamics or New Relic.

To monitor the system (our code), Apcera supports both internal and external monitoring using third-party tools. Internal monitoring monitors the components of the Apcera system. External monitoring monitors the hosts running key components of the system.

To configure internal monitoring (Zabbix), you populate the web_hostnames field of the chef.apzabbix section of the cluster.conf.erb (Terraform) or cluster.conf file. The following examples demonstrate both types of configurations. If you are using Terraform, you configure monitoring in the cluster.conf.erb file, otherwise configure monitoring using the cluster.conf file.

NOTE: To configure monitoring alerts using Zabbix, you must set the attribute chef.continuum.cluster_name in the cluster.conf to get a persistent value for the cluster-name.

Monitoring configuration example

The following configuration example shows the base cluster.conf settings for cluster monitoring using Zabbix.

chef: {
  "apzabbix": {
    "disable_default_site": true,
    "cachesize": "500MB",
    "web_hostnames": ["zabbix.mycluster.apcera-platform.io"],
    "sslonly": true,

    "db": {
      "hostport": "localhost:5432",
      "master_user": "postgres",
      "master_pass": "DB_ADMIN_PASSWORD",
      "zdb_user": "zabbix"
      "zdb_pass": "DB_USER_PASSWORD"
    }
    "users": {
      "guest": { "user": "guest", "pass": "GUEST_USER_PASSWORD" },
      "admin": { "user": "admin", "pass": "ADMIN_USER_PASSWORD", "delete": false }
    },
  } # apzabbix
} 

If you are using Terraform some of the Monitoring parameters will be auto-populated from other sources. Refer to the instructions for your platform.

Monitoring configuration parameters

The following table describes each of the monitoring configuration parameters that you can set.

When using AWS RDS to provide your Postgres database for monitoring, be sure to create the database with Postgres version 9.6. AWS has deprecated earlier versions of Postgres DB and version 10 uses a incompatible syntax.

Configuring Monitoring email alerts

You can configure the cluster to send notifications of Zabbix-triggered monitoring alters to an email address, a pager address, or both. The email block is used to configure email alerts from Zabbix. The pagerduty entry is used to send alert notifications to a pager.

For example, using cluster.conf:

chef: {
  "apzabbix": {
    "disable_default_site": true,
    "cachesize": "500MB",
    "web_hostnames": ["clustermon.mycluster.example.com"],
    "sslonly": true,
    "db": {
      "hostport": "MonitoringPostgresEndpoint.us-west-2.rds.amazonaws.com:5432",
      "master_user": "acme_ops",
      "master_pass": "DB_ADMIN_PASSWORD",
      "zdb_user": "zabbix",
      "zdb_pass": "DB_USER_PASSWORD"
    },
    "users": {
      "guest": { "user": "monitoring", "pass": "PASSWORD" },
      "admin": { "user": "admin", "pass": "PASSWORD", "delete": false }
    },
    "pagerduty": {
      "key": "API-ACCESS-TOKEN-GOES-HERE"
    },
    "email": {
      "sendto": "target@example.com",
      "smtp_server": "localhost",
      "smtp_helo": "localhost",
      "smtp_email": "from@example.com"
    }
  } # apzabbix
}

Or using cluster.conf.erb:

  "apzabbix": {
    ...
    "pagerduty": {
      "key": "API-ACCESS-TOKEN-GOES-HERE"
    }
    "email": {
      "sendto": "target@example.com",
      "smtp_server": "localhost",
      "smtp_helo": "localhost",
      "smtp_email": "from@example.com"
    }
  } # apzabbix
}

NOTE: The subject of the email alerts is {TRIGGER.STATUS}: <cluster-name> - {HOST.NAME1} - {TRIGGER.NAME}. The format of the email message is as follows. See also the macros documented in the Zabbix manual.

name:{TRIGGER.NAME}
id:{TRIGGER.ID}
status:{TRIGGER.STATUS}
hostname:{HOSTNAME}
ip:{IPADDRESS}
value:{TRIGGER.VALUE}
event_id:{EVENT.ID}
severity:{TRIGGER.SEVERITY}

Postfix

You can configure the chef.postfix section of the cluster.conf to define a site specific mail relay. This may be necessary if you are using an environment where the Zabbix server is unable to send email directly and and you need a relay host for email notification.

Here is the configuration section you need to include:

chef: {
  # Define a site specific mail relay, if necessary
  "postfix": {
    "main": {
      "relayhost": "10.0.0.50"
    }
  }

For example, with the following configuration, the file /etc/postfix/main.cf contains relayhost = foo.example.com.

chef: {
  "postfix": {
    "main": {
      "relayhost": "foo.example.com"
    }
  }

Monitoring Google Auth

You can configure your cluster.conf file to instruct Zabbix to monitor Google Auth.