Configuring Data and Package Storage

This document describes data and package storage configuration details for your Apcera deployment.

Metrics Manager storage

The Metrics Manager tracks the utilization of cluster resources (disk, CPU, RAM, network) over time using Graphite, a real-time graphing system. It can have heavy I/O requirements, depending on the number of metrics being sent to the Graphite server, which is affected by the number of running jobs and the number of IMs.

If the load increases beyond what the disk can handle it can lead to high IO wait times, missing metrics, and possible stability issues on the server. For this reason, you should choose a high-performance storage interface for the Metric Manager, such as NVMe/SSD or SAN.

Audit log DB (HA)

To store the audit logs, Apcera requires an auditlog-database machine configured in HA mode.

If the auditlog-database is configured and deployed on the cluster, the Audit log tab appears on the web console and its access is governed by poliyc. If the auditlog-database is not configured, the Audit log tab is not be available on the web console.

The following examples demonstrate how to configure the auditlog-database in the cluster.conf file.

Specific provider

The following example cluster.conf snippet shows how to configure the audit log DB for a specific provider, in this case vSphere:

machines: {
  # auditlog database
  audit: {
    cpu:    2
    memory: 4096
    disks: [
        { size: 50, purpose: auditlog }
    ]
    suitable_tags: [ "auditlog-database" ]
  }
}

components: {
  # auditlog databases
   auditlog-database: 2
}

Generic provider

The following example cluster.conf snippet shows how to configure the Audit log for a generic IP address provider, in this case vCloud:

machines: {
  audit: {
    hosts: [ "10.224.214.48","10.224.214.49"]
    suitable_tags: [ "auditlog-database" ]
  }
}

components: {
  # auditlog databases
   auditlog-database: 2
}

Riak package store

For non-AWS deployments, Apcera uses Riak which is an S3-compatible object store for packages. Note that configuration depends on the router component to load balance requests. Cluster nodes hosting Riak must be able to resolve packages.s3.$base_domain to one or more Apcera routers via DNS during the cluster deployment.

Specific provider

The following example cluster.conf snippet shows how to configure the Riak object store machine host for a specific provider such as the vSphere or OpenStack provider. (This example is for vSphere.)

machines: {
  object_storage: {
    cpu: 2
    memory: 16384
    disk: [
      { size: 100, purpose: riak}
    ]
    suitable_tags: [
      "riak-node"
    ]
  }
}

components: {
           riak-node: 3
}

Generic provider

The following cluster.conf snippet shows how to configure Riak for a generic (IP address) provider. (This example is for vCloud).

machines: {
  object_storage: {
    hosts: [ "10.224.214.135", "10.224.214.132", "10.224.214.133" ]
    suitable_tags: [
      "riak-node"
    ]
  }
}

components: {
           riak-node: 3
}

Garbage Collection Settings

By default, when packages are removed from the Apcera Platform using Riak as the storage backend, packages are not instantly cleaned up. There are a few thresholds that need to be met before Riak's internal Garbage Collection process kicks in.

These values are configurable starting with release 508.

Default values if not explicitly configured:

chef: {
      "riak": {
        "config": {
            "bitcask": {
                "dead_bytes_merge_trigger": 67108864,
                "dead_bytes_threshold": 16777216,
                }
        }
      },

      "riak_cs": {
          "config": {
              "riak_cs": {
                  "leeway_seconds": 60,
                  "gc_interval": 300,
                  "gc_retry_interval": 4500,
              }
          }
      }


  "continuum": {
....
}

The example below shows 'aggressive' settings for redeeming free space, and may not be appropriate in all situations, as the garbage collection processes can be very taxing on a Riak/RiakCS cluster.

chef: {
      "riak": {
        "config": {
            "bitcask": {
                "dead_bytes_merge_trigger": 0,
                "dead_bytes_threshold": 0,
                "log_needs_merge": true,
                }
        }
      },

      "riak_cs": {
          "config": {
              "riak_cs": {
                  "leeway_seconds": 300,
                  "gc_interval": 900,
                  "gc_retry_interval": 4500,
              }
          }
      }


  "continuum": {
....
}
  • leeway_seconds: this is the time between when a package is deleted, and the database actually marks it as deleted. This is to ensure that if someone or something else was downloading that particular package, they will have 5 minutes to retrieve the file.
  • gc_interval: this runs the garbage collection process every 120 seconds to clean up items that have been marked as deleted.
  • gc_retry_interval: this is run ever 900 seconds and attempts to remove any packages that failed (for whatever reason) during the initial garbage collection process.
  • dead_bytes_merge_trigger and dead_bytes_merge_threshold are threshold triggers in the storage backend. A value of 0 means that whenever a file is marked as deleted by garbage collection, it will be physically removed from the disk. When these thresholds are met, a 'merge' will be run which will re-converge the data in the Riak ring and free up space.

Package Manager LRU

Package Manager LRU is a feature that you can use to prune package files (tarballs) from the local PM cache based on least recently used (LRU). To understand how this feature works, it is necessary to review the PM component and its various modes of operation.

The behavior described herein only applies to tarballs associated with Apcera packages. It is not applicable to Docker image layers. See Docker cache for details on how files associated with Docker images are handled.

The Package Manager (PM) component is the repository for cluster packages (tarball files + metadata). When a job needs a package, if the host Instance Manager (IM) does not have a local cached copy, the IM downloads the package from the PM.

See Apcera Packages and PM Tagging for details on packages and how the PM interacts with Instance Managers (IMs).

You can run PMs in local mode or in s3 mode. The PM LRU acts in conjunction with the PM's s3 mode.

In local mode, the PM configuration setting cleanup_on_delete controls the removal of packages when it is deleted. If this setting is set to false, the local PM cache is not pruned when users delete package records. The JSON metadata record is removed, but the package tarball persists on the PM disk. If this setting is set to true, both the metadata and tarball is removed when a user deletes a package.

In production for high availability you run the PM in s3 mode with multiple PMs backed by remote storage (AWS S3 or Riak/S3) that provides the master datastore for package files. Apcera supports eventual consistency for package replication across clustered PMs. In s3 mode, cleanup_on_delete is always true.

Enabling the lru setting deletes package tarballs from the local PM cache when a user deletes a package. The LRU acts only on the local package cache, polling it based on the prune_interval_seconds (default 30 seconds) and pruning the least recently used tarball(s) exceeding the reserved_size_bytes.

To enable LRU for remote package datastore, you add the following block to the chef.continuum.package_manager section of the cluster.conf.

"chef"
  "continuum"
    "package_manager": {
      "lru": {
        "enabled":             true,
        "reserved_size_bytes": 4294967296
      }
    }

LRU parameters:

  • enabled – boolean enables LRU
  • reserved_size_bytes – int prune threshold
  • prune_interval_seconds – int default is 30 seconds, or set custom interval

In the example, LRU package pruning is enabled for the cluster. Every 30 seconds, the system checks the cache size of each PM. If it is over ~4.3 GB, the system will prune packages based on least recently used. Note that since the interval is not set, the default is used.