Encrypting Cluster Traffic (BETA)

You can enrypt the runtime traffic in your cluster using Internet Protocol Security (IPsec). This section describes how to enable IPsec encryption for your cluster, as well as considerations for using cluster encryption.

IPsec is a BETA feature. Please contact Apcera Support if you plan to enable IPsec for a production cluster.

Using cluster encryption

Apcera supports the use of the Internet Protocol Security (IPsec) to encrypt cluster traffic. IPsec provides IP packet-level encryption between two IP endpoints (the source and destination). Apcera uses the OpenSWAN IPsec implementation.

When IPsec is enabled for your cluster, all runtime communications are encrypted. This includes traffic between cluster components, such as the Job Manager, Package Manager, and Instance Manager, both TCP and HTTP traffic. It is a mesh network, so there will be encrypted tunnels between all nodes in the cluster, Central to IM, Central to Singleton, Singleton to IM, etc.

IPsec encryption only applies at cluster runtime, not during cluster creation or deployment. This means that communication traffic between the Orchestrator Server and any cluster host or component is not encrypted.

IPsec encryption is enabled during deployment of the Apcera release. Once enabled, encryption exists only between cluster hosts/components but not with Orchestrator. Any traffic between the Orchestrator host and the other Apcera cluster components are unencrypted.

In addition, the following types of traffic are not encrypted:

Performance considerations

There is an inherent tension between security and performance. If you enable IPsec encryption for your cluster, plan for a noticeable performance impact dependent on the specific platform, hardware acceleration, etc.

Creating a job is not that resource-intensive and you may not notice a performance hit here. However, if there is a lot of traffic between jobs (job links, virtual networks), you are likely to notice the performance hit.

If you are deploying a hybrid cluster across two or more providers (such as on AWS with remote IMs on OpenStack) and using a VPN connection to bridge the hosts, traffic flowing across the VPN will be double-encrypted and you will incur a double performance hit.

Configuring cluster encryption

To use IPsec encryption for your cluster, you must enable it and specify a random string to generate your own preshared key:

chef: {
  continuum: {
    "ipsec_mesh_enabled": <`true` or `false`>,
    "ipsec_mesh_preshared_key": <RANDOM_USER_DEFINED_STRING>

For example:

chef: {
  continuum: {
    "ipsec_mesh_enabled": true,
    "ipsec_mesh_preshared_key": abcdefghigklmnopqrstuvwxyz1234,

Once you have enabled IPsec encryption and set the key in cluster.conf, you run the standard orhcestrator-cli deploy commands to deploy the cluster. If the deployment is a cluster update or upgrade, you must reboot each cluster host and restart all jobs.

Generating the preshared key

The preshared key is a user-defined string of any length. Note, however, that the value for the ipsec_mesh_preshared_key parameter is not used for cluster encryption. The preshared key is only used to authenticate IPsec using ISAKMP. Once authenticated the IPsec protocol does dynamic key session allocation.

Apcera strongly recommends that the preshared key value contain a mixture of alphanumeric characters and be at least 30 characters in length. You must define your own key. Do NOT copy/paste the preshared key value we provide in the example. You should use a random string generator to create the preshared key, and encrypt and version control your cluster.conf file. See Considerations about IPsec Pre-Shared Keys for best practices concerning generating and maintaining preshared keys.

Reboot and job restart required

If you are updating an existing cluster to enable IPsec encryption, or removing (disabling) IPsec encryption, you must reboot each cluster host following the cluster host reboot procedure. During a normal reboot process, each Instance Manager (IM) host will experience a downtime of approximately 1 to 2 minutes during which job instances will not be available.

Note that after performing an upgrade/downgrade of IPsec, the system will automatically restart all job instances (containers). Job restart is required and performed automatically so that IPsec gets enabled for each job. Note that it may take up to a minute for the job restart to complete.

You do not need to reboot cluster hosts if you are deploying a new cluster for the first time that enables IPsec encryption.

Firewall rule requirements

All components except for the Orchestrator host must allow the following connections across all private networks components are assigned to:

  • Allow UDP from private_networks ports 1023-65535 to private_networks port 500
  • Allow UDP from private_networks ports 1023-65535 to private_networks port 4500
  • Allow ESP(50) from private_networks to private_networks
  • Allow AH(51) from private_networks to private_networks

ESP and AH are connectionless protocols like UDP. Tunnels must be allowed in cross-mesh.

These rules must be modified to fit your firewall environment. For example, in AWS with security groups, each security group would need an ingress rule and an egress rule for each of the above rules. In a standard cluster having "private_networks" ("", for example), unless otherwise covered by a more general rule, some example egress rules could be "allow UDP ports ALL destination" or "allow protocol ALL destination" (which is more general than all four rules above).