Installing Apcera EE on Ubuntu

This section provides instructions for installing the Apcera Platform Enterprise Edition 2.4 or later on Ubuntu 14.04 ("BareOS").

This documentation assumes some familiarity with deploying an Apcera cluster. If you have not deployed a cluster before, please familiarize yourself with the process by reviewing the installation documentation.

Minimum cluster requirements for BareOS

These instructions demonstrate how to create a minimum viable deployement (MVD) for a BareOS cluster:

  • 4 servers: Orchestrator, Singleton, Central and one Instance Manager (IM).
  • Base OS: Ubuntu 14.04 (minimal or server).
  • Each IM host must have a primary partition with 20GB space.
  • Adheres to the [minimum viable deployment] sizing guidelines.
Count Type RAM Disk Installed components
1 orchestrator 2GB 8GB orchestrator-server, orchestrator-database
1 central 4GB 20GB router, api-server, flex-auth-server, nats-server, job-manager, health-manager, cluster-monitor, metrics-manager, auditlog-database, component-database, events-server, auth-server
1 singleton 4GB 20GB package-manager (local), redis-server, graphite-server, statsd-server, stagehand
1 instance-manager 8GB 100GB instance-manager

Install Ubuntu 14.04

Install Ubuntu Server 14.04 on each machine host.

Partition each IM disk

Review the IM disk partitioning requirements.

Each Instance Manager (IM) machine host must have its disk partitioned so that the IM has a dedicated volume on which to run container (job) instances. The majority of the disk should be allocated to the IM. When you populate the cluster.conf file, you must provide a physical disk partition so that each IM can use LVM to create necessary logical partitions.

For each IM, you will need to create a primary partition sufficient for the OS (say 20 GB) ("/dev/sda1"), and another partition that is not formatted for the job instances.

For example, on a single disk machine using /dev/sda, it can look like this:

Disk /dev/sda: 64.4 GB, 64424509440 bytes
255 heads, 63 sectors/track, 7832 cylinders, total 125829120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006d8c7

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048    39063551    19530752   83  Linux
/dev/sda2        39063552   125829119    43382784   83  Linux

Using the above example with a single disk:

$ fdisk /dev/sda

Then "n" to create a new partition, "p" for primary, "2" for number, accept defaults for start and end, then "w" to write changes to disk.

Understand DNS requirements

There are three phases to a BareOS deployment: machine provisioning, cluster deployment, and cluster operations.

To provision machines you run the provisioning scripts. To run these scripts, all servers in the cluster must have internet access.

To deploy the cluster, you populate the cluster.conf file and run it using orchestrator-cli deploy commands. External DNS is not required to deploy the cluster. Note that during the installation, Apcera will change the hostnames of the initial servers.

The orchestrator-cli (using chef) sets the hostname to clustername-uuid, where the uuid is the one seen in the orchestrator ssh menu. This makes it possible to correlate logs back to their hosts after extracting them to another system for analysis. The same chef recipe adds the local hostname to /etc/hosts on each host to allow local name resolution. There is no interaction with DNS for the phase of the installation, and none should be required. Nothing outside the host is aware of the hostname. All cluster services are configured by IP.

To use the cluster after deployment, you must create public DNS records that point to the IP address of the cluster HTTP Router (NGNIX). You will need two DNS records to be created: {DOMAIN} and *.{DOMAIN}. The cluster FQDN and wildcard entries point to the HTTP router. If you are running multiple HTTP routers, the entries point to the load balancer you have in front of them.

See [Configuring DNS] below for details.

Download the provisioning scripts

Review the requirements for and description of the BareOS provisioning scripts.

The provisioning scripts are available for download to licensed Apcera customers at the Customer Support Portal.

Once Ubuntu 14.04 is installed on each server, and each IM is partitioned, there are two scripts to run as root (or sudo):

  • provision_base.sh: Apply to all servers except orchestrator type; this script must be ran on all servers except the Orchestrator.
  • provision_orchestrator.sh: Apply to orchestrator type only; this script is run on the Orchestrator host only.

These scripts update the kernel, create required users, add repositories, set up OS level security, run Chef recipes, and install necessary agents for each cluster server.

Each script requires internet access as well as access to a valid DNS to resolve external hostnames like "apcera.com."

The scripts change the OpenSSH SSH daemon configuration file named sshd_config to only allow the following users to SSH into the hosts:

  • On orchestrator node: ops, orchestrator, and root users.
  • On all other nodes: ops, and root users.

If necessary you can cusomtize the scripts to add users and your own SSH keys.

Add SSH keys for Orchestrator access

You use Orchestrator to deploy and manage a cluster. Typically you connect to the Orchestrator host remotely via SSH to deploy a cluster. To do so you must load a private SSH key into the local ssh-agent and include the matching public SSH key in the cluster.conf.

(1) On a Unix machine you can generate an SSH key pair for such purposes as follows:

ssh-keygen    // Enter password and file path

(2) For orchestrator-cli commands to work on remote servers in the cluster, you must load a valid private SSH key must in the shell/session context (ssh-agent).

This is done by the following set of actions (where simple.pem is the name of a private SSH key you have generated):

Start the ssh-agent (if necessary):

$ eval $(ssh-agent)
Agent pid 9473

Check that there is no key:

$ ssh-add -l
The agent has no identities.

Add the key:

$ ssh-add simple.pem
Identity added: simple.pem (simple.pem)

(3) The last requirement is to add the public SSH key to the cluster.conf file. Once a "orchestrator deploy" occurs, Chef will be managing the key list. The following example shows you how to add a pulic SSH key to cluster.conf.

chef: {
  "continuum": {

    [...]

    "ssh": {
      "custom_keys":[
       "ssh-rsa AAAAB3Nz...W6S6tDCH orchestrator@bare04",
       # The below is the mykey.pub to allow mykey.pem to be loaded in ssh-agent and `orchestrator-cli` to work
       "ssh-rsa AAAAB...22O/CN1Lxz"
      ]
    },

(4) Once you deploy the cluster, you can connect to the Orchestrator host via SSH as follows:

ssh -A orchestrator@123.123.123.123     // Connect to the Orchestrator host

Cutomize the provisioning scripts (optional)

Optionally you may want to customize the BareOS provisioning scripts to:

The information in this section is optional. You do not need to make any customizations for your initial BareOS installation.

Defining additional users

During the provision_x.sh phase, the scripts will modify how a user can accesses ssh. The scripts change the sshd_config to only allow some users to SSH in, and change how the identity check is performed.

If you need to add additional cluster admin users, you can either modify the provision_xx.sh script or the sshd_config file.

For example:

  ...
  AllowUsers ops orchestrator root myuser
  ...

If you made changes to sshd_config, restart the ssh service for changes to take effect:

$ service restart ssh

Here is another sshd_config modification example:

AuthorizedKeysFile /etc/ssh/userauth/%u

Then, to allow remote access for the above user, place the public SSH key named in the matching location:

cat lmcnise_key.pub > /etc/ssh/userauth/lmcnise
chmod 600 /etc/ssh/userauth/lmcnise

Adding SSH keys

Only the cluster.conf entry cited above is required to ensure remote SSH access. Once a "orchestrator deploy" occurs Chef will be managing the key list. For troubleshooting purposes, you can add your own keys in the provision_xx.sh scripts to ensure that you retain access to the host prior to the deploy.

As shown below the provisioning scripts contain two SSH public keys (apcera-user-ca, apcera-special-ops) that get deployed on all machines. These keys are for Apcera staff only.

provision_base.sh

  1. root user (line 165):
# Write the SSH keys for the root user. This includes the master Apcera root key,
# as well as the user certificate authority key.
(
cat <<'EOP'
cert-authority ssh-rsa AAAAB...N apcera-user-ca
ssh-rsa AAAAB3N...4n/D apcera-special-ops
EOP
)> /etc/ssh/userauth/root
  1. ops user (line 207):
# Write the SSH keys for the ops user. This includes the master Apcera ops key,
# as well as the user certificate authority key.
(
cat <<'EOP'
cert-authority ssh-rsa AAA...5N apcera-user-ca
ssh-rsa AAA...4n/D apcera-special-ops
EOP
) > /etc/ssh/userauth/ops

provision_orchestrator

  1. root user (line 165):
# Write the SSH keys for the root user. This includes the master Apcera root key,
# as well as the user certificate authority key.
(
cat <<'EOP'
cert-authority ssh-rsa AAA...5N apcera-user-ca
ssh-rsa AAA...4n/D apcera-special-ops
EOP
) > /etc/ssh/userauth/root
  1. ops user (line 207):
# Write the SSH keys for the ops user. This includes the master Apcera ops key,
# as well as the user certificate authority key.
(
cat <<'EOP'
cert-authority ssh-rsa AAA...5N apcera-user-ca
ssh-rsa AAA...4n/D apcera-special-ops
EOP
) > /etc/ssh/userauth/ops

Optionally you can add your public SSH key (mykey.pub) to both the root and ops sections in the provision_xx.sh scripts.

For example, to add a SSH key to a provision_xx.sh script, copy and paste contents of mykey.pub for the ops user. Doing so will ensure that the key is distributed to all servers in the cluster for both root and ops during bootstraping.

(
cat <<'EOP'
cert-authority ssh-rsa AAA...5N apcera-user-ca
ssh-rsa AAA...4n/D apcera-special-ops
ssh-rsa AAV...45n/D mykey
EOP
) > /etc/ssh/userauth/ops

Populate the cluster.conf file

To deploy a BareOS cluster, you manually populate the cluster.conf file. Review the populating the cluster.conf documentation so you are familiar with this configuration file.

The cluster.conf file uses an Apcera proprietary format called dconf. Refer to the documentation for details.

Download one of the example cluster.conf files for BareOS from the Customer Support Portal.

Update the cluster.conf file with the information specific to your environment. Required information is commented in the file as "TODO".

The following information describes specific configuration information in cluster.conf required for BareOS installations.

Add partition device for the IM host(s)

Refer to the configuring volum mounts documentation.

In the cluster.conf file for a BareOS installation, you must specify the IM partition to allocate for the running apcera jobs (containers, capsules, etc.). The partition information is required, and can be found in the chef.mounts.instance-manager section. The "device" value must match the name of the created partition that is allocated for the Apcera jobs. In the example below it is "/dev/sda2".

chef: {

 [...]

 "mounts": {
    "instance-manager": {
      "device": "/dev/sda2"
    }

Configure cluster authentication

You must configure one or more identity providers so that users can log on to the cluster and deploy containers.

You configure the identity provider in the chef.continuum.auth_server.identity section of cluster.conf.

For initial BareOS cluster deployments you can use basic authentication as described below.

For production deployments you should use a supported third-party identity provider such as Google Auth or LDAP.

Basic auth

In the example BareOS cluster.conf, basic auth is enabled and the user name "admin" and password are present. Update the password accordingly.

The "admin" user is also listed in the chef.continuum.admins section section of cluster.conf. This is the root admin user for accessing the cluster after initial deployment. This entry is required otherwise the user cannot access the cluster.

"chef"
  "continuum"
    "auth_server": {
      "identity": {
        "basic": {
          "enabled": true,
          "users": [
            {
              "name": "admin",
              "password": "MyStRoNgPaSsWoRd"
            }
          ]
        },
      },
    }
    "admins": [
      "username@apcera.me"
    ],
  }
}

Since the basic auth user name is "admin", you must the update the "admins" section to be as follows. Note that the @apcera.me suffix distinguishes the "basic auth" usernames from names that come from other auth providers, such as kahuna@gmail.com.

 "admins": [
       "admin@apcera.me"
     ],

Other auth providers

Apcera supports several identity providers in addition to Basic auth. Refer to the documentation for your chosen provider.

Add public SSH key

To access to the Orchestrator host and cluster hosts remotely using SSH, add your SSH key to cluster.conf as shown below. See also adding SSH keys for Orchestrator access above.

chef: {
  "continuum": {
    "ssh": {
      "custom_keys":[ "ssh-rsa SSHKEY" ]
    },
    ...
  }
}

Deploy the cluster

To deploy the latest Apcera Platfform release from the cloud, use the command orchestrator-cli deploy --update-latest-release --config cluster.conf. Refer to the Orhestrator documentation for complete details on the deployment process.

Alternatively, to speed up deployment time, you can download the release bundle tarball and copy it to the Orchestrator host. This will prevent various round-trips between the Orchestrator server and the internet. This approach is described below.

(1) Download the appropriate release from the Customer Support Portal.

NOTE: To upgrade to release 2.4.0, you must be on either 2.2.3 or 2.2.2.

(2) Securely copy the cluster.conf file and release bundle tarball to the Orchestrator host.

For example:

scp cluster.conf orchestrator@172.27.16.41:~
scp release-2.4.0-f284c8e.tar.gz orchestrator@172.27.16.41:~

(3) Connect to the Orchestrator host remotely via SSH:

ssh -A orchestrator@172.27.16.41

(4) For first time deployments only, you must initialize the Orchestrator DB:

$ orchestrator-cli init

(5) Validate the cluster configuration:

Always perform a dry run to validate the cluster.conf file.

$ orchestrator-cli deploy -c bareos-cluster.conf --release-bundle release-2.4.0-f284c8e.tar.gz --dry-run

(6) Deploy the cluster:

$ orchestrator-cli deploy -c bareos-cluster.conf --release-bundle release-2.4.0-f284c8e.tar.gz

Troubleshooting

Use orchestrator-cli help to list available command options.

Make sure you ran a --dry-run and validated the cluster.conf file.

If the initial run of orchestrator-cli deploy fails, try to resolve the error and re-run the deployment. Note that a failed deploy may result in only partial monitoring of the servers in the cluster.

To troubleshoot deployment errors, start by viewing the chef-client-*.log on the Orchestrator host:

Run command orchestrator-cli ssh to list all the components (that Orchestrator can connect to).
Run command ls -ltr to list the logs on that host.
Run command less chef-client-20151211030033.log (for example) to scroll through the log and list the errors.

If an iniitial deployment fails and you cannot resolve the issue, to clean up after a failed deployment run orchestrator-cli teardown and then on each of the BareOS hosts run rm /etc/chef/* to reset their Chef configuration. Note that each time you do a tear down you will have to update your DNS records.

To pull the component logs:

orchestrator-cli collect logs all

On the different hosts being setup, the logs are in /var/log/orchestrator-agent/. The file to look at is "current":

$  tail -f /var/log/orchestrator-agent/current

Connect to the cluster

Once you have successfully deployed the cluster, connet to it as follows.

Configure DNS

Refer to the Configuring DNS documentation.

To access the cluster after deployment, you must create public DNS records that point to the cluster HTTP Router (NGNIX). You will need to create two DNS records that point to the IP address of the router or to the load balancer if you are running multiple routers:

  • Fully-qualified domain name (FDQN): {DOMAIN} and
  • Wildcard entry: *.{DOMAIN}

For example, using dnsmasq (/etc/dnsmasq.conf), the *.{DOMAIN} and {DOMAIN} are set to apcera.test and point to the IP address of the host in the cluster running the "router" component as defined in cluster.conf. In the example config this is the Central host at 192.168.50.31:

address=/#.apcera.test/192.168.50.31
address=/.apcera.test/192.168.50.31

Log on to the web console

Make sure that the browser is using the proper DNS.

Log on to the web console at http://console.bareos.apcera.test where "bareos" is the cluster name and "apcera.test" the domain.

Use APC

Download the APC client and install it.

Target your cluster and log in:

$ ./apc target http://apcera.test
Note: connections over HTTP are insecure and may leak secure information.

Targeted [http://apcera.test]
$ ./apc login --basic

Connecting to http://apcera.test...

Username: admin
Password: ********

Install Apcera packages

Refer to the Working with Packages documentation.

Apcera requires packages to create containers. You can use APC to import package files (*.cntmp) or build packages from scripts (*.conf).

The bare minimum package requirements for an Apcera cluster are the two Ubuntu OS packages shown below. Note that these two packages will only let you deploy Docker images and capsules, not applications from source code.

  • ubuntu-14.04-apc3.cntmp (or ubuntu-14.04.conf)
  • ubuntu-14.04-build-essential-apc3.cntmp (ubuntu-14.04-build-essential-apc4.conf)

To deploy applications from source code you must import or build the packages that satisfy the application's dependencies.

To import package files using APC, log on to the Apcera Support site and dowload the latest packages ZIP file. Extract the contents of the ZIP and import the packages as follows:

apc import *.cntmp --batch    // To import all package files
apc import go-1.5.3.cntmp     // To import an individual package file

Alternatively you can use APC to build packages using the package script file:

apc package build runtimes/go-1.5.3.conf

The script builds the *.cntmp file, which you can then import using APC as described above.

See also the knowledgebase article Adding Packages to your Apcera Platform Enterprise Edition Environment at the Support site.

Package details and troubleshooting

Every package binary has both depends and provides within its metadata. If the depends are not met by a matching provides somewhere within the Apcera list of available packages then the deployment of the job fails with an error such as the following:

Error - No packages satisfy dependency "os.linux"

An Apcera container image requires both the OS and execution parts in the binary. The Apcera approach to building containers is different than Docker. The Docker approach is based on a Dockerfile where you explicitly specifying all the dependencies and packaging all dependencies at once in a single image. Apcera on the other hand builds the container based on the packages that meet the dependencies.

The "ubuntu-14.04-apc3.cntmp" package provides the following to satisfy the OS dependency:

  • "os.linux"
  • "os.ubuntu"
  • "os.linux-14.04"
  • "os.linux-14.04-apc3"

Apcera requires the os.linux package to run a Docker image because the system creates a short helper job to download the image. That short lived job runs on an IM and depends on a base Linux package. Apcera supports running Docker images, but does so using its own container technology.

Apcera Hello World

(1) Import (or build and import) the following packages using APC:

  • "ubuntu-14.04-apc3.cntmp"
  • "nginx-1.11.1.cntmp"

(2) Clone or download the Apcera sample apps from the GitHub repository.

git clone https://github.com/apcera/sample-apps.git

(3) Deploy a static web site using APC.

$ apc app create mywebsite --start --batch

(4) Open the web site in a browser. It should print out "Hello Apcera" as expected.

...
App should be accessible at "http://mywebsite-h3f435.apcera.test"
Success!