Implementing Job Affinity

Jobs are the base unit of work in Apcera. A job is an executable workload comprising one or more packages and the metadata needed to run the job. There are various types of jobs, including apps, capsules, stagers, service gateways, and semantic pipelines.

Overview

Job affinity gives you the ability to dictate where a job runs in relation to another job. To use job affinity, it is important to understand certain architectural features of Apcera.

In an Apcera cluster, the Instance Manager (IM) component is responsible for running job instances. A cluster contains several IMs (typically between 3 - 5), each running on a separate node in the cluster. Job instances are assigned to IMs randomly based on availability, resource consumption, and other internal criteria. For example, if a cluster has 3 available IMs with similar loads, and there are 3 job instances to be run, each IM will be assigned 1 job instance.

Job affinity lets you override the default behavior and control where instances of a job run in relation to another job. By using job affinity you can ensure that two jobs always run on the same IM or never run on the same IM, or that a job should prefer to run on an IM with a certain job or avoid running on an IM with a certain job.

For example, consider a scenario in which you want to run a service on the same IM as an app that consumes the service. Or, perhaps you want to do the opposite: ensure that a service never runs on the same IM as an app that consumes that service. Job affinity lets you achieve either of these architectural objectives in an elegant fashion.

Using job affinity

There are two methods for implementing job affinity: job attract and job repel. The former is used to group two job instances on a single IM; the latter is used to disallow two job instances on the same IM. By default job affinity is a soft requirement, meaning Apcera will make a best effort to implement the affinity, but it is not guaranteed. Optionally you can make job affinity a hard requirement.

To create job attract affinity use the following syntax:

apc job attract <target-job> --to <attracted-job> [--hard]

The job attract command creates an affinity for instances of the target-job to start on IMs running instances of the attracted-job. Instances of the target-job can still run on other IMs if the IM running the attracted-job is not available or is sufficiently loaded. You can make job affinity a hard requirement using the --hard flag, in which case the target-job must run on the same IM as the attracted-job or not at all.

Hard affinity is enforced dynamically: if you stop an anchor job, then instances of a job attracted to it are stopped as well because of the "hard" rule.

To create job repel affinity use the following syntax:

apc job repel <target-job> --from <repelled-job> [--hard]

The job repel command creates an affinity for instances of the target-job to start on IMs not running instances of the repelled-job. Instances of the target-job can still run on the same IM as the repelled-job if other IMs are not available or are sufficiently loaded. You can make job anti-affinity a hard requirement using the --hard flag, in which case the IM running the repelled-job can never run the target-job.

After you create a job affinity tag, you must restart the target-job for the affinitization to take effect.

Instance anti-affinity

If you have a job with multiple instances, and you always want those instances to start on different Instance Managers, you can add a job repel tag to repel the job from itself.

apc job repel <my-job> --from <my-job> [--hard]

New instances of the job will be started on different Instance Managers. If you use the --hard flag, you will only be able to start one instance of each job per Instance Manager. e.g. If you have 5 IMs, you'll only be able to start 5 instances of the job.

If you don't use the --hard flag, then "soft" affinity is assumed, and the Apcera Platform will make a best effort to place the instances on different IMs. You get a wide distribution of instances, but you may end up with several instances on the same Instance Manager.

Here's an example of a job repelled from itself, increasing the instance count from 1 to 4, and verifying that each instance is running on a different Instance Manager host:

$ apc job repel my-job --from my-job
Success!

$ apc job instances my-job
Looking up "my-job"... done
Health Score: 100%
Running Instances: 1/1
╭──────────┬─────────┬────────┬───────────────────╮
│ UUID     │ Status  │ Uptime │ Host              │
├──────────┼─────────┼────────┼───────────────────┤
│ 13094305 │ RUNNING │ 23h2m  │ goodtwin-e4722660 │
╰──────────┴─────────┴────────┴───────────────────╯

$ apc job update my-job --instances 4
╭─────────────────────────────────╮
│       Job Update Settings       │
├────────────┬────────────────────┤
│       FQN: │ job::/myns::my-job │
│ Instances: │ 4                  │
╰────────────┴────────────────────╯

Is this correct? [Y/n]: Y
Application instances updated from 1 to 4 successfully.
Applying update... done
Start Command: ./bash_start.sh
Success!

$ apc job instances my-job
Looking up "my-job"... done
Health Score: 100%
Running Instances: 4/4
╭──────────┬─────────┬────────┬───────────────────╮
│ UUID     │ Status  │ Uptime │ Host              │
├──────────┼─────────┼────────┼───────────────────┤
│ 13094305 │ RUNNING │ 23h4m  │ goodtwin-e4722660 │
│ 2114bb70 │ RUNNING │ 23s    │ goodtwin-3ce6bbc7 │
│ 23e51ae9 │ RUNNING │ 23s    │ goodtwin-db33512d │
│ 4b302d81 │ RUNNING │ 23s    │ goodtwin-3ae87906 │
╰──────────┴─────────┴────────┴───────────────────╯

Removing job affinity

Removing an existing job affinity declaration reverts job instance assignment to the default behavior, that is, random based on IM availability, current resource consumption, and other internal criteria.

To remove job affinity use the following syntax:

apc job attract <target-job> --to <attracted-job> --remove

To remove job anti-affinity use the following syntax:

apc job repel <target-job> --from <repelled-job> --remove

You must restart the target-job for job affinity removal to take effect.

Applying job affinity tags

You can add or remove a job affinity tag to an existing job using either the web console or APC. For example, to do this using APC:

apc job attract <app-name> --to <app-name> --hard
...
Success!

In order for the job affinity tag to take an effect, you must restart the job.

If you add or remove a job affinity tag using the web console, you are prompted to restart the job. However, if you update a job and add or remove a job affinity tag using APC, you are not prompted to restart the job. You must remember to do this for the job affinity tag to take effect.

Job affinity examples

For the following examples, assume that you have 2 jobs, one called foo and the other called bar, and that foo is the target job and bar is the anchor job (either attracted or repelled).

apc job attract foo --to bar

This example creates a preference for new instances of target job foo to be run on the same IM where instances of attracted job bar are running. The optional --hard flag is not declared, thus the affinity is a soft requirement. In this case Apcera will make a best effort to run instances of the two jobs on the same IM, but it is not guaranteed.

apc job repel foo --from bar --hard

This example establishes the hard requirement that new instances of target job foo must never run on the same IM as instances of repelled job bar. With this configuration, if other IMs are not available to run job foo, it will not be started at all. For this reason you should be cautious about making job affinity a hard requirement.

apc job attract foo --to bar --remove

This example removes an existing job affinity arrangement and reverts to the default job instance assignment behavior.

All above examples assume that the target-job is restarted after creating or removing the affinitization.