Implementing Job Affinity
Jobs are the base unit of work in Apcera. A job is an executable workload comprising one or more packages and the metadata needed to run the job. There are various types of jobs, including apps, capsules, stagers, service gateways, and semantic pipelines.
Job affinity gives you the ability to dictate where a job runs in relation to another job. To use job affinity, it is important to understand certain architectural features of Apcera.
In an Apcera cluster, the Instance Manager (IM) component is responsible for running job instances. A cluster contains several IMs (typically between 3 - 5), each running on a separate node in the cluster. Job instances are assigned to IMs randomly based on availability, resource consumption, and other internal criteria. For example, if a cluster has 3 available IMs with similar loads, and there are 3 job instances to be run, each IM will be assigned 1 job instance.
Job affinity lets you override the default behavior and control where instances of a job run in relation to another job. By using job affinity you can ensure that two jobs always run on the same IM or never run on the same IM, or that a job should prefer to run on an IM with a certain job or avoid running on an IM with a certain job.
For example, consider a scenario in which you want to run a service on the same IM as an app that consumes the service. Or, perhaps you want to do the opposite: ensure that a service never runs on the same IM as an app that consumes that service. Job affinity lets you achieve either of these architectural objectives in an elegant fashion.
Using job affinity
There are two methods for implementing job affinity:
job attract and
job repel. The former is used to group two job instances on a single IM; the latter is used to disallow two job instances on the same IM. By default job affinity is a soft requirement, meaning Apcera will make a best effort to implement the affinity, but it is not guaranteed. Optionally you can make job affinity a hard requirement.
job attract affinity use the following syntax:
apc job attract <target-job> --to <attracted-job> [--hard]
job attract command creates an affinity for instances of the
target-job to start on IMs running instances of the
attracted-job. Instances of the
target-job can still run on other IMs if the IM running the
attracted-job is not available or is sufficiently loaded. You can make job affinity a hard requirement using the
--hard flag, in which case the
target-job must run on the same IM as the
attracted-job or not at all.
Hard affinity is enforced dynamically: if you stop an anchor job, then instances of a job attracted to it are stopped as well because of the "hard" rule.
job repel affinity use the following syntax:
apc job repel <target-job> --from <repelled-job> [--hard]
job repel command creates an affinity for instances of the
target-job to start on IMs not running instances of the
repelled-job. Instances of the
target-job can still run on the same IM as the
repelled-job if other IMs are not available or are sufficiently loaded. You can make job anti-affinity a hard requirement using the
--hard flag, in which case the IM running the
repelled-job can never run the
After you create a job affinity tag, you must restart the
target-jobfor the affinitization to take effect.
If you have a job with multiple instances, and you always want those instances to start on different Instance Managers, you can add a job repel tag to repel the job from itself.
apc job repel <my-job> --from <my-job> [--hard]
New instances of the job will be started on different Instance Managers. If you use the
--hard flag, you will only be able to start one instance of each job per Instance Manager. e.g. If you have 5 IMs, you'll only be able to start 5 instances of the job.
If you don't use the
--hard flag, then "soft" affinity is assumed, and the Apcera Platform will make a best effort to place the instances on different IMs. You get a wide distribution of instances, but you may end up with several instances on the same Instance Manager.
Here's an example of a job repelled from itself, increasing the instance count from 1 to 4, and verifying that each instance is running on a different Instance Manager host:
$ apc job repel my-job --from my-job Success! $ apc job instances my-job Looking up "my-job"... done Health Score: 100% Running Instances: 1/1 ╭──────────┬─────────┬────────┬───────────────────╮ │ UUID │ Status │ Uptime │ Host │ ├──────────┼─────────┼────────┼───────────────────┤ │ 13094305 │ RUNNING │ 23h2m │ goodtwin-e4722660 │ ╰──────────┴─────────┴────────┴───────────────────╯ $ apc job update my-job --instances 4 ╭─────────────────────────────────╮ │ Job Update Settings │ ├────────────┬────────────────────┤ │ FQN: │ job::/myns::my-job │ │ Instances: │ 4 │ ╰────────────┴────────────────────╯ Is this correct? [Y/n]: Y Application instances updated from 1 to 4 successfully. Applying update... done Start Command: ./bash_start.sh Success! $ apc job instances my-job Looking up "my-job"... done Health Score: 100% Running Instances: 4/4 ╭──────────┬─────────┬────────┬───────────────────╮ │ UUID │ Status │ Uptime │ Host │ ├──────────┼─────────┼────────┼───────────────────┤ │ 13094305 │ RUNNING │ 23h4m │ goodtwin-e4722660 │ │ 2114bb70 │ RUNNING │ 23s │ goodtwin-3ce6bbc7 │ │ 23e51ae9 │ RUNNING │ 23s │ goodtwin-db33512d │ │ 4b302d81 │ RUNNING │ 23s │ goodtwin-3ae87906 │ ╰──────────┴─────────┴────────┴───────────────────╯
Removing job affinity
Removing an existing job affinity declaration reverts job instance assignment to the default behavior, that is, random based on IM availability, current resource consumption, and other internal criteria.
To remove job affinity use the following syntax:
apc job attract <target-job> --to <attracted-job> --remove
To remove job anti-affinity use the following syntax:
apc job repel <target-job> --from <repelled-job> --remove
You must restart the
target-jobfor job affinity removal to take effect.
Applying job affinity tags
You can add or remove a job affinity tag to an existing job using either the web console or APC. For example, to do this using APC:
apc job attract <app-name> --to <app-name> --hard ... Success!
In order for the job affinity tag to take an effect, you must restart the job.
If you add or remove a job affinity tag using the web console, you are prompted to restart the job. However, if you update a job and add or remove a job affinity tag using APC, you are not prompted to restart the job. You must remember to do this for the job affinity tag to take effect.
Job affinity examples
For the following examples, assume that you have 2 jobs, one called
foo and the other called
bar, and that
foo is the target job and
bar is the anchor job (either attracted or repelled).
apc job attract foo --to bar
This example creates a preference for new instances of target job
foo to be run on the same IM where instances of attracted job
bar are running. The optional
--hard flag is not declared, thus the affinity is a soft requirement. In this case Apcera will make a best effort to run instances of the two jobs on the same IM, but it is not guaranteed.
apc job repel foo --from bar --hard
This example establishes the hard requirement that new instances of target job
foo must never run on the same IM as instances of repelled job
bar. With this configuration, if other IMs are not available to run job
foo, it will not be started at all. For this reason you should be cautious about making job affinity a hard requirement.
apc job attract foo --to bar --remove
This example removes an existing job affinity arrangement and reverts to the default job instance assignment behavior.
All above examples assume that the
target-jobis restarted after creating or removing the affinitization.