Job Scheduling

This documentation describes how to use Apcera job scheduling to control the placement of your workloads across hybrid cloud infrastructures.

Overview

Apcera Platform job scheduling gives you the power to deploy your applications and services across hybrid environments, including cloud and on-premises infrastructures. Job scheduling lets you tag a job with metadata that identifies some desired or undesired property of an Instance Manager (IM). You can apply job scheduling tags based on provider, location, and machine characteristics. You can tag individual jobs using the APC client, or use policy rules to automate job scheduling for multiple jobs.

Check out the following video that demonstrates how to implement job scheduling.

Use case

Consider a scenario where your developer and QA teams use cloud infrastructures to develop and test applications but in production you want these apps to run on-premises. As depicted below, job scheduling lets you easily control the placement of your apps with precision. Jobs tagged “aws” are deployed to the IM on AWS. Jobs tagged with other tags are deployed to an IM in another cloud provider. Jobs tagged "onprem" are deployed to a on-premise host running OpenStack. Apcera components for managing jobs in the cluster also reside on-premise.

drawing

Job scheduling tags

By default the IMs in your cluster are deployed with a pre-defined set of job scheduling tags. There are three basic types of tags: provider, location, and machine.

The following table lists and describes examples of the provider tags for job scheduling.

PROVIDER TAG DESCRIPTION
aws Use to specify that the IM will run on a host provided by AWS
google Use to specify that the IM will run on a host provided by GCE
openstack Use to specify that the IM will run on a on-premise host running OpenStack

The following table lists and describes examples of the location tags for job scheduling.

LOCATION TAG DESCRIPTION
us-west-1 Use to specify that the IM will run on an AWS host in the us-west-1 region
eu-west-1 Use to specify that the IM will run on an AWS host in the eu-west-1 region
ap-southeast-1 Use to specify that the IM will run on an AWS host in the eu-west-1 region
us-central1-a Use to specify that the IM will run on an GCE host in the us-central1-a region
europe-west1-b Use to specify that the IM will run on an GCE host in the europe-west1-b region
asia-east1-c Use to specify that the IM will run on an GCE host in the asia-east1-c region

Machine tags for job scheduling will be specific to your cluster and requirements.

The above tags are examples only. The tags in your cluster will be unique to your requirements and environments. When you are ready to implement job scheduling, you can run the APC command apc cluster im list to view the IMs and tags in your cluster, or you can use the web console. Both of these methods for determining the available tags in your cluster are discussed in more detail below.

Tagging jobs

You use job scheduling tags to map a job to the best available IM. A job scheduling tag is a string that identifies some desired or undesired property of an IM. When you assign a tag to a job, you must also specify the quality of service for that scheduling tag: hard, soft, or negative.

To implement job scheduling, it is important to understand how jobs are assigned to IMs in a cluster. When you instruct the system to start a job (using APC or the Web Console), the Job Manager (JM) component sends a broadcast message to the IMs to start one or more instances of the job. The first IM to respond is assigned the job. The use of the hard and soft tag arguments alters the default job-to-IM job assignment.

Hard tags

A hard tag on a job must be fulfilled by an IM to run a job. If you tag a job with the hard requirement, only IMs with that same tag can respond to the start request from the JM. Any IM that is not tagged with the same tag as the job cannot respond to the start request. For example, if you tag a job with the hard tag “dc-1” and no IMs have this tag, the start request is not processed. Typical use cases for hard tags include the need to run a job on a specific provider or in a specific location, or requiring some relationship between two jobs.

Soft tags

A soft tag on a job should be fulfilled by an IM, but is not guaranteed. The response time of an IM to a start request from the JM is a function of the number of soft tags on the job contained in the request satisfied by that IM. For example, if you assign the soft tag to a job, initially only IMs with the same tag respond to the request. IMs not tagged with the same tag will wait to respond to the request. After a specified interval (default is 1 second), IMs that do not have the same tag as the job will answer the start request, under the assumption that IMs containing the same tag were not available or would have already responded.

Negatives tags

You can use a negation operator to preface a job scheduling tag and implement anti-affinity. A job scheduling tag beginning with the ~ character is interpreted as a request to run the job on IMs that do not satisfy the tag. For example, using the ~AWS tag instructs Apcera to run the job on an IM that is not tagged AWS.

Applying job scheduling tags

You can apply (add or remove) a job scheduling tag to an existing job using either the web console or APC. For example, to do this using APC:

apc app update <app-name> -ha <tag>
...
Success!

In order for the job affinity tag to take an effect, you must restart the job.

If you add or remove a job scheduling tag using the web console, you are prompted to restart the job. However, if you update a job and add or remove a job scheduling tag using APC, you are not prompted to restart the job. You must remember to do this for the job scheduling tag to take effect.

APC commands for job tags

There are several APC commands that you can use to apply tags to jobs, remove tags, and list tags. The following table lists and describes these tags:

APC COMMAND DESCRIPTION
apc app create Supports adding and removing job scheduling tags during app creation
apc job update Supports adding and removing job scheduling tabs during app update
apc cluster instance-manager list Lists the state of all IM components in the cluster
apc cluster instance-manager show UUID Provides details on an individual IM in the cluster
apc cluster tag list Lists each unique scheduling tag and the IMs that implement them
apc job attract Creates job affinity (See Job Affinity)
apc job repel Creates job anti-affinity (See Job Affinity)

apc app/capsule create

APC commands for creating apps and capsules support adding job scheduling tags. The table below lists the command options for declaring scheduling tags at job creation time:

TAG COMMAND DESCRIPTION
-ht, --hard-tags One or more (comma-separated) hard scheduling tags to create for the app or capsule.
-st, --soft-tags One or more (comma-separated) soft scheduling tags to add to the app or capsule.

The above commands are supported for apc app create and apc capsule create.

Examples:

apc app create my-app --hard-tags aws

apc capsule create my-cap --image ubuntu --soft-tags google

apc app create my-app --hard-tags aws, google

apc app/job update

APC commands for updating jobs support adding and removing job scheduling tags. The table below lists the command options for updating a job with a scheduling tag:

TAG COMMAND DESCRIPTION
-ha, --hard-tags-add Hard scheduling tags to add to the app or capsule.
-sa, --soft-tags-add Soft scheduling tags to add to the app or capsule.
-hd, --hard-tags-del Hard scheduling tags to delete from the the app or capsule.
-sd, --soft-tags-del Soft scheduling tags to delete from the the app or capsule.

The above commands are supported for apc app update and apc capsule update.

Examples:

apc app update my-app --hard-tags-add aws

apc job update my-cap --soft-tags-add google

apc job update my-app -hd aws

For apps you can use either apc app update or apc job update. For capsules you must use apc job update.

Identifying tags

There are several APC commands for listing and showing the tags available for use for job scheduling.

cluster instance-manager list

The ‘cluster instance-manager list’ command lists the IM nodes available to run job instances and the corresponding tags defined for each IM.

For example (the second command is the shortcut version):

apc cluster instance-manager list

apc cluster im list

cluster instance-manager show

The ‘cluster instance-manager show’ command lists details about a particular IM, including available tags, as well as the live job instances running on the specified IM.

For example (the second command is the shortcut version):

apc cluster instance-manager show <IM Name> and <IM UUID>

apc cluster im show <IM Name> and <IM UUID>

cluster tag list

The ‘cluster tag list’ command lists the available job scheduling tags in the cluster and the number of IMs that implement each listed tag.

For example:

apc cluster tag list

apc cluster version

The ‘cluster info’ command returns the version of the cluster. This may be needed to verify what system-defined tags are available for your cluster and for support-related needs.

apc cluster info

Example result:

Apcera Platform version: 449.3.0

Monitoring IMs using the web console

The web console interface (http://console.cluster-name.domain) provides an Instance Manager module that you can use to visualize and monitor the IMs in your cluster.
The Cluster module lists the hostname, UUID, uptime, and job scheduling tags for each IM in your cluster:

In addition, the Cluster module provides you with a real-time graphical display of each IM in your cluster. This display gives you the names of the jobs running on each IM, the number of jobs and instances running on each IM, and the job scheduling tags for each IM.

Using policy to auto-tag jobs

Apcera lets you use policy to automate job tagging and scheduling.

Policy is the anchor of trust across your environments, providing a single control and management plane for your hybrid deployments. Instead of tagging jobs manually, which may be prone to inconsistency and human error, the recommended approach is to use policy rules to automate the application of job scheduling tags.
For example, you can define a policy that automatically creates one or more tags for any jobs created in the designated policy realm. Using policy to assign job scheduling tags lets you automate job scheduling based on established criteria.

There are two types of policy rule claims you can use for job scheduling: tag.hard and tag.soft. The value of the claim is one or more tags that are automatically added to a job in the defined realm to which the policy applies (resource, namespace, or FQN). You can preface the claim value with the negation operator (~).

For example, consider the following policy rule:

job::/ {
    { schedulingTag.soft aws }
}

The realm for this policy is the job::/ resource. As such, this claim will be checked for all jobs in the cluster, and the system will apply the soft “aws” tag to any job created or updated in the cluster. Attempts to delete the tag from a job using APC will throw an error. To remove the tag, you must alter or disable the policy.

For more fine-grained tagging, you can create a policy that applies to a namespace, such as:

job::/test {
    { schedulingTag.hard aws }
}

The realm for this policy is the job::/test namespace. As such, this claim will apply the hard “aws” tag to any job created or updated in that namespace, but not to all jobs in the cluster.

Here is another example using the job::/prod namespace:

job::/prod {
    { schedulingTag.hard openstack }
}

The realm for this policy is the job::/prod namespace. As such, this claim will apply the hard “onprem” tag to any job created or updated in that namespace.

Lastly, here is example policy rule applying a job scheduling tag to a full FQN:

job::/sandbox/dev99::my-service {
    { schedulingTag.hard google }
}

The realm for this policy is an FQN. As such, this claim will apply the hard “google” tag to the named service job only. This is the equivalent of applying a tag using an APC command.

You must restart the job for the policy tagging to take effect. You can use the command job show <jobname> --compliance to check if the job is running on an IM without the tag. You can also use apc policy on to check if the policy is being applied to the job.

Using job scheduling with job affinity

Although job scheduling and job affinity are two distinct features, they are complementary. For example, you could use job scheduling to apply a hard tag to jobs in the job::/test namespace to run on IMs deployed to AWS EC2 infrastructure. You could then use job affinity to attract or repel other jobs to or from jobs that are tagged to run on AWS IMs.

Using staging tags

In addition to using tags to influence job scheduling, you can also use tags to control the scheduling of stagers and staging coordinators.

When you deploy a job, the Apcera staging process is responsible for creating the runtime package for the job. The components of the staging process, including stagers and staging coordinators, are jobs in the system. There may be some scenarios where you want to control or influence where a stager or staging coordinator job runs. Staging scheduling tags let you do this.

You use policy to implement stager scheduling. For example:

package::/sandbox/alfred {
   { staging.schedulingTag.hard "aws" }
}

This policy will force all stagers and staging coordinators that stage packages in the namespace "package::/sandbox/alfred" (and its descendent namespaces) to run on an IM that satisfies the "aws" tag. This tag may be needed if it is possible that an IM that the job cannot access (due to a hard tag) will host a stager or staging coordinator.

The system natively supports a hard affinity between the staging coordinator and any stagers it produces. Thus, the generic staging.schedulingTag.hard policy claim type operates on both staging coordinators and stagers.

Because staging coordinators are created dynamically by the system on job deployment or update, you cannot use APC to apply staging tags as you can with job scheduling tags. You must do this using policy.

Unlike job scheduling tags, there are no soft scheduling tags supported on the package realm. In addition, stagers cloned into a namespace are not affected by job scheduling tags (i.e., plain old scheduling tags).

Using Package Manager tags

The Package Manager (PM) is an Apcera component that stores and manages packages in the cluster. The first time an Instance Manager (IM) is tasked to run a job, the IM interfaces with the PM to download the packages the IM needs to run that job. Thereafter, the package is cached locally on the IM. If you are are running a production cluster, you will have more than one PM with which your IMs can communicate. If you are running an HA cluster, you will have at least three PMs.

As part of the Apcera hybrid cloud tagging feature, you can also tag a Package Manager, for example datacenter=“aws-west”. Any IM that has the same tag will first attempt to download the package from that tagged PM before trying other PMs.

Tagging a PM makes your cluster datacenter aware, so that an IM can first attempt to download packages from a PM within its local datacenter. For Staging Coordinators, the system creates an environment variable that informs the IMs of the local datacenter package cache. If there is no PM within the local datacenter, the IM will query the system as usual and download the packages from a randomly selected PM.

Like Instance Manager tags, Package Manager tagging is done at the cluster level using the cluster.conf file when you create or update a cluster. You cannot apply a PM tag using APC or the web console. If you want to use this feature, you must do so when creating or updating your cluster. See the documentation for your installation platform for details.