Using Subnet Pools and Configurable Virtual Networks

This documentation describes how to use subnet pools for configurable virtual networks.

Requirements for using subnet pools

To use subnet pools for configurable virtual networks, you must use the following:

Requirement Description
APC (or API) Subnet pool creation is designed for cluster administrators. You cannot use the web console or multi-resource manifests to create subnet pools (but you can use either to join an existing pool when you create a network).
Local IPAM You must use local IPAM, not global IPAM. For new 3.0 installations, local IPAM is the default (no administration needed). If you are upgrading to release 3.0, you must enable local IPAM and you must migrate to new networks.
Policy Permission to create, read, update, and delete subnet pools is controlled by policy on the subnetpool::/ realm.

Using the default system subnet pool

The default system subnet pool is 10.224.0.0/12.

With the default configuration, the command apc network create <network-name> creates a new network in the default system subnet pool and in the default user namespace.

For example, the following command creates a network named net1 with the CIDR range 10.224.0.0/12.

apc network create net1

To specify a custom namespace for the network:

apc network create /prod/retail::customernetwork

To sepcify a custom pool for the network, use the --pool command:

apc network create net1 --pool my-pool-1

With the default configuration, any network created is given the default address range:

apc network show my-network
╭────────────────────┬──────────────────────────────────────╮
│ my-network         │                                      │
├────────────────────┼──────────────────────────────────────┤
│ Name:              │ my-network                           │
│ FQN:               │ network::/sandbox/admin::my-network  │
│ UUID:              │ 636f7c4c-8e34-417a-818e-323fbc22e47f │
│ Description:       │                                      │
│ Pool UUID:         │ system-default-pool-key              │
│ Created By:        │ admin@apcera.me                      │
│ Updated By:        │                                      │
│ Network Address:   │ 10.224.0.0/12                        │
│ Network Endpoints: │                                      │
╰────────────────────┴──────────────────────────────────────╯

For most use cases the default system subnet pool is recommended. If 10.224.0.0/12 conflicts with an underlying network, you can override the system default subnet pool using a custom default subnet pool.

Creating custom default subnet pool

A custom default subnet pool is a user-defined subnet pool and IP address range that is set to be the default pool.

If the default system subnet pool (10.224.0.0/12) conflicts with your network, you can define your own default subnet pool that will override the system default subnet pool.

To create a custom default subnet pool, you use the subnet pool create command with the --prefix argument specifying the desired subnet range and the --default flag to assign it as the default pool:

apc subnet pool create /::pool1 --prefix 10.192.0.0/12 --default

With this configuration, the command apc network create my-new-network creates a new virtual network in the root namespace using the custom default subnet pool (10.192.0.0/12). In this case the system default subnet pool is overridden by the custom range you have set as the default.

The typical namespace conventions do not apply to the default subnet pool. In other words, creating a default subnet pool in the /prod or /dev namespace does not limit the use of this pool to networks in the same namespace. As a best practice, create custom default subnet pool in the root namespace to avoid confusion.

For example:

apc network show my-new-network
╭────────────────────┬─────────────────────────────────────────╮
│ my-new-network     │                                         │
├────────────────────┼─────────────────────────────────────────┤
│ Name:              │ my-new-network                          │
│ FQN:               │ network::/sandbox/admin::my-new-network │
│ UUID:              │ 4aa7fdbc-348d-49ca-96f2-0d01a38f4fed    │
│ Description:       │                                         │
│ Pool UUID:         │ a20524f0-3a69-4c0a-802a-12ffa79140c5    │
│ Created By:        │ admin@apcera.me                         │
│ Updated By:        │                                         │
│ Network Address:   │ 10.192.0.0/12                           │
│ Network Endpoints: │                                         │
╰────────────────────┴─────────────────────────────────────────╯

To revert to the default system subnet pool, you will need to delete the custom default subnet pool you created.

Creating custom subnet pools

For testing purposes, or if your requirements dictate that you have additional networks but do not want to change the default, you can create one or more custom subnet pools and then assign a virtual network to each pool on creation using the --pool option with network create.

For example:

apc subnet pool create p1 --prefix 10.192.0.0/12
apc subnet pool create p2 --prefix 192.168.0.0/16
apc network create net1 --pool p1
apc network create net2 --pool p2

Specifying max containers per host

There is no limitation on the address range you can specify for subnet pools, but whatever you choose directly affects your cluster.

For example:

apc subnet pool create p1 --prefix 10.192.0.0/12
Creating subnet-pool... done
Success!
The current configuration allows a maximum of 2047 hosts and 510 containers per host.
To adjust these values consult the --max-containers-per-host parameter.

This network, by way of the subnet pool, can support 2047 Instance Managers (hosts) and a maximum of 510 containers per IM in that network.

Compare the above subnet pool with the following:

apc subnet pool create p2 --prefix 192.168.0.0/16
Creating subnet-pool... done
Success!
The current configuration allows a maximum of 127 hosts and 510 containers per host.
To adjust these values consult the --max-containers-per-host parameter.

This subnet allows a cluster size up to 127 IMs with 510 containers per host. In this case if you scale your cluster beyond 127 IMs, instances of these networks will not be scheduled on the new IMs. If you would like new jobs to be scheduled on the newly added IMs, you would need to create a new network and then move the jobs to this network. The old network will only scale to the old set of IMs.

If you need a /16 prefix and 127 IMs is too limiting but you are not going to have 500+ containers on every IM joining that network, you can use the --max-containers-per-host parameter (or -c) to limit the number of jobs on an IM that can join the network:

apc subnet pool create p3 --prefix 192.168.0.0/16 --max-containers-per-host 254
Creating subnet-pool... done
Success!
The current configuration allows a maximum of 255 hosts and 254 containers per host.
To adjust these values consult the --max-containers-per-host parameter.

Now you get 255 IMs and on each IM 254 containers can join a virtual network of that pool.

Subnet pool caveats

The last number of the CIDR address given with the --prefix option (or -p) should be greater than zero and less than or equal to 28 (> 0 and <= 28).

When using --max-containers-per-host (or -c), the value you provide should be a power of 2 minus 2 and between 6 and 1022. This number defines the number of bits for the suffix. Per standard networking practice, we do not allow all-zero and all-one addresses.

Managing subnet pools

This section describes how to manage subnet pools using APC, including:

List subnet pools

Use the command subnet pool list to view your cluster's available subnet pools. By default, the list is filtered by the current namespace.

For example:

apc subnet pool list
Working in "/sandbox/admin"
╭──────────────────┬────────────────┬───────────────┬─────────┬──────────────╮
│ Subnet Pool Name │ Namespace      │ Prefix        │ Type    │ User Default │
├──────────────────┼────────────────┼───────────────┼─────────┼──────────────┤
│ p1               │ /sandbox/admin │ 10.192.0.0/12 │ private │ true         │
╰──────────────────┴────────────────┴───────────────┴─────────┴──────────────╯

Use the command option -l to show the fully-qualified-names and UUID of each subnet pool.

For example:

apc subnet pool list -l
Working in "/sandbox/admin"
╭────────────────────────────────┬──────┬──────────────────────────────────────┬───────────────┬─────────┬──────────────╮
│ FQN                            │ Name │ UUID                                 │ Prefix        │ Type    │ User Default │
├────────────────────────────────┼──────┼──────────────────────────────────────┼───────────────┼─────────┼──────────────┤
│ subnetpool::/sandbox/admin::p1 │ p1   │ 7313917e-3bee-43bd-8ec6-a9ed2bd38f44 │ 10.192.0.0/12 │ private │ true         │
╰────────────────────────────────┴──────┴──────────────────────────────────────┴───────────────┴─────────┴──────────────╯

Show subnet pool details

The subnet pool show command shows detailed information about a specific subnet pool.

For example:

apc subnet pool show p1
╭──────────────────────────┬──────────────────────────────────────╮
│ p1                       │                                      │
├──────────────────────────┼──────────────────────────────────────┤
│ Name:                    │ p1                                   │
│ FQN:                     │ subnetpool::/sandbox/admin::p1       │
│ UUID:                    │ 7313917e-3bee-43bd-8ec6-a9ed2bd38f44 │
│ Description:             │                                      │
│ Created By:              │ admin@apcera.me                      │
│ Updated By:              │                                      │
│ User Default:            │ true                                 │
│ Pool Type:               │ private                              │
│ Prefix:                  │ 10.192.0.0/12                        │
│ Max Containers Per Host: │ 256                                  │
│ Networks:                │                                      │
│  (1) UUID:               │ 2de7447e-4090-43b2-94e8-666e241510b1 │
╰──────────────────────────┴──────────────────────────────────────╯

Delete subnet pool

The command subnet pool delete <subnet-pool-name> deletes an existing subnet pool from your cluster.

For example:

apc subnet pool delete p1

apc subnet pool delete /prod/dev::pool-a

The subnet pool must be empty to be deleted. This means the subnet pool cannot have any member networks or you will receive the following error:

apc subnet pool delete p1
Looking up "p1"... error
Error: subnet pool has 2 existing networks, which must be removed before the subnet pool can be deleted.

To delete a subnet pool with members networks, all such networks must first be deleted.

Policy requirements for subnet pools

The creation and use of subnet pools is governed by policy permissions on the realm subnetpool::/.

Policy is also required to create and join virtual networks.