Creating and Using Virtual Networks
Within an Apcera cluster you can define one or more virtual networks, each of which is allocated its own subnet. A job that belongs to a virtual network is allocated a virtual IP address from the network's subnet and automatically gets network connectivity to all other jobs on the same network. Both HTTP and UDP traffic between jobs on the same virtual network is tunneled over the underlying network and isolated from other jobs not on the network.
For example, the following illustration shows two virtual networks named vnet-1 and vnet-2. Jobs J1 and J2 (running on Host-1) and J5 and J6 (running on Host-2) belong to vnet-1, while jobs J3, J4 and J7 belong to vnet-2. The size of the subnet assigned to each virtual network is /24 (in CIDR notation). Virtual IP addresses are assigned to jobs out of this subnet. In the case illustrated above, members of the "vnet-1" network are allocated IP addresses between 192.168.1.1–192.168.1.255, and members of the "vnet-2" network are allocated IPs between 192.168.2.1–192.168.2.255.
Note: Virtual networks on Google Cloud Engine (GCE) or Microsoft Azure requires Apcera Platform release 2.2.1. In addition, Enterprise Edition users must modify their cluster.conf file to enable virtual network support on those cloud providers (see Enabling VXLAN). Community Edition release 2.2.1 and later supports virtual networks on GCE and Azure automatically without any extra configuration.
Creating and Deleting Virtual Networks
To create a virtual network you use the
apc network create <network> command, where
<network> is the network's local name or fully-qualified name. For example, the following creates a virtual network named
prod-net in the current namespace:
apc network create prod-net
The following command creates a network with the same local name in the
apc network create /prod::prod-net
Each virtual network you create is allocated its own subnet. To view a virtual network's subnet you can use the
apc network show <network> or
apc network list commands. See Listing and Showing Virtual Networks.
To delete a virtual network, use the
apc network delete <network> command:
apc network delete /prod::prod-net
Joining Jobs to a Virtual Network
apc network join <network> --job <job-name> command joins an existing job to a virtual network. For example, the following joins the job named
testapp to the network
test-network located in the
apc network join /networks::test-network --job testapp
You can optionally assign a discovery address to the job using the
apc network join <network> --job <job-name> --discovery-address <string> command. The discovery address is used to configure a DNS address through which instances can be discovered by other instances belonging to the same virtual network. For instance, the following joins the
testapp job to the
test-network and assigns the job the discovery address of
apc network join test-network --job testapp --discovery-address test-app
Any other jobs that are members of
test-network can access this job at
test-app.apcera.local. For more information, see Network discovery.
You can also join a job to a network when creating or updating a job or application. To do this, include the
--network parameter in the
apc app create (or
apc app update) command to specify the network to which the job should be added. For example, the following command creates an application named
app2 and also adds it to the network
apc app create testapp --network test-network
Jobs that join a network are assigned virtual IPs from the network's subnet. To see which jobs belong to a network and their associated IP addresses, you use the
apc network show command (see Listing and Showing Virtual Networks).
A job can be a member of one virtual network at a time. If a job is already a member of a virtual network, APC will display an error message if you try to add it to another network:
apc network join other-network --job testapp Error: job is already a member of a network.
If a job has multiple instances, all instances will be allocated their own IPs and be visible on the network.
Removing Jobs from a Virtual Network
To remove a job from a virtual network, you can use the
apc network leave command or
apc app update command. For example, the following are equivalent:
apc network leave /networks::test-network --job testapp apc app update testapp --network /networks::test-network -leave
Listing and Showing Virtual Networks
apc network list command lists available virtual networks, including the name, namespace, and the network's assigned subnet. For example, the following shows that there are two virtual networks (
prod) that are assigned subnets of 192.168.1.0/24 and 192.168.2.0/24, respectively.
apc network list Working in "/sandbox/joe.user" ╭──────────────┬──────────────────────┬────────────────╮ │ Network Name │ Namespace │ Subnet │ ├──────────────┼──────────────────────┼────────────────┤ │ prod │ /apcera/prod │ 192.168.2.0/24 │ │ net1 │ /dev/tests │ 192.168.1.0/24 │ ╰──────────────┴──────────────────────┴────────────────╯
To view details about a specific virtual network, use the
apc network show <network> command. The output of this command (as shown below) displays the network's assigned subnet, as well as any member jobs and their assigned IP addresses and MAC addresses. The Network Address field shown below indicates that the
net1 network is assigned a subnet of 192.168.1.0/24. The Network Endpoints field indicates the network has two member jobs named
node-2, which are assigned IP addresses of 192.168.1.1 and 192.168.1.2, respectively.
apc network show network::/dev::net1 ╭────────────────────┬──────────────────────────────────────╮ │ net1 │ │ ├────────────────────┼──────────────────────────────────────┤ │ Name: │ net1 │ │ FQN: │ network::/dev/tests::net1 │ │ UUID: │ 773fd48c-241d-4975-8379-cca21ed3e8c8 │ │ Description: │ │ │ Created By: │ email@example.com │ │ Updated By: │ │ │ Network Address: │ 192.168.1.0/24 │ │ Network Endpoints: │ │ │ Job FQN: │ job::/dev::node-1 │ │ Interface: │ │ │ IPv4 address: │ 192.168.1.1/24 │ │ MAC: │ da:dc:a3:64:51:30 │ │ Job FQN: │ job::/dev::node-2 │ │ Interface: │ │ │ IPv4 address: │ 192.168.1.2/24 │ │ MAC: │ ba:44:f8:b7:4a:5a │ ╰────────────────────┴──────────────────────────────────────╯
Enabling Broadcast and Multicast Routes
By default, network broadcast and multicast networks are not enabled in a virtual network. You can optionally enable broadcast and multicast routes on jobs in a virtual network. Enabling broadcast on a job provides a well-defined IP address (255.255.255.255) that all jobs in the same virtual network that also have broadcast can access. Enabling a multicast route enables the job to to join the multicast group you specified (22.214.171.124/24, for example). All jobs in the same multicast group can communicate with each other.
To enable broadcast or multicast support for a job, the job must belong to a virtual network, for example:
apc network join devnet --job app1
To enable the broadcast route on a job, update the job definition with the
-be) flag, for example:
apc job update app1 --network devnet --broadcast-enable
255.255.255.255 to the job's routing table. You can confirm this by SSH'ing into the job instance and running the Linux
route -n command, for example:
apc app update app1 --allow-ssh apc app connect app1 # route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 169.254.0.6 0.0.0.0 UG 0 0 0 veth_b0681fbe 169.254.0.6 * 255.255.255.254 U 0 0 0 veth_b0681fbe 192.168.1.0 * 255.255.255.0 U 0 0 0 vi-21e8-066ea5 255.255.255.255 * 255.255.255.255 UH 0 0 0 vi-21e8-066ea5
To enable multicast group support for a job, update the job with the
--multicast-add flag and specify the CIDR of the network subnet to enable for multicast communication, for example:
apc job update app1 --network devnet --multicast-add 126.96.36.199/24
Note: The CIDR must specify a valid multicast address range between 188.8.131.52 and 184.108.40.206.
To remove the broadcast route from a job, you update the job with
apc job update app1 --network devnet --broadcast-disable
To remove a multicast group support from a job, you update the job with the
--multicast-delete flag, specifying the same CIDR used to add the multicast route:
apc job update app1 --network devnet --multicast-delete 220.127.116.11/24
For a complete example of using the broadcast route to communicate between jobs , see the broadcast-virtual-net sample application. The sample consists of two Go applications: one app (the server) listens on its broadcast address (255.255.255.255) for UDP connections, and a client application that sends UDP messages to the broadcast address.
Policy and Virtual Networks
Virtual networks are resources within an Apcera cluster (like jobs or packages) and have their own resource type ("network"). For instance, the fully-qualified name of the virtual network named
retail_network in the
/prod namespace is
network::/prod::retail_network. Policy can be written against virtual networks to control the following:
- If a particular user is allowed to create or join a virtual network in a given namespace.
- If a particular job is allowed to join a virtual network in a given namespace.
- The number of virtual networks that can be created in a a given namespace.
For more information on managing policy and virtual networks, see the following:
- Network Permissions lists and describes available permissions (claims) for network resources.
- Network Policy Example provides example policy documents for virtual networks.
Virtual Network Reachability Example
This example demonstrates basic network reachability and discovery between two capsules in the same virtual network. Both capsules are assigned network discovery addresses. You then connect (SSH) to each capsule and ping the other using its assigned discovery address.
To ping a capsule from another capsule in the same network:
Create a new virtual network in the current namespace:
apc network create ping-net
Create two capsules,
apc capsule create cap-1 --image linux --batch apc capsule create cap-2 --image linux --batch
Join each capsule to
ping-netand assign a network discovery address to each one:
apc network join ping-net --job cap-1 --discovery-address cap-1 apc network join ping-net --job cap-2 --discovery-address cap-2
This results two DNS entries being added to the virtual network,
cap-2.apcera.local, that are assigned to the
apc capsule connect cap-1
cap-2at its DNS address,
root@ip-169-254-0-41:/root$ ping cap-2.apcera.local PING cap-2.apcera.local (192.168.3.2) 56(84) bytes of data. 64 bytes from 192.168.3.2: icmp_seq=1 ttl=64 time=0.353 ms 64 bytes from 192.168.3.2: icmp_seq=2 ttl=64 time=0.080 ms 64 bytes from 192.168.3.2: icmp_seq=3 ttl=64 time=0.085 ms
In another terminal, connect to
apc capsule connect cap-2 root@ip-169-254-0-41:/root$ ping cap-1.apcera.local PING cap-1.apcera.local (192.168.3.1) 56(84) bytes of data. 64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=0.353 ms 64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=0.080 ms 64 bytes from 192.168.3.1: icmp_seq=3 ttl=64 time=0.085 ms
When joining a job to a virtual network you can optionally specify the job's discovery address. This address is used to create a DNS address through which job instances can be discovered by other instances belonging to the same virtual network. The DNS address is composed of the discovery address you provide concatenated with
.apcera.local. For example, if you assign the discovery address "myapp" to a job, the DNS address for the job instance will be
You assign a network discovery address to a job with the
--discovery-address flag when joining the job to a network.
The syntax is as follows:
apc network join <network> --job <job-name> --discovery-address <string>
Or, using shortcut flags:
apc network join <network> -j <job-name> -da <string>
For example, the following command adds the job
app1 to the
testnet network and assigns the discovery address
my-app to the job:
apc network join testnet --job myapp --discovery-address my-app
This results in the DNS address
my-app.apcera.local being assigned to the job. Any other job that's a member of the same virtual network can access the job at this address. See Virtual Network Reachability Example for an example of this. Jobs outside a virtual network cannot lookup or connect to jobs in that network.
Note: If a job has more than one instance, one is picked randomly when it is looked up.
Discovery addresses are case-insensitive and may consist of letters and numbers. Dots and hyphens are also allowed, except at the beginning and end of the discovery address.
If a job hasn't been assigned a discovery address, it can still be discovered by its UUID. For example, a job with the UUID of "f81d4fae-7dec-11d0-a765-00a0c91e6bf6" can be discovered at the address of
You can find a job's UUID in a number of ways:
- At runtime, job instances can read the
CNTM_JOB_UUIDenvironment variable to get the job's UUID.
- Using APC, you can find a job's UUID with the
apc job list -lcommand. The
-loption includes each job's UUID in the command output.
When viewing a job's details view in the web console, it's UUID is displayed in the browser's location field.