Creating and Using Virtual Networks

Within an Apcera cluster you can define one or more virtual networks, each of which is allocated its own subnet. A job that belongs to a virtual network is allocated a virtual IP address from the network's subnet and automatically gets network connectivity to all other jobs on the same network. Both HTTP and UDP traffic between jobs on the same virtual network is tunneled over the underlying network and isolated from other jobs not on the network.

For example, the following illustration shows two virtual networks named vnet-1 and vnet-2. Jobs J1 and J2 (running on Host-1) and J5 and J6 (running on Host-2) belong to vnet-1, while jobs J3, J4 and J7 belong to vnet-2. The size of the subnet assigned to each virtual network is /24 (in CIDR notation). Virtual IP addresses are assigned to jobs out of this subnet. In the case illustrated above, members of the "vnet-1" network are allocated IP addresses between 192.168.1.1–192.168.1.255, and members of the "vnet-2" network are allocated IPs between 192.168.2.1–192.168.2.255.

Alt text

Note: Virtual networks on Google Cloud Engine (GCE) or Microsoft Azure requires Apcera Platform release 2.2.1. In addition, Enterprise Edition users must modify their cluster.conf file to enable virtual network support on those cloud providers (see Enabling VXLAN). Community Edition release 2.2.1 and later supports virtual networks on GCE and Azure automatically without any extra configuration.

Creating and Deleting Virtual Networks

To create a virtual network you use the apc network create <network> command, where <network> is the network's local name or fully-qualified name. For example, the following creates a virtual network named prod-net in the current namespace:

apc network create prod-net

The following command creates a network with the same local name in the /prod namespace.

apc network create /prod::prod-net

Each virtual network you create is allocated its own subnet. To view a virtual network's subnet you can use the apc network show <network> or apc network list commands. See Listing and Showing Virtual Networks.

To delete a virtual network, use the apc network delete <network> command:

apc network delete /prod::prod-net

Joining Jobs to a Virtual Network

The apc network join <network> --job <job-name> command joins an existing job to a virtual network. For example, the following joins the job named testapp to the network test-network located in the /networks namespace:

apc network join /networks::test-network --job testapp

You can optionally assign a discovery address to the job using the apc network join <network> --job <job-name> --discovery-address <string> command. The discovery address is used to configure a DNS address through which instances can be discovered by other instances belonging to the same virtual network. For instance, the following joins the testapp job to the test-network and assigns the job the discovery address of test-app.

apc network join test-network --job testapp --discovery-address test-app

Any other jobs that are members of test-network can access this job at test-app.apcera.local. For more information, see Network discovery.

You can also join a job to a network when creating or updating a job or application. To do this, include the --network parameter in the apc app create (or apc app update) command to specify the network to which the job should be added. For example, the following command creates an application named app2 and also adds it to the network test-network:

apc app create testapp --network test-network

Jobs that join a network are assigned virtual IPs from the network's subnet. To see which jobs belong to a network and their associated IP addresses, you use the apc network show command (see Listing and Showing Virtual Networks).

A job can be a member of one virtual network at a time. If a job is already a member of a virtual network, APC will display an error message if you try to add it to another network:

apc network join other-network --job testapp
Error: job is already a member of a network.

If a job has multiple instances, all instances will be allocated their own IPs and be visible on the network.

Removing Jobs from a Virtual Network

To remove a job from a virtual network, you can use the apc network leave command or apc app update command. For example, the following are equivalent:

apc network leave /networks::test-network --job testapp

apc app update testapp --network /networks::test-network -leave

Listing and Showing Virtual Networks

The apc network list command lists available virtual networks, including the name, namespace, and the network's assigned subnet. For example, the following shows that there are two virtual networks (net1 and prod) that are assigned subnets of 192.168.1.0/24 and 192.168.2.0/24, respectively.

apc network list
Working in "/sandbox/joe.user"
╭──────────────┬──────────────────────┬────────────────╮
│ Network Name │ Namespace            │ Subnet         │
├──────────────┼──────────────────────┼────────────────┤
│ prod         │ /apcera/prod         │ 192.168.2.0/24 │
│ net1         │ /dev/tests           │ 192.168.1.0/24 │
╰──────────────┴──────────────────────┴────────────────╯

To view details about a specific virtual network, use the apc network show <network> command. The output of this command (as shown below) displays the network's assigned subnet, as well as any member jobs and their assigned IP addresses and MAC addresses. The Network Address field shown below indicates that the net1 network is assigned a subnet of 192.168.1.0/24. The Network Endpoints field indicates the network has two member jobs named node-1 and node-2, which are assigned IP addresses of 192.168.1.1 and 192.168.1.2, respectively.

apc network show network::/dev::net1
╭────────────────────┬──────────────────────────────────────╮
│ net1               │                                      │
├────────────────────┼──────────────────────────────────────┤
│ Name:              │ net1                                 │
│ FQN:               │ network::/dev/tests::net1            │
│ UUID:              │ 773fd48c-241d-4975-8379-cca21ed3e8c8 │
│ Description:       │                                      │
│ Created By:        │ joe.user@apcera.com                  │
│ Updated By:        │                                      │
│ Network Address:   │ 192.168.1.0/24                       │
│ Network Endpoints: │                                      │
│  Job FQN:          │ job::/dev::node-1                    │
│   Interface:       │                                      │
│    IPv4 address:   │ 192.168.1.1/24                       │
│    MAC:            │ da:dc:a3:64:51:30                    │
│  Job FQN:          │ job::/dev::node-2                    │
│   Interface:       │                                      │
│    IPv4 address:   │ 192.168.1.2/24                       │
│    MAC:            │ ba:44:f8:b7:4a:5a                    │
╰────────────────────┴──────────────────────────────────────╯

Enabling Broadcast and Multicast Routes

By default, network broadcast and multicast networks are not enabled in a virtual network. You can optionally enable broadcast and multicast routes on jobs in a virtual network. Enabling broadcast on a job provides a well-defined IP address (255.255.255.255) that all jobs in the same virtual network that also have broadcast can access. Enabling a multicast route enables the job to to join the multicast group you specified (225.1.1.0/24, for example). All jobs in the same multicast group can communicate with each other.

To enable broadcast or multicast support for a job, the job must belong to a virtual network, for example:

apc network join devnet --job app1

To enable the broadcast route on a job, update the job definition with the --broadcast-enable (or -be) flag, for example:

apc job update app1 --network devnet --broadcast-enable

This adds 255.255.255.255 to the job's routing table. You can confirm this by SSH'ing into the job instance and running the Linux route -n command, for example:

apc app update app1 --allow-ssh
apc app connect app1
# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         169.254.0.6     0.0.0.0         UG    0      0        0 veth_b0681fbe
169.254.0.6     *               255.255.255.254 U     0      0        0 veth_b0681fbe
192.168.1.0     *               255.255.255.0   U     0      0        0 vi-21e8-066ea5
255.255.255.255 *               255.255.255.255 UH    0      0        0 vi-21e8-066ea5

To enable multicast group support for a job, update the job with the --multicast-add flag and specify the CIDR of the network subnet to enable for multicast communication, for example:

apc job update app1 --network devnet --multicast-add 224.1.1.0/24

Note: The CIDR must specify a valid multicast address range between 224.0.0.0 and 239.255.255.255.

To remove the broadcast route from a job, you update the job with --broadcast-disable (or -bd) flag:

apc job update app1 --network devnet --broadcast-disable

To remove a multicast group support from a job, you update the job with the --multicast-delete flag, specifying the same CIDR used to add the multicast route:

apc job update app1 --network devnet --multicast-delete 224.1.1.0/24

For a complete example of using the broadcast route to communicate between jobs , see the broadcast-virtual-net sample application. The sample consists of two Go applications: one app (the server) listens on its broadcast address (255.255.255.255) for UDP connections, and a client application that sends UDP messages to the broadcast address.

Policy and Virtual Networks

Virtual networks are resources within an Apcera cluster (like jobs or packages) and have their own resource type ("network"). For instance, the fully-qualified name of the virtual network named retail_network in the /prod namespace is network::/prod::retail_network. Policy can be written against virtual networks to control the following:

  • If a particular user is allowed to create or join a virtual network in a given namespace.
  • If a particular job is allowed to join a virtual network in a given namespace.
  • The number of virtual networks that can be created in a a given namespace.

For more information on managing policy and virtual networks, see the following:

Virtual Network Reachability Example

This example demonstrates basic network reachability and discovery between two capsules in the same virtual network. Both capsules are assigned network discovery addresses. You then connect (SSH) to each capsule and ping the other using its assigned discovery address.

To ping a capsule from another capsule in the same network:

  1. Create a new virtual network in the current namespace:

     apc network create ping-net
    
  2. Create two capsules, cap-1 and cap-2:

     apc capsule create cap-1 --image linux --batch
     apc capsule create cap-2 --image linux --batch
    
  3. Join each capsule to ping-net and assign a network discovery address to each one:

     apc network join ping-net --job cap-1 --discovery-address cap-1
     apc network join ping-net --job cap-2 --discovery-address cap-2
    

    This results two DNS entries being added to the virtual network, cap-1.apcera.local and cap-2.apcera.local, that are assigned to the cap-1 and cap-2 jobs, respectively.

  4. Connect to cap-1:

     apc capsule connect cap-1
    
  5. Ping cap-2 at its DNS address, cap-2.apcera.local:

     root@ip-169-254-0-41:/root$ ping cap-2.apcera.local
     PING cap-2.apcera.local (192.168.3.2) 56(84) bytes of data.
     64 bytes from 192.168.3.2: icmp_seq=1 ttl=64 time=0.353 ms
     64 bytes from 192.168.3.2: icmp_seq=2 ttl=64 time=0.080 ms
     64 bytes from 192.168.3.2: icmp_seq=3 ttl=64 time=0.085 ms
    
  6. In another terminal, connect to cap-2 and ping cap-1 at cap-1.apcera.local

     apc capsule connect cap-2
    
     root@ip-169-254-0-41:/root$ ping cap-1.apcera.local
     PING cap-1.apcera.local (192.168.3.1) 56(84) bytes of data.
     64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=0.353 ms
     64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=0.080 ms
     64 bytes from 192.168.3.1: icmp_seq=3 ttl=64 time=0.085 ms
    

Network Discovery

When joining a job to a virtual network you can optionally specify the job's discovery address. This address is used to create a DNS address through which job instances can be discovered by other instances belonging to the same virtual network. The DNS address is composed of the discovery address you provide concatenated with .apcera.local. For example, if you assign the discovery address "myapp" to a job, the DNS address for the job instance will be myapp.apcera.local.

You assign a network discovery address to a job with the --discovery-address flag when joining the job to a network.

The syntax is as follows:

apc network join <network> --job <job-name> --discovery-address <string>

Or, using shortcut flags:

apc network join <network> -j <job-name> -da <string>

For example, the following command adds the job app1 to the testnet network and assigns the discovery address my-app to the job:

apc network join testnet --job myapp --discovery-address my-app

This results in the DNS address my-app.apcera.local being assigned to the job. Any other job that's a member of the same virtual network can access the job at this address. See Virtual Network Reachability Example for an example of this. Jobs outside a virtual network cannot lookup or connect to jobs in that network.

Note: If a job has more than one instance, one is picked randomly when it is looked up.

Discovery addresses are case-insensitive and may consist of letters and numbers. Dots and hyphens are also allowed, except at the beginning and end of the discovery address.

If a job hasn't been assigned a discovery address, it can still be discovered by its UUID. For example, a job with the UUID of "f81d4fae-7dec-11d0-a765-00a0c91e6bf6" can be discovered at the address of f81d4fae-7dec-11d0-a765-00a0c91e6bf6.apcera.local.

You can find a job's UUID in a number of ways:

  • At runtime, job instances can read the CNTM_JOB_UUID environment variable to get the job's UUID.
  • Using APC, you can find a job's UUID with the apc job list -l command. The -l option includes each job's UUID in the command output.
  • When viewing a job's details view in the web console, it's UUID is displayed in the browser's location field.

    Alt text