File Share Services - NFS and APCFS

The Apcera Platform provides the Apcera File Share (APCFS) service for persistent container storage.

An APCFS service provides:

Configuring APCFS services

APCFS provides an implementation of the Network File System (NFS). NFS is a distributed protocol for accessing network files as if they were in the client's local file system. In addition, Apcera provides the nfs service gateway for integrating with an NFS server such as APCFS.

Starting with the Apcera 2.0.0 Release, when you create a cluster using the Apcera Community Edition it automatically creates:

  • A single APCFS NFS server (not HA) located on the central1 server.
  • A single exported NFS directory on the central1 server called /data.
  • A registered APCFS provider named apcfs.

Starting with the Apcera 2.0.0 Release, when you create a cluster using the Apcera Enterprise Edition, depending on the settings in your cluster.conf configuration file, the Apcera Platform creates:

  • Zero or more NFS servers (not HA) installed on their own VMs.
  • Zero or more Highly Available (HA) NFS servers installed at one or more data centers, each with one or more exported volumes.

The APCFS provider

You will need to create a provider for each mount point on each NFS server in order to read or write data on that NFS server. If you created multiple NFS servers you will want to create a new provider for each server. If you have multiple mount points on one server you will want to create a separate provider for each mount point.

The APCFS NFS provider is used to create isolated storage services. When you create a service based on the APCFS provider, the service creates a separate directory on the NFS server under the exported directory named /data/$UUID/. The service can bound to one or more jobs, all of which can access the files in the /data/$UUID/ directory.

If you create another service for the same APCFS provider the new service will create a new, unique directory with a new UUID.

This unique directory serves as the isolated storage for jobs bound to this service, keeping their data separate, secure, and distinct from other services created for the same provider. By default all NFS providers create isolated services. You can optionally create a shared provider for an NFS server mountpoint. All jobs bound to any service created for this provider will mount and share the same mount point exported by the NFS server. See Creating a connection to an external, shared NFS server.

If you create more jobs and bind them to the same service they all see the files in "nfs://$IP/data/export1/$UUID/“. If you destroy a job the data is still there.

To check and see if an APCFS provider has already been created for you:

apc provider list

You should see a provider named apcfs in the list, already created for you. If you don't see it and need to create one, you can register the provider using the following syntax:

apc provider register /apcera/providers::apcfs --type nfs --url nfs://{host-ip}:{port}/{export-path}?version=4 --description "Apcera NFSv4 File Service"

Where:

  • The string apcfs is the user-defined name of the NFS server you are registering as a provider.
  • The host-ip parameter is the IP address of the NFS server.
  • The port parameter is the NFS port. The default NFS port is 2049 unless overridden in the URL.
  • The export-path is the file path exported by the NFS server.
  • The version=4 tells the provider to use NFSv4. The APCFS provider supports multiple versions of the NFS protocol. See Support for NFS Protocols for more information
  • The description is optional but recommended.

If you do not provide an NFS version number the version will default to NFSv4 for Apcera Platform Release 2.4.0 and later. The version defaults to NFSv3 in earlier releases.

For example, the following commands register an APCFS provider in the /apcera/providers namespace. The NFSv4 server is located at the IP address 10.0.0.141 and exports the /data NFS volume:

apc provider register /apcera/providers::apcfs --type nfs --url nfs://10.0.0.141/data?version=4 --description "Apcera NFSv4 FS"

Or, to specify a different server, port and export path, creating the provider in your current namespace:

apc provider register nfs_provider2 --type nfs --url nfs://192.168.1.99:2050/mount/path?version=4 --description "NFSv4 FS #2"

Support for NFS Protocols

Starting with the Apcera 2.0.0 Release, the NFS servers deployed by the Apcera Platform all support NFSv3, NFSv4, and NFSv4.1 and the NFS service gateway can connect to any NFS server using NFSv3, NFSv4, or NFSv4.1.

To create an NFSv4 provider, append 'version=4' to the URL options, for example:

apc provider register nfs-ubuntu-1 --type nfs --url "nfs://$IP/data?version=4"

Or, add --version as an extended parameter, for example:

apc provider register nfs-ubuntu-1 --type nfs --url "nfs://$IP/data" -- --version 4

The version number can be "3", "4", or "4.1".

If you do not provide an NFS version number the version will default to NFSv4 for Apcera Platform Release 2.4.0 and later. The version defaults to NFSv3 in earlier releases.

Older builds with an Apcera internal NFS server (non-HA)

If your original, deployed cluster was built using a build prior to 508.1.0 (Released February 25, 2016), then the internal NFS server that was installed is NFSv3-only and remains NFSv3-only through successive upgrades. To upgrade these servers Apcera Enterprise customers have these options:

  1. Do nothing, remain on an NFSv3-only server.

  2. Upgrade the internal server from NFSv3 to one that supports NFSv3, v4 and v4.1. This requires modifications to your cluster.conf file, re-running orchestrator, and restarting all jobs that use the NFS server for persistent storage. See NFSv4 server cluster configuration for guidance.

  3. Leave the NFSv3 server alone and create a second server that can run NFSv3, v4, and v4.1. Create new NFSv4 providers that use the new server and use it for new jobs requiring persistent storage. This requires modifications to your cluster.conf file and re-running orchestrator. Jobs do not have to be restarted. See NFSv4 server cluster configuration for guidance. Note that with this set up you will need to precisely name the servers, such as "nfsv3-server" and "nfsv4-server".

Creating APCFS services and bindings

You can create an APCFS service and job binding using the apc service create and apc service bind commands.

The following command creates an APCFS service in your current namespace:

apc service create apcfs-service --provider /apcera/providers::apcfs --description "NFS storage"

And the following command binds a job to that NFS service:

apc service bind apcfs-service --job my-job

Here is another example of a APCFS service creation:

apc service create file-service-1 --provider /apcera/providers::apcfs --description "file system 1"

The first time that a job is bound to this service, a new, unique directory will be created on the NFS server under the server's exported directory, e.g. /data. This directory will be the root directory for every job that's bound to that service.

For instance, to create two Linux capsules that are bound to this service:

apc capsule create test-capsule-1 --image linux --allow-egress
apc service bind file-service-1 --job test-capsule-1
apc capsule create test-capsule-2 --image linux --allow-egress
apc service bind file-service-1 --job test-capsule-2

You can then connect to the capsule:

apc capsule connect test-capsule-1

If you run df -k you'll see the unique NFS export subdirectory mounted as /nfs/data/ on both capsules.

Both capsules share this directory, and both can read and write to files in this shared directory. This is why you should create a new service for each type of job, app, or capsule that needs persistent storage. If you use the same service for a bunch of different jobs one job could delete the data of another job, or read the data from another job.

  • Creating a new service allows you to isolate data so that it can only be used by the jobs that need to access that data.
  • Binding the same service to different jobs allows those jobs to share data between them.

Overriding the default mount path

By default APCFS services created from the NFS provider and bound to a job will mount at the path /nfs/$exported_volume_path. In the case of the non-HA server, that's /nfs/data. By default the HA server will mount at /nfs/apcfs-default.

You can override the default mount path using the --mountpath extended argument flag when you bind the job to the service.

For example:

apc service bind apcfs-service --job my-job -- --mountpath /mount/path

Or:

apc service bind file-service-1 --job test-capsule-1 -- --mountpath /new/mount/point

If you log into this capsule and run df -k you'll see the unique NFS export subdirectory mounted as /new/mount/point/.

APCFS example using a capsule

This example demonstrates how to bind a capsule to an external file share, in this case APCFS.

1. Create a capsule

apc capsule create mycap -i linux

2. Register the NFS provider if necessary (may already be registered)

apc provider register /apcera/providers::apcfs --type nfs --url "nfs://{ip}:{port}/{path}" -- --version=4

3. Create the APCFS service

apc service create apcfs-service -p /apcera/providers::apcfs

4. Bind the capsule to the APCFS service

apc service bind apcfs-service -j mycap

5. SSH into the capsule

apc capsule connect mycap

6. Type the following commands to view the default mount path (/nfs/data)

cd /
ls
cd nfs
ls
cd data
ls
exit

7. Unbind the job from the service

apc service unbind apcfs-service -j mycap

8. Bind the job to the APCFS service again at a different mount point

This time, override the default mount path using the --mountpath flag:

apc service bind apcfs-service -j mycap -- --mountpath /new/mount/point

9. SSH into the capsule

apc capsule connect mycap

10. Check the mount point

Run df -k and you'll see the unique APCFS export subdirectory mounted as /new/mount/point/:

screenshot

Creating a provider for the Enterprise Edition internal HA NFS server

Starting with the Apcera 2.0.0 Release, when you create a cluster using the Apcera Enterprise Edition you may deploy zero or more Highly Available (HA) NFS servers at one or more data centers, each with one or more exported volumes.

Assuming that you have Apcera deploy one HA NFS server with all default options selected, you'll end up with a 3-node GlusterFS cluster containing one volume named apcfs-default. Note that the default volume name is derived from the datacenter name. Without configuration changes (using all defaults), the datacenter name is default. However, if you explicitly name the datacenter in the configuration, the default volume name string is apcfs-<datacenter-name>.

Each of your Instance Managers (IMs) will be running a Ganesha-NFS daemon which connects to the Gluster cluster. The Ganesha-NFS daemon has a host entry apcfs-ha.localdomain, so any job running on that Instance Manager can make an NFS connection to nfs://apcfs-ha.localdomain/apcfs-default (or nfs://apcfs-ha.localdomain/apcfs-<datacenter-name> if you have provided a custom datacenter name).

To create a provider for the default HA NFS server, create a service for the provider, and bind that service to a capsule you could do this:

apc provider register /apcera/providers::apcfs-ha --type nfs --url "nfs://apcfs-ha.localdomain/apcfs-default?vers=4" --description "HA NFSv4" --batch
apc service create apcfs-ha-service -p /apcera/providers::apcfs-ha -d "export apcfs-default" --batch
apc capsule create test-capsule --image linux -ae --batch
apc service bind apcfs-ha-service -j test-capsule --batch -- --mountpath /var/lib/mysql
apc capsule connect test-capsule

The last command should connect you to the test-capsule capsule. Once connected df -k should list all file systems, including the HA NFS filesystem mounted on /var/lib/mysql.

Creating a connection to an external, shared NFS server

Connecting to an external, shared NFS server for the purpose of reading or writing existing data which is also shared with applications outside of the Apcera Platform can be done by creating a shared NFS provider.

A shared NFS provider connects directly to the mount point exported by the NFS server. All jobs bound to any service created for this provider will mount and share the same mount point exported by the NFS server.

Once you create a shared provider for an NFS server mountpoint, you cannot create an isolated provider for the same NFS server and mount point. The reverse is also true: If you have an isolated provider for an NFS server mountpoint, you cannot create a shared provider for the same NFS server and mount point.

Since the users inside of containers do not have the same UID/GID as a user outside of a container with the same name, the root user running a process inside of a container does not have the same UID as the root user on your external NFS server. Apcera jobs will not be able to chown or chmod files on the external NFS server, nor would they be able to install any applications that require the installation to chown or chmod files residing on the NFS server.

To create a shared provider use the extended parameter --shared true:

apc provider register legacy-nfs --type nfs --url nfs://{IP}/{mount point}?version={NFS version} --description "Legacy NFS server" -- --shared true

Troubleshooting connections to an NFS server

If you have your own NFS server and you want to use it for storage, you can create Apcera providers, services, and bindings that connect to your NFS servers.

If you have problems connecting to the NFS server from Apcera try these troubleshooting steps:

  • Does the bind fail with the error failed to mount: protocol not supported? This happens if you created an NFSv4 provider for a server that only supports NFSv3. Either upgrade the server so that it supports NFSv4 or delete the provider and create a new provider that uses NFSv3.
  • Do Docker apps fail with lock errors? Newer Docker apps require NFSv4, and will only work when your server supports NFSv4 and the provider you created was set up to use NFSv4.
  • Can you mount the export from the NFS server on another server? If you can't mount the export from somewhere else you won't be able to mount it from Apcera.
  • Try logging into the NFS server and tail the NFS log. Do you see connection attempts from Apcera?
    • If the logs show nothing then Apcera isn't even contacting the server. Check firewall settings.
    • If the logs show an error message then Apcera is connecting but something else is wrong. Google the error message and try to fix the problem.
  • If the server is in a different subnet than the Apcera cluster the connection request coming from Apcera has been NAT-ed and will be coming from a high port.By default most NFS servers reject requests coming from high ports, so make sure that your server's export settings enable connections from high ports. On Linux servers this means adding the unfortunately named "insecure" flag to the NFS server's export line. e.g.:
# /etc/exports
/opt/nfsdata *(rw,insecure)

The * is the list of IP addresses and/or subnets that are allowed to mount the /opt/nfsdata directory. We recommend that you do not use *. If the export is just for your Apcera cluster put the cluster's subnet in that spot. e.g.:

# /etc/exports
/opt/nfsdata 10.0.0.0/24(rw,insecure)

This is just an example. Use the subnet configured for your cluster.

APCFS example using Redmine

Redmine is an open source project management application. This example shows how to use an NFS store for Redmine's issue attachments. The example assumes you have a test Redmine job named redmine.

Create an NFS service for storing the Redmine data

1. Create the redmine-apcfs service.

apc service create redmine-apcfs -p /apcera/providers::apcfs

2. Bind the Redmine job to the APCFS service.

apc service bind redmine-apcfs -j redmine

To override the default and mount to a different path, use the --mountpath flag. For example:

apc service bind redmine-apcfs -j redmine -- --mountpath /mount/path

Test Redmine persistence

To test the NFS implementation, you create a directory for Redmine's attachments. From the URI used to register the provider, we know that NFS will be mounted at /nfs/data in the container (assuming you did not override the default mount path). We will do this using an app console session for the redmine job and then redeploy the app.

3. Create an app console session.

apc app console redmine

4. Connect to the console session.

apc console connect redmine

5. Update the Redmine job.

Create the directory for Redmine attachments:

root@ip-10-0-1-12:/# mkdir /nfs/data/files
root@ip-10-0-1-12:/# chmod 777 /nfs/data/files
root@ip-10-0-1-12:/# exit

6. Update Redmine's config file to save attachments to this new directory:

production:
  attachments_storage_path: /nfs/data/files

7. Redeploy the Redmine app then create an issue with an attachment to see it working.

Additional APCFS examples