Creating and Managing Packages

This article primarily describes how to create a custom package. To understand how packages are used within the system, refer to the package resolution section of the documentation.

A package is a collection of binary data, typically in a tarball, and associated metadata in a JSON file. A job, including an app, service, service gateway, stager, or anything else that runs in Apcera, starts as a package.

Apcera provides packages for operating systems such as Ubuntu 14.04, software such as nginx-1.11.0, and runtimes such as go-1.6. Refer to the Apcera Package Scripts repository for a list of the package scripts that Apcera provides. If you need a package that is not provided out-of-the-box, you can write you own package following the steps and examples in this article.

Overview

Package metadata includes provides and dependencies, and tells other packages what kinds of capabilities they can look for in this package. Each provide and dependency has a type, which can be either os, package, runtime, or file. In addition, provides and dependencies have a name designed to make each more specific. For example, an os dependency might have the name ubuntu. A runtime dependency might have the name ruby, but it might have a more specific name, say ruby 1.9.3. Stagers use provides and dependencies to arrive at a specific set of packages that satisfies all dependencies.

See the package example for more information.

The apc package command has the following subcommands:

Subcommand Purpose
build Creates a new package from a .conf file. See Creating a Package from a package script
create Creates a new package from local sources using staging pipelines. See Creating a Package using staging pipelines
delete Delete an existing package
download Download the raw package contents to a local file
export Export package(s) into a single package file
from file Create a new package from a tarball
list Show all packages in the current or specified namespace
replace Replace an existing package with a new one
show Show attributes of a package
update Change attributes of a package

Once a package exists, the show and update subcommands work together. show enables you to see the state of all changeable attributes, and update enables you to change any of them.

The list subcommand works the same as it does for other Apcera entities. It produces a list of all packages in the user's current namespace or the namespace provided with the optional --namespace flag.

The build command is used to create a package and is described in the next section.

Apcera typically has several packages of given type. It picks the best match it can when resolving packages for your app. For more details, refer the tutorial on resolving packages. You can modify the package's dependencies and provides only when the package is not ready.

Creating a Package with Staging Pipelines

The apc package create command creates a package from local source code, similar to how you use apc app create to create an application from local source code. But where the apc app create command creates a new package (the "application package") and links it to a newly created job, the apc package create command simply creates the application package for later use.

For example, suppose you have an HTML web application you want to make available as a package. The following command creates a new package ("website-package") from the example-static sample application:

apc package create website-package --path sample-apps/example-static --provides package.website
Deploy path [sample-apps/example-static]: 
╭───────────────────────────────────────────────────────────────╮
│                        Package Settings                       │
├───────────────────┬───────────────────────────────────────────┤
│              FQN: │ package::/sandbox/admin::website-package  │
│        Directory: │ /src/sample-apps/example-static           │
│ Staging Pipeline: │ (will be auto-detected)                   │
╰───────────────────┴───────────────────────────────────────────╯

Is this correct? [Y/n]: 
Packaging... done
Creating package "website-package"... done
Uploading package contents... 100.0% (809/809 B)
[staging] Subscribing to the staging process...
[staging] Beginning staging with 'stagpipe::/apcera::static-site' pipeline, 1 stager(s) defined.
[staging] Launching instance of stager 'static-site'...
[staging] Downloading package for processing...
[staging] Validating an index.htm or index.html file exists
[staging] Staging is complete.
Success!

As you can see in the above output, the static-site staging pipeline was automatically selected based on the contents of the source directory (an index.html file, in this case). By default, all files and folders at the specified --path are included in the generated package. You can exclude files from the built package by adding a .apcignore file to the target path. See Excluding files from built packages.

You can then create an application from the new package with the apc app from package command, for example:

apc app from package mywebsite \
-p /sandbox/admin::website-package \
-routes mysite.example.apcera-platform.io

Creating a Package from a Configuration File

The workflow for creating a package from a package script is as follows:

  1. Assemble the required elements in a directory.
  2. Create a package configuration file in the same directory.
  3. Include a build script in the configuration file.

Once you have done this, you build the package using the following command:

apc package build

When you build a package, Apcera does the following:

  • Looks in the directory containing the configuration file for the resources specified in the configuration file.
  • Gives the package the name supplied in the configuration file.

    You can use the --name option to specify a different name, for example:

    apc package build --name my-package

  • Uses a staging pipeline called compiler that is designed to build packages.

    You can use the --staging option to specify a different stager.

The configuration file uses a simple build definition format, which tells Apcera which sources and dependencies the package requires. Apcera's package configuration format is similar to those of services like Homebrew or MacPorts.

Sample Configuration File

The following sample package configuration file is for the Node.js v0.10.21 runtime.

To see more examples, check out the Apcera Package Scripts repository on GitHub.

name:    "node"
version: "0.10.21"

sources [
  { url: "https://apcera-sources.s3.amazonaws.com/node/node-v0.10.21.tar.gz",
    sha256: "7c125bf22c1756064f2a68310d4822f77c8134ce178b2faa6155671a8124140d" },
]

build_depends [ { package: "build-essential" } ]
depends       [ { os: "linux" } ]
provides      [ { runtime: "node" },
                { runtime: "node-0.10" },
                { runtime: "node-0.10.21" } ]

environment { "PATH": "/opt/apcera/node-0.10.21/bin:$PATH" }

cleanup [ "/usr/lib/python2.7" ] # this will come from it using python
                                 # it will generate *.pyc files

build (
      export INSTALLPATH=/opt/apcera/node-0.10.21

      sudo ln -f /usr/bin/python2.7 /usr/bin/python

      tar xzvf node-v0.10.21.tar.gz
      (
        cd node-v0.10.21
        ./configure --prefix=${INSTALLPATH}
        sudo make install
      )

      sudo mkdir -p ${INSTALLPATH}/etc
      echo 'cache = /app/.npm' > npmrc
      sudo chown root:root npmrc
      sudo mv npmrc ${INSTALLPATH}/etc/npmrc

      sudo rm -f /usr/bin/python
)

Configuration File Elements

Using 'package show' explains how to view attributes of a package. This section describes how to specify those attributes.

Here are the top-level elements of a configuration file:

  • name (required)

    The package name can be any string. APC uses name as part of the default package name.

  • version (required)

    The package version can be any string. APC uses version as part of the default package name.

  • sources (optional)

    sources is an array of URLs of remote files to be downloaded and verified. Each URL in the array is actually a set of attribute-value pairs, where the attributes are url, name, sha256, sha1, and md5.

    url is required, the rest are optional. The URL must resolve to a publicly available file. It can include a username and password for basic authentication.

    sha256, sha1, and md5 denote file verification methods, and the associated value is the corresponding key. The build fails if the verification fails.

    When used within the sources array, name is the name to use for the file in the build script. The default name is the final path element of the URL.

    Here is a sample sources element:

    sources [
      { url: "https://apcera-sources.s3.amazonaws.com/ruby/ruby-1.9.3-p448.tar.gz",
      sha256: "2f35e186543a03bec5e603296d6d8828b94ca58bab049b67b1ceb61d381bc8a7" },
      { url: "https://apcera-sources.s3.amazonaws.com/ruby/customyaml-0.1.0.tar.gz",
      name: "yaml",
      sha256: "7bf81554ae5ab2d9b6977da398ea789722e0db75b86bffdaeb4e66d961de6a37" },
      { url: "https://apcera-sources.s3.amazonaws.com/ruby/rubygems-2.1.7.tgz",
      sha256: "76288eb4238378189d1ef6b27a2f94241d3d21ba96ce5c9a6b1534a33ec1609f" },
      { url: "https://apcera-sources.s3.amazonaws.com/ruby/bundler-1.3.5.gem",
      sha256: "08b89047f7b829f3e197a28fb1bde74c3f5cfea1552f99dfba237fee30eaffe4" },
    ]
    
  • include_files (optional)

    include_files is an array of file paths that should exist on your local system. When APC runs a build, it uploads those files to the Apcera cluster and makes them available to the build script. The uploaded files have the same names as on the local system. APC generates the MD5, SHA1, and SHA256 locally and validates all of those hashes on the remote system after uploading them.

    Here is an example of an include_files section:

    include_files [
    "nginx-1.4.3.tar.gz",
    "patches/request_start_variable.patch",
    "patches/unused-variable.patch"
    ]
    

    The include_files parameter is capped at 128MB per file. Therefore, include_files is not designed for processing large files into packages. Although you can work around this limitation to some extent by increasing the memory size allocated to the Compiler Stager when you create a job or package, for large packages the recommended approach is to use apc package from file or the sources.url parameter in the package configuration file. See creating large packages for details.

  • depends (required)

    The depends block contains a list of package dependencies that are available before executing the build script. They remain in the package and are required for any job that wishes to use the package. If there are ambiguities or policy conflicts with the specified dependencies, Apcera uses dependency resolution to resolve them. Here is a sample depends block:

    depends       [ { os: "linux" },
                  { package: "apache-2.2" } ]
    
  • build_depends (optional)

    The build_depends block contains a list of package dependencies used only during the building process. These dependencies are automatically removed from the package after the build script finishes successfully. As with depends, Apcera uses dependency resolution for these dependencies. Here is a sample build_depends block:

    build_depends [ { package: "build-essential" } ]
    
  • provides (optional)

    Each provides in the block creates a provides on the package. Apcera dependency resolution looks at these.

    Here is a sample provides section for a Python 3.3.2 package:

    provides      [ { runtime: "python" },
                  { runtime: "python-3.3" },
                  { runtime: "python-3.3.2" } ]
    
  • environment (optional)

    The environment block enables you specify environment variables that should be set on the package after the build script has completed.

    The block is a set of colon-separated key value pairs: environment variable names and their values.

    Environment variables can come from multiple packages and from the running job. If you set environment variables that might also be set on other packages, have them expand on each other. For example, if you want to add a path to PATH, set it as /my/path:$PATH so that your part is added to the front of the existing PATH.

    Here is a sample environment block:

    environment
      { "PATH": "/opt/apcera/apache-2.2.25/bin:$PATH",
        "APACHE_ROOT": "/opt/apcera/apache-2.2.25",
        "APACHE_MODULES_PATH": "/opt/apcera/apache-2.2.25/modules" }
    
  • cleanup (optional) — A list of paths to be cleaned up before the final snapshot

    The cleanup section is a list of paths that should be removed after the build script is finished, before it takes the final snapshot of the filesystem.

    An example of the cleanup section is:

    cleanup [
      "/var/lib/dpkg",
      "/etc"
    ]
    
  • build (required)
    The build script section describes the format of the build script included in the configuration file.

Within the build configuration file, any text between a # and the end of the line is a comment.

  • snapshot_path (optional) - Directory of the image to take the snapshot of.

    The snapshot_path block specifies the directory of the image to take the snapshot of. Normally, a snapshot is taken of a currently running job. However, in case of a package script, you want the snapshot to be for the image that is being created (not the snapshot of the stager that created the image).

    Example usage:

    snapshot_path "/rootfs"
    

    For a more complete example, see the Ubuntu 14.04.3 package script.

Build Script

The build script generates the package. The build script is a bash script that takes the source files and turns them into the final package.

The working directory at the start of the script is a temporary Apcera directory created under /tmp, and containing the files specified in the sources block and the include_files list.

The build script is executed as a non-root user, but it does have access to passwordless sudo for gaining root access. The temporary directory that is created and the files in it are owned by the non-root user as well.

The build script is expected to take the sources to a final result. If the script exits with non-zero status, Apcera assumes the build failed. Otherwise it succeeded.

The build script can alter any parts of the isolated filesystem in Apcera. At the end of the build, the system cleans up the paths specified by the cleanup list and removes anything in /tmp. It then uses Apcera's snapshot functionality to capture the files that changed while the build script ran.

The build block can contain any Bash statements without extra escape characters. The only requirement is that a closing parenthesis is the first and only character of the final line of the block. You can format arbitrarily within the the block, but it's a good idea to indent the build script to prevent the closing parenthesis of an internal structure from accidentally terminating the block.

The following example illustrates this format:

build (
      export BUILDPATH=`pwd`
      export INSTALLPATH=/opt/apcera/ruby-1.9.3-p448

      tar xzvf yaml-0.1.4.tar.gz
      (
          cd yaml-0.1.4
          ./configure --disable-shared --with-pic
          make
      )

      tar xzvf ruby-1.9.3-p448.tar.gz
      (
        cd ruby-1.9.3-p448
        export LDFLAGS="-L${BUILDPATH}/yaml-0.1.4/src/.libs $LDFLAGS"
        export CPPFLAGS="-I${BUILDPATH}/yaml-0.1.4/include $CPPFLAGS"
        ./configure --prefix=${INSTALLPATH} --enable-shared --disable-install-doc
        make
        sudo make install
      )

      tar zxvf rubygems-2.1.7.tgz
      (
        cd rubygems-2.1.7
        sudo ${INSTALLPATH}/bin/ruby setup.rb
      )

      sudo ${INSTALLPATH}/bin/gem install bundler-1.3.5.gem --no-ri --no-rdoc
)

Using package show

The package show command is used to view a package's environment variables. It also shows detailed information about an individual package, including its dependencies and what it provides.

Example:

apc package show Go_1.1.2
Package:Go_1.1.2
FQN:package::/::Go_1.1.2
State:ready
Dependencies:os: linux
package: git
package: bzr
package: mercurial
Provides:runtime: go
runtime: go-1.1
runtime: go-1.1.2
Environment:GOPATH="/opt/apcera/go"
GOROOT="/opt/apcera/go1.1.2.linux-amd64"
PATH="/opt/apcera/go1.1.2.linux-amd64/bin:$PATH"

Creating an App from a Package

You can create an app from a package using the following command syntax:

apc app from package <app-name> -p <pkg-name> [optional-args]

Creating an app from a package may be useful for certain types of packages, such as when you want to use a capsule to host a service provider. When you snapshot a capsule, the snapshot is saved as a package. Using this package you can create an app, for example:

apc app from package mysql-app -p snapshot-mysql-cap-142670904 -dr --start-cmd 'sudo /bin/sh /usr/bin/mysqld_safe

To get the package name, run command apc package list.

Processing Large Packages

See how to upload and create large package files.