When a container is built and/or used, it is important to ensure that the image is built by following best practices in terms of security, efficiency, performance, etc. This article will go over some of the key points VMware Tanzu Application Catalog (Tanzu Application Catalog) takes into account when publishing containers. It covers image tagging, non-root configuration and arbitrary UIDs, the importance of reducing size and dependencies, and the release process, including CVE scanning and tests.
A Docker tag is a label used to uniquely identify a Docker image. It allows users to deploy a specific version of an image. A single image can have multiple tags associated with it.
Every time Tanzu Application Catalog publishes a new version of an image, the associated tags are also updated to make it easier for users to get the latest version.
Tanzu Application Catalog uses rolling tags (a tag that may not always point to the same image) for its Docker container images. To understand how this works, let's use the Tanzu Application Catalog etcd container image tags as an example:
3, 3-debian-10, 3.4.13, 3.4.13-debian-10-r8, latest
latesttag always points to the latest revision of the etcd image.
3tag is a rolling tag that always points to the latest revision of etcd 3.x.
3.4.13tag is a rolling tag that points to the latest revision of etcd 3.4.13. It will be updated with different revisions or daily releases but only for etcd 3.4.13.
3-debian-10tag points to the latest revision of etcd 3.x for Debian 10, in case there are other distros supported.
When Tanzu Application Catalog releases container images - typically to upgrade system packages - it fixes bugs or improves the system configuration and also updates container tags to point to the latest revision of the image. Therefore, the rolling tags shown above are dynamic; they will always point to the latest revision or daily release of the corresponding image.
Continuing with the example above, the
3.4.13 tag might point to the etcd 3.4.13 revision 8 today, but it will refer to the etcd 3.4.13 revision 9 when Tanzu Application Catalog next updates the container image.
The suffix revision number (
rXX) is incremented every time that Tanzu Application Catalog releases an updated version of the image for the same version of the application. As explained in the next section, suffixed tags are also known as immutable tags.
A static, or immutable, tag always points to the same image. This is useful when you depend on a specific revision of an image For example, if you use the tag
3.4.13-debian-10-r8, this tag will always refer to etcd
3.4.13 revision 8. The use of this tag ensures that users get the same image every time.
Which tag should you use and when? Follow these guidelines:
There are two types of Tanzu Application Catalog container images: root and non-root. Non-root images add an extra layer of security and are generally recommended for production environments. However, because they run as a non-root user, privileged tasks such as installing system packages, editing configuration files, creating system users and groups, and modifying network information, are typically off-limits.
This section gives you a quick introduction to non-root container images, explains possible issues you might face using them, and also shows how to modify them to work as root images.
By default, Docker containers are run as
root users. This means that you can do whatever you want in the container, such as install system packages, edit configuration files, bind privilege ports, adjust permissions, create system users and groups, or access networking information.
With a non-root container, you can't do any of this. A non-root container must be configured only for its main purpose, for example, run the NGINX server.
A non-root container is a container in which the user executing the processes is not the
root user but a unprivileged user, like
1001. This is usually modified through the
USER instruction in the Dockerfile.
Non-root containers are recommended for the following reasons:
Security: Non-root containers are more secure. If there is a container engine security issue, running the container as an unprivileged user will prevent any malicious code from gaining elevated permissions on the container host. Learn more about Docker's security features.
Platform restrictions: Some Kubernetes distributions (such as OpenShift) run containers using random UUIDs. This approach is not compatible with root containers, which must always run with the
root user's UUID. In such cases, root-only container images will simply not run and a non-root image is a must. Learn more about random UUIDs
Non-root containers could also have some issues when used for local development:
Write failures on mounted volumes: Docker mounts host volumes preserving the host UUID and GUID. This can lead to permission conflicts with non-root containers, as the user running the container may not have the appropriate privileges to write on the host volume.
Write failures on persistent volumes in Kubernetes: Data persistence in Kubernetes is configured using persistent volumes. Kubernetes mounts these volumes with the root user as the owner; therefore, non-root containers don't have permissions to write to the persistent directory.
Issues with specific utilities or services: Some utilities (eg. Git) or servers (eg. PostgreSQL) run additional checks to find the user in the
/etc/passwd file. These checks will fail for non-root container images.
Tanzu Application Catalog non-root containers fix the above issues:
initContainerfor changing the volume permissions properly. As the image runs as non-root by default, it is necessary to adjust the ownership of the persistent volume so that the container can write data to it. By default, the charts are configured to use Kubernetes Security Context to automatically change the ownership of the volume. However, this feature does not work in all Kubernetes distributions. As an alternative, the charts support using an
initContainerto change the ownership of the volume before mounting it in the final destination.
libnss-wrapperpackage, which defines custom userspace files to ensure the software acts correctly.
If you wish to run a Tanzu Application Catalog non-root container image as a root container image, you can do it by adding the line
user: root right after the
image: directive in the container's
docker-compose.yml file. After making this change, restart the container and it will run as the
root user with all privileges instead of an unprivileged user.
In Kubernetes, the user that executes the container can be customized by using Security Context.
On some platforms like OpenShift, to support running containers with volumes mounted in a secure way, images must run as an arbitrary user ID. When those platforms mount volumes for a container, they configure the volume so it can only be written to by a particular user ID, and then run the image using that same user ID. This ensures the volume is only accessible to the appropriate container, but requires that the image is able to run as an arbitrary user ID.
That means a non-root container executing on a platform with this policy can't assume anything about the UUID. These platforms change the default container user to an arbitrary UUID, but the GUID is unmodified and containers are executed as
XXX is the arbitrary UUID).
Tanzu Application Catalog images are configured with the proper permissions for the user and group in order to meet the requirements of these platforms. They do this by ensuring that the
XXX user belongs to the
root group and that the directories have the appropriate read, write and execution permissions.
Each container should have only one concern. Decoupling applications into multiple containers makes it easier to scale horizontally and reuse containers. For instance, a web application stack might consist of three separate containers, each with its own unique image, to manage the web application, database, and an in-memory cache in a decoupled manner.
Although all Tanzu Application Catalog images follow this good practice, there are cases where two or more processes need to be executed at the same time in the same image. One such case is that of the Tanzu Application Catalog PostgreSQL with Replication Manager Docker Image where, apart from the
postgres process, there is a separate process for the
repmgr daemon. There are also other cases where the application spawns additional processes on its own.
It is therefore important to take a decision about the number of processes per container, keeping in mind the goal of keeping each container as clean and modular as possible.
Keep the following important security considerations in mind:
As indirectly described in the previous sections, it is important to follow the "Principle of least privilege" (POLP), an important concept in computer security. This refers to the practice of limiting access rights for users to the bare minimum permissions they need to perform their work.
In the same way, a good security practice is to install and maintain only the minimum necessary dependencies in a container image. It is also important to reduce the size of the images to improve the security, performance, efficiency, and maintainability of the containers.
Package installation in Tanzu Application Catalog images (also applicable to already-installed packages) is usually done using the
install_packages script. This tool was created to install system packages in a smart way for container environments. Apart from installing packages only with the required dependencies (no recommended packages or documentation), it also removes the cache and unnecessary package repositories.
Tanzu Application Catalog automatically re-releases its container catalog every 24 hours (this can be modified by the customer). In terms of security, releasing the Tanzu Application Catalog containers on a daily basis ensures that the system packages and components bundled in the image are up-to-date from the package repositories.
As explained previously, this approach means that a new immutable tag is produced every day, increasing the revision number. At the same time, rolling tags are updated to point to this new immutable tag.
Apart from daily releases, there are other processes that can trigger a new release. For example, if there is a new version (major, minor, or patch) of the main component, Tanzu Application Catalog's tracking system detects this new upstream release and trigger a new release of the Tanzu Application Catalog image, which uses the
-r0 tag suffix.
Before a new image is released, antivirus scanners and other tests are executed. If these are unsuccessful, the release is blocked. These are discussed in the following sections
If you are running development containers to create a proof of concept or for production workloads, you will probably already be aware of CVEs that may affect the container's operating system and packages. There are various tools/scanners to check containers for CVEs, such as Clair, Anchore, Notary and others.
There are two ways of ensuring the health of containers: using a virus scan or a CVE scan.
The virus scan is executed during the release process. The virus scan performed by Tanzu Application Catalog uses antivirus engines for scanning the files present in the container, stopping the release if a positive is detected.
While the antivirus scan is a blocking step when releasing a container, the CVE scan is a tool executed periodically to trigger new releases. This tool analyzes the containers bundled by the Tanzu Application Catalog Helm charts. If it finds a CVE, it triggers the release of the affected container.
During the release process, all containers are tested to work with all deployment technologies with which they are likely to be used:
Two types of tests are executed for each deployment method:
Containers should follow modern cryptographic standards for security. If customers require compliance with FIPS 140-2, Tanzu Application Catalog containers can ship a FIPS-enabled version of OpenSSL. In a FIPS-enabled kernel, OpenSSL (and the applications using it) will only use FIPS-approved encryption algorithms. In the case of applications that have a FIPS mode (such as Elasticsearch), this would be enabled as well.
By implementing the above points in the Tanzu Application Catalog build and release process, Tanzu Application Catalog ensures that its container images are built following best practices in terms of security and performance and can be safely used on most platforms as part of production deployments.