Using Kubernetes Profiles

This topic describes how to create and use Kubernetes profiles for VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) provisioned Kubernetes clusters.

Overview

Kubernetes profiles enable TKGI cluster administrators and cluster managers to customize Kubernetes component settings for their TKGI-provisioned Kubernetes clusters.

You can use a TKGI Kubernetes profile to configure Kubernetes components, such as kube-apiserver on the control plane or the kubelet on each node. Uses for Kubernetes profiles include encrypting secrets in an etcd database, adding an OIDC provider, and using a ResourceQuota admission control plugin. For more information about supported configurations, see TKGI Kubernetes Profile Use Cases below.

To create a TKGI Kubernetes profile:

• Create a TKGI Kubernetes Profile

To manage Kubernetes profiles:

• Manage TKGI Kubernetes Profiles

To use a Kubernetes profile:

• Use a TKGI Kubernetes Profile

Only TKGI cluster administrators and TKGI cluster managers can work with Kubernetes profiles:

• A cluster administrator defines a Kubernetes profile using a JSON configuration file and uses the TKGI CLI to create the Kubernetes profile from the definition.
• A cluster manager applies existing Kubernetes profiles to their clusters.

For information about the permissions required to create or use a Kubernetes profile, see Who Creates and Manages TKGI Kubernetes Profiles below.

Who Creates and Manages TKGI Kubernetes Profiles

To create and use a Kubernetes profile, you must have the either the pks.clusters.admin or pks.clusters.manage role.

If a user with the pks.clusters.admin-read-only role attempts to create a Kubernetes profile, they see the following error:

You do not have enough privileges to perform this action. Please contact the TKGI administrator.

“k8s” to “kubernetes” Alias in TKGI CLI

In the TKGI CLI, all commands that include k8s-profile are aliased to also use kubernetes-profile. For example, the tkgi k8s-profiles and tkgi kubernetes-profiles commands are equivalent.

For brevity, this documentation uses the k8s- versions.

Create a TKGI Kubernetes Profile

To create a TKGI Kubernetes profile:

1. Create a JSON-formatted Kubernetes profile. For information on the supported Kubernetes profile format, see TKGI Kubernetes Profile Format below.

2. Use the TKGI CLI to define the Kubernetes profile within TKGI. For information on creating the Kubernetes profile, see The tkgi create-k8s-profile Command, below.

TKGI Kubernetes Profile Format

To create a Kubernetes profile, you must first define the profile configuration using a JSON-formatted file that specifies Kubernetes parameters. For information on the supported configurations, see TKGI Kubernetes Profile Parameters below.

The basic structure of a Kubernetes profile:

{  
    "name": "PROFILE-NAME",  
    "description": "PROFILE-DESCRIPTION",  
    "customizations": [  
        {  
            "component": "K8-COMPONENT-NAME",  
            "arguments": {  
                "KEY": "VALUE"  
            }  
        }  
    ],  
    "experimental_customizations": [  
        {  
            "component": "K8-COMPONENT-NAME",  
            "arguments": {  
                "KEY": "VALUE"  
            }  
        }  
    ]  
}  

Where:

• PROFILE-NAME is your name for the Kubernetes profile.
• PROFILE-DESCRIPTION is a short description of the profile.
• K8-COMPONENT-NAME is the name of the Kubernetes component to customize. For example, kube-apiserver or kubelet.
• KEY is a supported Kubernetes component parameter. For example, service-node-port-range, maximum-dead-containers, or feature-gates.
• VALUE is a value for the parameter. For example, APIListChunking=true.

TKGI Kubernetes Profile Parameters

The Kubernetes profile JSON can include the following parameters:

Parameter	Type	Description
name*	String	Name of the Kubernetes profile.
description	String	Description of the Kubernetes profile.
customizations*	Map	A block that defines supported, validated K8s options using the component, arguments, and file-arguments parameters below.
experimental_ customizations	Map	A block that defines unsupported, unvalidated K8s options using the component, arguments, and file-arguments parameters below.
component*	String	The name of a K8s component, e.g. kubelet, kube-apiserver.
arguments*	Map	Parameters for each component, as one or more Key:Value pairs. Multiple values must be separated by commas, without spaces.
file-arguments*	Map	Parameters whose values are stored as files on the local machine.

^*Parameters marked with an * are mandatory.

Note: If you specify the same parameter in both customizations and experimental_customizations, the configuration in customizations takes precedence.

The tkgi create-k8s-profile Command

After you have defined a Kubernetes profile configuration in a JSON file you can create the Kubernetes profile in TKGI.

You must be either a TKGI cluster administrator or cluster manager to create a Kubernetes profile in TKGI.

To create a Kubernetes profile in TKGI, run the following TKGI CLI command:

tkgi create-k8s-profile CONFIGURATION-FILE

Where CONFIGURATION-FILE is the path to a JSON configuration file defining your Kubernetes profile. For information on the required JSON file format, see Kubernetes Profile Format above.

For example:

cat profile3-docs.json  
{  
    "name": "my-profile3",  
    "description": "My profile description",  
    "customizations": [  
        {  
            "component": "kube-apiserver",  
            "arguments": {  
                "service-node-port-range": "30000-40000"  
            }  
        }  
    ],  
    "experimental_customizations": [  
        {  
            "component": "kubelet",  
            "arguments": {  
                "maximum-dead-containers": "1000",  
                "feature-gates": "APIListChunking=true"
            }  
        }  
    ]  
}  
user ~/workspace: ./tkgi create-k8s-profile k8s-profile3.json  
Kubernetes profile my-profile3 successfully created  

Manage TKGI Kubernetes Profiles

Tanzu Kubernetes Grid Integrated Edition cluster administrators and managers can perform the following operations on Kubernetes profiles and the clusters that use them:

• List TKGI Kubernetes Profiles
• Delete a TKGI Kubernetes Profile
• View TKGI Kubernetes Profile Details

List TKGI Kubernetes Profiles

To list available Kubernetes profiles, run the following command:

tkgi k8s-profiles

For example:

$ tkgi k8s-profiles  
K8s-profile        Owner     Created Date  
Basic-k8s-profile  Alana     Tue, 05 Nov 2019 16:28:15 PST  

The command output differs by user role:
- pks.cluster.admin see a list of Kubernetes profiles that all users created.
- pks.cluster.manage see a list of only the Kubernetes profiles that they created.

Delete a TKGI Kubernetes Profile

To delete a Kubernetes profile, run the following command:

tkgi delete-k8s-profile KUBERNETES-PROFILE-NAME

Where KUBERNETES-PROFILE-NAME is the name of the Kubernetes profile you want to delete.

For example:

$ user ~/workspace: ./tkgi delete-k8s-profile my-profile3  
Are you sure you want to delete the Kubernetes profile my-profile3? (y/n): y  
Deletion of my-profile3 completed  

Note: You cannot delete a Kubernetes profile that is in use. Before deleting a Kubernetes profile, you must disassociate it from all clusters or delete all clusters it is associated with.

Both pks.clusters.admin and pks.clusters.manage users can delete Kubernetes profiles. If a pks.clusters.admin-read-only user attempts to delete a Kubernetes profile, they see the following error:

You do not have enough privileges to perform this action. Please contact the TKGI administrator.

View TKGI Kubernetes Profile Details

To view details of a TKGI Kubernetes profile, run the following command:

tkgi k8s-profile KUBERNETES-PROFILE-NAME

Where KUBERNETES-PROFILE-NAME is the name of the Kubernetes profile you want to view.

For example:

tkgi k8s-profile Basic-k8s-profile  
Name:                    Basic-k8s-profile  
Owner:                   Alana  
Created Date             Tue, 05 Nov 2019 16:28:15 PST  
Description:             Kubernetes profile for customer A  
….  
<KEY> :                  <VALUE>  

• Users with the pks.cluster.admin can view the details of any Kubernetes profile; users with the pks.cluster.manage role can view details of Kubernetes profiles that they created.
• Once you create or update a cluster with an encryption profile, you cannot assign any other Kubernetes profiles to that cluster. Because decryption is not straightforward, applying another profile can have nondeterministic outcome.

Use a TKGI Kubernetes Profile

Kubernetes profiles let you customize Kubernetes configuration parameters at the time of cluster creation. You can also assign a custom Kubernetes configuration to an existing cluster.

For information on how to assign a Kubernetes Profile to a cluster:

• Create a TKGI Cluster with a Kubernetes Profile
• Assign a Kubernetes Profile to an Existing TKGI Cluster

For information on the supported scenarios for using Kubernetes Profiles:

• Validated vs Experimental Customizations
• Kubernetes Profile Use Cases

Create a TKGI Cluster with a Kubernetes Profile

You can assign a Kubernetes profile to a Kubernetes cluster at the time of cluster creation.

To create an Tanzu Kubernetes Grid Integrated Edition-provisioned Kubernetes cluster with a Kubernetes profile, run the following command:

tkgi create-cluster CLUSTER-NAME --external-hostname HOSTNAME --plan PLAN-NAME --kubernetes-profile KUBERNETES-PROFILE-NAME

Where:

• CLUSTER-NAME is a unique name for your cluster.

  Note: Use only lowercase characters when naming your cluster if you manage your clusters with Tanzu Mission Control (TMC). Clusters with names that include an uppercase character cannot be attached to TMC.

• HOSTNAME is your external hostname used for accessing the Kubernetes API.
• PLAN-NAME is the name of the Tanzu Kubernetes Grid Integrated Edition plan you want to use for your cluster.
• KUBERNETES-PROFILE-NAME is the name of the Kubernetes profile you want to use for your cluster.

Assign a Kubernetes Profile to an Existing TKGI Cluster

TKGI supports changing the Kubernetes profile for an already created cluster. You can use this procedure to:

• assign a Kubernetes profile to a cluster that does not have one, or
• change a cluster’s existing profile to a new one

WARNING: Update the Kubernetes profile only on a TKGI cluster that has been upgraded to the current TKGI version. For more information, see Tasks Supported Following a TKGI Control Plane Upgrade in About Tanzu Kubernetes Grid Integrated Edition Upgrades.

This is the procedure to change a cluster’s Kubernetes profile:

1. Do one of the following

   • Choose a new Kubernetes profile for the cluster. See List Kubernetes profiles.
   • Define and create a new Kubernetes profile as described in Create a Kubernetes Profile.
     • The name of the new Kubernetes profile must be unique and different from the previously assigned Kubernetes profile.

2. If you are updating a cluster that uses a public cloud CSI driver, see Limitations on Using a Public Cloud CSI Driver in Release Notes for additional requirements.

3. Run the following command to update the cluster with the new Kubernetes profile:

   tkgi update-cluster CLUSTER-NAME --kubernetes-profile NEW-KUBERNETES-PROFILE-NAME
   

   Where:

   • CLUSTER-NAME is the name of the existing Kubernetes cluster.
   • NEW-KUBERNETES-PROFILE-NAME is the name of the new Kubernetes profile you want to apply to the cluster.

Validated vs Experimental Customizations

A Kubernetes profile configures settings for Kubernetes components in two JSON code blocks, customizations and experimental_customizations. See Kubernetes Profile Format for details. The code blocks differ as follows:

• customizations block:

  • TKGI checks the validity of configurations in this block. If you run tkgi create-k8s-profile on a profile with invalid configurations in customizations, the command returns an error.
  • The TKGI team supports clusters configured with tested, validated parameters in this block.

• experimental_customizations block:
  

  Warning: Experimental customizations are not validated or supported.

  • TKGI imposes no restrictions on the contents of this block.
  • Configurations in this block are neither tested nor supported.
  • If a customer wants to use an unsupported configuration, contact the TKGI team with the parameters that they want tested, validated, and supported.

TKGI Kubernetes Profile Use Cases

Kubernetes profiles let you customize the Kubernetes configuration parameters on a cluster.

Use cases for Kubernetes profiles include:

Use Case	Description
Encrypt a secret in an etcd database.	Use an encryption provider to encrypt secrets in a cluster’s etcd database. For more information, see Encrypt Secrets in an etcd Database.
Limit the resource usage of incoming requests.	Use the ResourceQuota admission control plugin to restrict incoming requests by resource usage. For more information, see Admission Control: ResourceQuota below.
Assign an IP range for the NodePort Service.	Use service-node-port-range to specify an IP range for for NodePort services. For more information, see Set Service Node Port Range below.
Add an OIDC provider.	Customize a cluster’s OIDC provider by deploying a dex connector or other OIDC provider to its pod. For more information, see Adding an OIDC Provider.
Restrict Apiserver client authentication.	Set requestheader-allowed-names for Apiserver client authentication. For more information, see Restrict Request Header Names below.
Define the service cluster IP range.	Change the service cluster IP range. For more information, see Modify the Service Cluster IP Range below.
Configure Pod Security Admission.	Configure cluster-specific PSA in TKGI. For more information, see Pod Security Admission in a TKGI Cluster in Pod Security Admission in TKGI.

Admission Control: ResourceQuota

To create a Kubernetes profile that includes the ResourceQuota admission control plugin:

• Follow the Create a Kubernetes Profile instructions.

• Include the following customizations in your profile configuration file:

    "customizations": [
        {
            "component": "kube-apiserver",
            "arguments": {
                "enable-admission-plugins": PLUGINS-LIST
            }
        }
    ],
  

  Where:

  • PLUGINS-LIST is one of the following:
    • The string "ResourceQuota".
    • A comma-delimited string list of validated plugins that includes ResourceQuota.

For more information, see ResourceQuota in the Kubernetes documentation.

Set Service Node Port Range

To create a Kubernetes profile that uses service-node-port-range for NodePort type services:

• Follow the Create a Kubernetes Profile instructions.

• Include the following customizations in your profile configuration file:

    "customizations": [
        {
            "component": "kube-apiserver",
            "arguments": {
                "service-node-port-range": PORT-RANGE
            }
        }
    ],
  

  Where PORT-RANGE is a CIDR notation IP range from which to assign service cluster IPs, such as 30000-40000.

If the specified PORT-RANGE is not valid, the tkgi create-k8s-profile command returns an error invalid value for service-node-port-range.

For more information, see Type NodePort in the Kubernetes documentation.

Restrict Request Header Names

To create a Kubernetes profile that uses requestheader-allowed-names for Apiserver client authentication:

• Follow the Create a Kubernetes Profile instructions.

• Include the following customizations in your profile configuration file:

    "customizations": [
        {
            "component": "kube-apiserver",
            "arguments": {
                "requestheader-allowed-names": COMMON-NAMES
            }
        }
    ],
  

  Where COMMON-NAMES is a string list of valid Common Name values in the signed client certificate, such as "cn1.com,c2.com".

For more information, see Kubernetes Apiserver Client Authentication in the Kubernetes documentation.

Modify the Service Cluster IP Range

To create a Kubernetes profile that modifies the service cluster IP range:

• Follow the Create a Kubernetes Profile instructions.

• Include the following customizations in your profile configuration file:

    "customizations": [
        {
            "component": "kube-apiserver",
            "arguments": {
                "service-cluster-ip-range": IP-RANGE
            }
        }
    ],
  

  Where IP-RANGE is a CIDR notation IP range from which to assign service cluster IPs. The IP range can be a maximum of two dual-stack CIDRs and must not overlap with any IP ranges assigned to nodes or pods.

For more information, see kube-apiserver Options in the Kubernetes documentation.