This topic describes how to enable and use Pod Security Policies in VMware Tanzu Kubernetes Grid Integrated Edition (TKGI).

Note: When the PodSecurityPolicy option is enabled for a Kubernetes cluster plan, a cluster administrator must define the policy, role, and role binding that gives cluster users (developers) permission to deploy pods to the cluster. See Workflow for Enabling PSPs.

About Pod Security Policies

In Kubernetes, a Pod Security Policy (PSP) is a cluster-level resource that controls security sensitive aspects of the pod specification. PodSecurityPolicy objects define a set of conditions that a pod must run with in order to be accepted into the system, as well as defaults for related fields. For more information, see Pod Security Policies in the Kubernetes documentation.

There are various ways to implement the use of PodSecurityPolicy objects in Kubernetes. One common approach is the use of role based access control (RBAC) objects: roles and role bindings. For a PodSecurityPolicy to take effect, the Kubernetes cluster User or ServiceAccount launching a workload must have the use permission on the desired PodSecurityPolicy object.

A role contains rules that represent a set of permissions. A role can be defined within a namespace with a Role, or cluster-wide with a ClusterRole. For more information, see Role and ClusterRole in the Kubenetes documentation.

A role binding grants the permissions defined in a role to a user or set of users. A role binding holds a list of subjects (users, groups, or service accounts), and a reference to the role being granted. Permissions can be granted within a namespace with a RoleBinding, or cluster-wide with a ClusterRoleBinding. For more information, see RoleBinding and ClusterRoleBinding in the Kubenetes documentation.

Note: The most commonly used Kubernetes workloads (deployment, replication controller, for example) are spinning up pods using a service account. It is this entity that requires the use permission on the PSP. Just because the kubectl-user has the use permission on a PSP does not mean your workload will spin up (unless you are doing simple pod-level workloads using kubectl run). To grant a service account use permission for a PSP, the user attempting to create the role binding must also have use on that PSP.

Default Pod Security Policies in TKGI

Tanzu Kubernetes Grid Integrated Edition ships with two default PSPs: PKS Privileged and PKS Restricted.

PSP Description
PKS Privileged Allows privileged access to pod containers, which allows the container to do almost everything a host can do. See Privileged in the Kubernetes PSP documentation for more information.
PKS Restricted Restricts privileged access to pod containers.

By default, when PSPs are enabled for a plan, the cluster administrator is bound to the PKS Privileged PSP. This policy grant gives the cluster administrator permission to deploy pods. Other users will not be able to deploy pods unless the cluster administrator creates and binds a PSP for such users. The PKS Restricted PSP is example for such purposes.

Default PSPs with TKGI

Note: The default PSPs will be updated and overwritten on upgrade. You should not edit the default PSPs.

Enabling Pod Security Policies in TKGI

Enabling PSPs is done during configuration of Tanzu Kubernetes Grid Integrated Edition in the Plan section of the tile configuration. Refer to the TKGI tile configuration documentation for your IaaS for details.

Note: To use PodSecurityPolicy features, you must use Ops Manager v2.10.17 or later.

Enabling Pod Security Policy

Note: Windows in Kubernetes does not support the PodSecurityPolicy Admission Plugin feature.

A Kubernetes cluster created from a plan with the PodSecurityPolicy option enabled will require cluster users to have a binding that grants use on an appropriate PSP to deploy pods. Enabling the PodSecurityPolicy option is a security feature. The design goal is to make the Kubernetes cluster more secure. Once enabled, PodSecurityPolicy acts on creation and modification of pods in that cluster or namespace, and determines the actions that should be permitted based on the requested security context and the applied PSP. See PSP Policy Order for more information on PSP application order when multiple PSPs are in use.

WARNING: Enabling the PodSecurityPolicy option on install or upgrade of Tanzu Kubernetes Grid Integrated Edition will prevent developers from using the cluster as they would expect unless the proper PSP, role, and role binding are configured by the cluster administrator before cluster deployment.

New Installations of TKGI: PSPs Are Not Enabled for Any Plan

For new Tanzu Kubernetes Grid Integrated Edition installations, the PodSecurityPolicy option is not enabled by default in a plan. If the PodSecurityPolicy option is enabled, the cluster administrator will be able to deploy pods, but developers without a binding to a PSP will not. For new deployments, the cluster administrator will need to create one or more PSPs, roles, and role bindings for developers to deploy pods. Once you enable PSPs, you will need to define the RBAC objects and PSP for cluster users. See Workflow for Enabling PSPs and Configuring the pks-restricted PSP for Developers to Use with TKGI.

Upgrade of TKGI: PSPs Must Be Enabled per Plan

On upgrade of Tanzu Kubernetes Grid Integrated Edition, existing plans will not have the PodSecurityPolicy option enabled. If the PSP option is enabled for a plan that is in use and did not previously leverage PSPs, and the cluster is upgraded, the cluster administrator will need to create the appropriate PSPs, roles, and role bindings BEFORE upgrading the clusters associated to that plan. Cluster upgrades when enabling PSPs on an existing plan can have unpredictable results on workloads if the appropriate PSPs are not enabled. If you are considering enabling PSPs on an existing plan to apply to all associated clusters, the following set of tasks is recommended:

Workflow for Enabling PSPs

Before you select the PodSecurityPolicy check box and enable PSPs for a new or existing plan, complete the following tasks:

  1. Review all existing pods to collect their security requirements for each cluster associated with that plan. There are some open-source tools to assist in this task, such as kube-psp-advisor.
  2. Create the appropriate PSPs, roles, and role bindings for each cluster associated with that plan to allow the pods to run after upgrade. See Configuring PSP for Developers to Use as a starting point.
  3. Enable PSPs by selecting the PodSecurityPolicy checkbox in the appropriate plan.
  4. Review the pending changes and verify that Upgrade all clusters errand is enabled.
  5. Apply the changes to update the clusters that use the plan with PSPs enabled.
  6. Verify that the workloads are in desired state after upgrade.
  7. If Upgrade all clusters errand was not enabled, run it manually and redeploy.

Alternatively, instead of enabling Upgrade all clusters errand, you can upgrade individual Kubernetes clusters through the TKGI Command Line Interface (TKGI CLI). For instructions on upgrading individual Kubernetes clusters, see Upgrading Clusters.

Configuring the pks-restricted PSP for Developers to Use with TKGI

This section describes how to define and configure a pod security policy for developers to use to access a TKGI provisioned cluster that has enabled PSPs.

At a high-level, the steps for configuring a PSP with appropriate RBAC controls are as follows:

  1. Define the PSP.
  2. Create a role.
  3. Create a role binding.

Step 1: Define the PSP

The first step is to define the PSP. We provide the PSP named pks-restricted for general development work in Kubernetes. To onboard cluster users (developers), the best practice is to start with the pks-restricted PSP. To create your own PSP, refer to the Kubernetes documentation.

To view the pks-restricted PSP, run the following command:

kubectl get psp pks-restricted -o yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  annotations:
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"policy/v1beta1","kind":"PodSecurityPolicy","metadata":{"annotations":{"apparmor.security.beta.kubernetes.io/allowedProfileNames":"runtime/default","apparmor.security.beta.kubernetes.io/defaultProfileName":"runtime/default","seccomp.security.alpha.kubernetes.io/allowedProfileNames":"docker/default","seccomp.security.alpha.kubernetes.io/defaultProfileName":"docker/default"},"name":"pks-restricted"},"spec":{"allowPrivilegeEscalation":false,"fsGroup":{"ranges":[{"max":65535,"min":1}],"rule":"MustRunAs"},"hostIPC":false,"hostNetwork":false,"hostPID":false,"privileged":false,"readOnlyRootFilesystem":false,"requiredDropCapabilities":["ALL"],"runAsUser":{"rule":"MustRunAsNonRoot"},"seLinux":{"rule":"RunAsAny"},"supplementalGroups":{"ranges":[{"max":65535,"min":1}],"rule":"MustRunAs"},"volumes":["configMap","emptyDir","projected","secret","downwardAPI","persistentVolumeClaim"]}}
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
  creationTimestamp: 2019-03-28T02:33:07Z
  name: pks-restricted
  resourceVersion: "261"
  selfLink: /apis/policy/v1beta1/podsecuritypolicies/pks-restricted
  uid: d25bb0eb-5101-11e9-abc1-0a834658b12e
spec:
  allowPrivilegeEscalation: false
  fsGroup:
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  requiredDropCapabilities:
  - ALL
  runAsUser:
    rule: MustRunAsNonRoot
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  volumes:
  - configMap
  - emptyDir
  - projected
  - secret
  - downwardAPI
  - persistentVolumeClaim

Step 2: Create the Role

The following ClusterRole grants permission to use the pks-restricted PSP resource. You can use this role for onboarding cluster users and granting them cluster access.

Below is the system-provided ClusterRole.

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: psp:restricted
rules:
- apiGroups:
  - extensions
  resources:
  - podsecuritypolicies
  resourceNames:
  - pks-restricted # the psp we are giving access to
  verbs:
  - use

Note: Because this is a ClusterRole example, rather than a Role example, the namespace is omitted.

For more information about Role and ClusterRole, see Role and ClusterRole in the Kubernetes documentation.

Step 3: Create the Role Binding

For onboarding cluster users, you have two options based on the mode of authentication you have chosen for TKGI, either internal or external:

  • Internal authentication uses a service account mechanism for role binding, such as SERVICE-UUID-cluster-admin, for example.
  • External authentication uses the OpenID Connect (OIDC) identity provider to interface with an external system such as LDAP or AD. In this case the role binding references the user or group name.

RoleBinding and ClusterRoleBinding configurations adhere to the following format:

apiVersion: rbac.authorization.k8s.io/v1
kind: ROLE-BINDING-TYPE
metadata:
  name: ROLE-BINDING-NAME
  namespace: NAMESPACE-NAME
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ROLE-REF-TYPE
  name: BIND-TO
subjects:
  - kind: ServiceAccount
    name: ACCOUNT-ID

Where:

  • ROLE-BINDING-TYPE is the binding type, either RoleBinding or ClusterRoleBinding.
  • ROLE-BINDING-NAME is your name for the role binding you are creating.
  • NAMESPACE-NAME is the name of the namespace the binding grants access within. namespace is optional for a ClusterRole binding.
  • ROLE-REF-TYPE is the roleRef type, either Role or ClusterRole. If the binding type is Role, the namespace property is required.
  • BIND-TO is the name of the ClusterRole to bind to.
  • ACCOUNT-TYPE is the type of account managed by the binding, either ServiceAccount, User, or Group.
  • ACCOUNT-ID is the account managed by the binding. The ACCOUNT-ID is case sensitive.

For example, the following ClusterRoleBinding examples bind specific groups or users to the psp:restricted ClusterRole, which provides permissions to use the pks-restricted PSP:

  • Example of binding a service account user:

    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: psp:restricted
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole # Must be Role or ClusterRole
      name: psp:restricted # The name of the ClusterRole to bind to
    subjects:
      - kind: ServiceAccount
        name: cluster-user
    
  • Example of binding an externally authenticated user named jane:

    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: psp:restricted
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole # This must be Role or ClusterRole
      name: psp:restricted # The name of the ClusterRole to bind to
    subjects:
      - kind: User
        name: jane # Name is case sensitive
    
  • Example of binding externally authenticated group members:

    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: psp:restricted
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole # This must be Role or ClusterRole
      name: psp:restricted # The name of the ClusterRole to bind to
    subjects:
      - kind: Group
        name: developers # Name is case sensitive
    

Note: Because these are ClusterRoleBinding examples, rather than RoleBinding examples, the namespace property is omitted.

For more information about RoleBinding and ClusterRoleBinding, see RoleBinding and ClusterRoleBinding in the Kubernetes documentation.

Administering PSPs, Roles, and RoleBindings

This section lists common kubectl commands for administering and managing PSPs and related objects in Kubernetes.

To view the default PSPs:

kubectl get psp
NAME                PRIV    CAPS   SELINUX    RUNASUSER          FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
pks-privileged      true    *      RunAsAny   RunAsAny           RunAsAny    RunAsAny    false            awsElasticBlockStore,azureDisk,azureFile,cephFS,configMap,csi,downwardAPI,emptyDir,fc,flexVolume,flocker,gcePersistentDisk,glusterfs,iscsi,nfs,persistentVolumeClaim,projected,portworxVolume,quobyte,rbd,scaleIO,secret,storageos,vsphereVolume
pks-restricted      false          RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim

To view the details of a particular PSP:

kubectl describe psp <psp-name>

For example:

kubectl describe psp pks-privileged
Name:  pks-privileged

Settings:
  Allow Privileged:                       true
  Allow Privilege Escalation:             true
  Default Add Capabilities:               <none>
  Required Drop Capabilities:             <none>
  Allowed Capabilities:                   *
  Allowed Volume Types:                   awsElasticBlockStore,azureDisk,azureFile,cephFS,configMap,csi,downwardAPI,emptyDir,fc,flexVolume,flocker,gcePersistentDisk,glusterfs,iscsi,nfs,persistentVolumeClaim,projected,portworxVolume,quobyte,rbd,scaleIO,secret,storageos,vsphereVolume
  Allow Host Network:                     true
  Allow Host Ports:                       0-65535
  Allow Host PID:                         true
  Allow Host IPC:                         true
  Read Only Root Filesystem:              false
  SELinux Context Strategy: RunAsAny
    User:                                 <none>
    Role:                                 <none>
    Type:                                 <none>
    Level:                                <none>
  Run As User Strategy: RunAsAny
    Ranges:                               <none>
  FSGroup Strategy: RunAsAny
    Ranges:                               <none>
  Supplemental Groups Strategy: RunAsAny
    Ranges:                               <none>
kubectl describe psp pks-restricted
Name:  pks-restricted

Settings:
  Allow Privileged:                        false
  Allow Privilege Escalation:              false
  Default Add Capabilities:                <none>
  Required Drop Capabilities:              ALL
  Allowed Capabilities:                    <none>
  Allowed Volume Types:                    configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
  Allow Host Network:                      false
  Allow Host Ports:                        <none>
  Allow Host PID:                          false
  Allow Host IPC:                          false
  Read Only Root Filesystem:               false
  SELinux Context Strategy: RunAsAny
    User:                                  <none>
    Role:                                  <none>
    Type:                                  <none>
    Level:                                 <none>
  Run As User Strategy: MustRunAsNonRoot
    Ranges:                                <none>
  FSGroup Strategy: MustRunAs
    Ranges:                                1-65535
  Supplemental Groups Strategy: MustRunAs
    Ranges:                                1-65535

To view the role bindings:

kubectl get rolebinding

To view the cluster role bindings:

kubectl get clusterrolebindings

In the results, the UUID-cluster-admin is a service account.

NAME                                                   AGE
9a605c3d-b2f3-4961-b2a4-804885d85eba-cluster-admin     23s
9e495241-65b0-43bb-ac3a-120cb4ee6533-cluster-admin     2d1h
c990e9cc-29fc-4696-8673-fc69a93904f4-cluster-admin     2d1h
cluster-admin                                          8d
system:basic-user                                      8d
telemetry-agent                                        8d

To view service accounts:

kubectl get sa
NAME                                   SECRETS   AGE
2c92e23f-6480-4265-9456-4c81bffaa9c6   1         2d1h
38cb4886-8e19-4da3-a522-3ebaa3d6e553   1         44h
default                                1         8d

To view the YAML for a cluster role binding:

kubectl get clusterrolebinding 9a605c3d-b2f3-4961-b2a4-804885d85eba-cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: 2019-04-05T18:54:04Z
  labels:
    generated: "true"
  name: 9a605c3d-b2f3-4961-b2a4-804885d85eba-cluster-admin
  resourceVersion: "983229"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/9a605c3d-b2f3-4961-b2a4-804885d85eba-cluster-admin
  uid: 2f24d7f0-57d4-11e9-abc1-0a834658b12e
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: 9a605c3d-b2f3-4961-b2a4-804885d85eba
  namespace: default

To view a cluster role:

kubectl get clusterrole cluster-admin
NAME            AGE
cluster-admin   8d

To view the YAML for a clusterrole:

kubectl get clusterrole cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: 2019-03-28T02:32:53Z
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: cluster-admin
  resourceVersion: "41"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/cluster-admin
  uid: c9c20648-5101-11e9-abc1-0a834658b12e
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
- nonResourceURLs:
  - '*'
  verbs:
  - '*'

Troubleshooting PSP Configuration

This section provides some troubleshooting tips for working with PodSecurityPolicy.

Verify Policy Order

The source of many common problems related to PodSecurityPolicy–including mutating, workloads passing validation, and workloads failing at runtime–can usually be traced to proper policy ordering. If you are experiencing such issues, see Policy Order in the Kubernetes documentation.

Verify Use Permission Grants

The most commonly used Kubernetes workloads (deployment and replication controller, for example) are spinning up pods using a service account. It is this entity that requires the use permission on the PSP. To grant a service account use permission for a PSP, the user attempting to create the role binding must also have use on that PSP.

PSP Files and Demo

Tanzu Kubernetes Grid Integrated Edition provides additional resources for implementing PSPs, including the two default PSPs provided with the product, example YAML files, and a PSP demo movie. Download the files at the following location: https://github.com/pivotal-cf/docs-pks/blob/master/demos/psp-demo.tar.gz.

check-circle-line exclamation-circle-line close-line
Scroll to top icon